Alert Conditions¶

On this page

Overview
Host Alerts
Replica Set Alerts
Agent Alerts
Backup Alerts
User Alerts
Group Alerts

Overview¶

When you create a group or global alert configuration, specify the targets and alert conditions described here.

Host Alerts¶

When configuring an alert that applies to hosts, you select the host type as well as the alert condition.

Host Types¶

For host type, you can apply the alerts to all MongoDB processes or to a specific type of process:

Host Type	Applies To
Any type	All the types described here.
Standalone	Any mongod instance that is not part of a replica set or sharded cluster and that is not used as a config server.
Primary	All replica set primaries.
Secondary	All replica set secondaries.
Mongos	All mongos instances.
Conf	All mongod instances used as config servers.

Host Alert Conditions¶

Status¶

The following conditions apply to a change in status for a MongoDB process:

is added¶: Sends an alert when Ops Manager starts monitoring or managing a mongod or mongos process for the first time.

is removed¶: Sends an alert when Ops Manager stops monitoring or managing a mongod or mongos process for the first time.

is added to replica set¶: Sends an alert when the specified type of mongod process is added to a replica set.

is removed from replica set¶: Sends an alert when the specified type of mongod process is removed from a replica set.

is down¶: Sends an alert when Ops Manager does not receive a ping from a host for more than 4 minutes. Under normal operation, the Monitoring Agent connects to each monitored host about once per minute. Ops Manager will not alert immediately, however, but waits 4 minutes in order to minimize false positives, as would occur, for example, during a host restart.

is recovering¶: Sends an alert when a secondary member of a replica set enters the RECOVERING state. For information on the RECOVERING state, see Replica Set Member States.

Asserts¶

The following alert conditions measure the rate of asserts for a MongoDB process, as collected from the MongoDB serverStatus command’s asserts document. You can view asserts through deployment metrics.

Asserts: Regular is¶: Sends an alert if the rate of regular asserts meets the specified threshold.

Asserts: Warning is¶: Sends an alert if the rate of warnings meets the specified threshold.

Asserts: Msg is¶: Sends an alert if the rate of message asserts meets the specified threshold. Message asserts are internal server errors. Stack traces are logged for these.

Asserts: User is¶: Sends an alert if the rate of errors generated by users meets the specified threshold.

Opcounter¶

The following alert conditions measure the rate of database operations on a MongoDB process since the process last started, as collected from the MongoDB serverStatus command’s opcounters document. You can view opcounters through deployment metrics.

Opcounter: Cmd is¶: Sends an alert if the rate of commands performed meets the specified threshold.

Opcounter: Query is¶: Sends an alert if the rate of queries meets the specified threshold.

Opcounter: Update is¶: Sends an alert if the rate of updates meets the specified threshold.

Opcounter: Delete is¶: Sends an alert if the rate of deletes meets the specified threshold.

Opcounter: Insert is¶: Sends an alert if the rate of inserts meets the specified threshold.

Opcounter: Getmores is¶: Sends an alert if the rate of getmore (i.e. cursor batch) operations meets the specified threshold. For more information on getmore operations, see the Cursors page in the MongoDB manual.

Opcounter - Repl¶

The following alert conditions measure the rate of database operations on MongoDB secondaries, as collected from the MongoDB serverStatus command’s opcountersRepl document. You can view these metrics on the Opcounters - Repl chart, accessed through deployment metrics.

Opcounter: Repl Cmd is¶: Sends an alert if the rate of replicated commands meets the specified threshold.

Opcounter: Repl Update is¶: Sends an alert if the rate of replicated updates meets the specified threshold.

Opcounter: Repl Delete is¶: Sends an alert if the rate of replicated deletes meets the specified threshold.

Opcounter: Repl Insert is¶: Sends an alert if the rate of replicated inserts meets the specified threshold.

Memory¶

The following alert conditions measure memory for a MongoDB process, as collected from the MongoDB serverStatus command’s mem document. You can view these metrics on the Ops Manager Memory and Non-Mapped Virtual Memory charts, accessed through deployment metrics.

Memory: Resident is¶: Sends an alert if the size of the resident memory meets the specified threshold. It is typical over time, on a dedicated database server, for the size of the resident memory to approach the amount of physical RAM on the box.

Memory: Virtual is¶: Sends an alert if the size of virtual memory for the mongod process meets the specified threshold. You can use this alert to flag excessive memory outside of memory mapping. For more information, click the memory chart’s i icon.

Memory: Mapped is¶: Sends an alert if the size of mapped memory, which maps the data files, meets the specified threshold. As MongoDB memory-maps all the data files, the size of mapped memory is likely to approach total database size.

Memory: Computed is¶: Sends an alert if the size of virtual memory that is not accounted for by memory-mapping meets the specified threshold. If this number is very high (multiple gigabytes), it indicates that excessive memory is being used outside of memory mapping. For more information on how to use this metric, view the non-mapped virtual memory chart and click the chart’s i icon.

B-tree¶

These alert conditions refer to the metrics found on the host’s btree chart. To view the chart, see Procedure.

B-tree: accesses is¶: Sends an alert if the number of accesses to B-tree indexes meets the specified average.

B-tree: hits is¶: Sends an alert if the number of times a B-tree page was in memory meets the specified average.

B-tree: misses is¶: Sends an alert if the number of times a B-tree page was not in memory meets the specified average.

B-tree: miss ratio is¶: Sends an alert if the ratio of misses to hits meets the specified threshold.

Lock %¶

This alert condition refers to metric found on the host’s lock % chart. To view the chart, see Procedure.

Effective Lock % is¶: Sends an alert if the amount of time the host is write locked meets the specified threshold. For details on this metric, view the lock % chart and click the chart’s i icon.

Background¶

This alert condition refers to metric found on the host’s background flush avg chart. To view the chart, see Procedure.

Background Flush Average is¶: Sends an alert if the average time for background flushes meets the specified threshold. For details on this metric, view the background flush avg chart and click the chart’s i icon.

Connections¶

The following alert condition measures connections to a MongoDB process, as collected from the MongoDB serverStatus command’s connections document. You can view this metric on the Ops Manager Connections chart, accessed through deployment metrics.

Connections is¶: Sends an alert if the number of active connections to the host meets the specified average.

Queues¶

The following alert conditions measure operations waiting on locks, as collected from the MongoDB serverStatus command’s globalLock document. You can view these metrics on the Ops Manager Queues chart, accessed through deployment metrics.

Queues: Total is¶: Sends an alert if the number of operations waiting on a lock of any type meets the specified average.

Queues: Readers is¶: Sends an alert if the number of operations waiting on a read lock meets the specified average.

Queues: Writers is¶: Sends an alert if the number of operations waiting on a write lock meets the specified average.

Page Faults¶

These alert conditions refer to metrics found on the host’s Record Stats and Page Faults charts. To view the charts, see Procedure.

Accesses Not In Memory: Total is¶: Sends an alert if the rate of disk accesses meets the specified threshold. MongoDB must access data on disk if your working set does not fit in memory. This metric is found on the host’s Record Stats chart.

Page Fault Exceptions Thrown: Total is¶: Sends an alert if the rate of page fault exceptions thrown meets the specified threshold. This metric is found on the host’s Record Stats chart.

Page Faults is¶: Sends an alert if the rate of page faults (whether or not an exception is thrown) meets the specified threshold. This metric is found on the host’s Page Faults chart.

Cursors¶

The following alert conditions measure the number of cursors for a MongoDB process, as collected from the MongoDB serverStatus command’s metrics.cursor document. You can view these metrics on the Ops Manager Cursors chart, accessed through deployment metrics.

Cursors: Open is¶: Sends an alert if the number of cursors the server is maintaining for clients meets the specified average.

Cursors: Timed Out is¶: Sends an alert if the number of timed-out cursors the server is maintaining for clients meets the specified average.

Cursors: Client Cursors Size is¶: Sends an alert if the cumulative size of the cursors the server is maintaining for clients meets the specified average.

Network¶

The following alert conditions measure throughput for MongoDB process, as collected from the MongoDB serverStatus command’s network document. You can view these metrics on a host’s Network chart, accessed through deployment metrics.

Network: Bytes In is¶: Sends an alert if the number of bytes sent to the database server meets the specified threshold.

Network: Bytes Out is¶: Sends an alert if the number of bytes sent from the database server meets the specified threshold.

Network: Num Requests is¶: Sends an alert if the number of requests sent to the database server meets the specified average.

Replication Oplog¶

The following alert conditions apply to the MongoDB process’s oplog. You can view these metrics on the following charts, accessed through deployment metrics:

Replication Oplog Window
Replication Lag
Replication Headroom
Oplog GB/Hour

The following alert conditions apply to the oplog:

Replication Oplog Window is¶: Sends an alert if the approximate amount of time available in the primary’s replication oplog meets the specified threshold.

Replication Lag is¶: Sends an alert if the approximate amount of time that the secondary is behind the primary meets the specified threshold.

Replication Headroom is¶: Sends an alert when the difference between the primary oplog window and the replication lag time on a secondary meets the specified threshold.

Oplog Data per Hour is¶: Sends an alert when the amount of data per hour being written to a primary’s oplog meets the specified threshold.

DB Storage¶

This alert condition refers to the metric displayed on the host’s db storage chart. To view the chart, see Procedure.

DB Storage is¶

Sends an alert if the amount of on-disk storage space used by extents meets the specified threshold. Extents are contiguously allocated chunks of datafile space.

DB storage size is larger than DB data size because storage size measures the entirety of each extent, including space not used by documents. For more information on extents, see the collStats command.

DB Data Size is¶: Sends an alert if approximate size of all documents (and their paddings) meets the specified threshold.

Journaling¶

These alert conditions refer to the metrics found on the host’s journal - commits in write lock chart and journal stats chart. To view the charts, see Procedure.

Journaling Commits in Write Lock is¶: Sends an alert if the rate of commits that occurred while the database was in write lock meets the specified average.

Journaling MB is¶: Sends an alert if the average amount of data written to the recovery log meets the specified threshold.

Journaling Write Data Files MB is¶: Sends an alert if the average amount of data written to the data files meets the specified threshold.

WiredTiger Storage Engine¶

The following alert conditions apply to a MongoDB process’s WiredTiger storage engine, as collected from the MongoDB serverStatus command’s wiredTiger.cache and wiredTiger.concurrentTransactions documents.

You can view these metrics on the following charts, accessed through deployment metrics:

Tickets Available
Cache Activity
Cache Usage

The following are the alert conditions that apply to WiredTiger:

Tickets Available: Reads is¶: Sends an alert if the number of read tickets available to the WiredTiger storage engine meet the specified threshold.

Tickets Available: Writes is¶: Sends an alert if the number of write tickets available to the WiredTiger storage engine meet the specified threshold.

Cache: Dirty Bytes is¶: Sends an alert when the number of dirty bytes in the WiredTiger cache meets the specified threshold.

Cache: Used Bytes is¶: Sends an alert when the number of used bytes in the WiredTiger cache meets the specified threshold.

Cache: Bytes Read Into Cache is¶: Sends an alert when the number of bytes read into the WiredTiger cache meets the specified threshold.

Cache: Bytes Written From Cache is¶: Sends an alert when the number of bytes written from the WiredTiger cache meets the specified threshold.

Replica Set Alerts¶

These alert conditions apply to replica sets.

Primary Elected¶

Sends an alert when a set elects a new primary. Each time Ops Manager receives a ping, it inspects the output of the replica set’s rs.status() method for the status of each replica set member. From this output, Ops Manager determines which replica set member is the primary. If the primary found in the ping data is different than the current primary known to Ops Manager, this alert triggers.

Primary Elected does not always mean that the set elected a new primary. Primary Elected may also trigger when the same primary is re-elected. This can happen when Ops Manager processes a ping in the midst of an election.

No Primary¶

Sends an alert when a replica set does not have a primary. Specifically, when none of the members of a replica set have a status of PRIMARY, the alert triggers. For example, this condition may arise when a set has an even number of voting members resulting in a tie.

If the Monitoring Agent collects data during an election for primary, this alert might send a false positive. To prevent such false positives, set the alert configuration’s after waiting interval (in the configuration’s Send to section).

Number of Healthy Members is¶

Sends an alert when a replica set has fewer than the specified number of healthy members. If the replica set has the specified number of healthy members or more, Ops Manager triggers no alert.

A replica set member is healthy if its state, as reported in the rs.status() output, is either PRIMARY or SECONDARY. Hidden secondaries and arbiters are not counted.

As an example, if you have a replica set with one member in the PRIMARY state, two members in the SECONDARY state, one hidden member in the SECONDARY, one ARBITER, and one member in the RECOVERING state, then the healthy count is 3.

Number of Unhealthy Members is¶

Sends an alert when a replica set has more than the specified number of unhealthy members. If the replica set has the specified number or fewer, Ops Manager sends no alert.

Replica set members are unhealthy when the agent cannot connect to them, or the member is in a rollback or recovering state.

Hidden secondaries are not counted.

Agent Alerts¶

These alert conditions apply to Monitoring Agents and Backup Agents.

Monitoring Agent is down¶: Sends an alert if the Monitoring Agent has been down for at least 7 minutes. Under normal operation, the Monitoring Agent sends a ping to Ops Manager roughly once per minute. If Ops Manager does not receive a ping for at least 7 minutes, this alert triggers. However, this alert will never trigger for a group that has no hosts configured.

Important

When the Monitoring Agent is down, Ops Manager will trigger no other alerts. For example, if a host is down there is no Monitoring Agent to send data to Ops Manager that could trigger new alerts.

Monitoring Agent is out of date¶: Sends an alert when the Monitoring Agent is not running the latest version of the software.

Backup Agent is down¶

Sends an alert when the Backup Agent for a group with at least one active replica set or cluster is down for more than 1 hour.

To resolve this alert:

Open the group in Ops Manager by typing the group’s name in the GROUP box.
Select the Backup tab and then the Backup Agents page to see what server the Backup Agent is hosted on.
Check the Backup Agent log file on that server.

Backup Agent is out of date¶: Sends an alert when the Backup Agent is not running the latest version of the software.

Backup Agent has too many conf call failures¶

Available only as a global alert.

Sends an alert when the cluster topology as known by monitoring does not match the backup configuration from conf calls made by the Backup Agent. Ops Manager sends the alert after the number of attempts specified in the maximumFailedConfCalls setting.

Backup Alerts¶

These alert conditions apply to Ops Manager Backup.

Oplog Behind¶

Sends an alert if the most recent oplog data received by Ops Manager is more than 75 minutes old.

To resolve this alert, see Receive “Oplog Behind” alert.

Resync Required¶

Sends an alert if the replication process for a backup falls too far behind the oplog to catch up. This occurs when the host overwrites oplog entries that backup has not yet replicated. When this happens, you must resync backup, as described in the procedure Resync a Backup.

Also, check the corresponding Backup Agent log. If you see a “Failed Common Points” test, one of the following may have happened.

A significant rollback event occurred on the backed-up replica set.
The oplog for the backed-up replica set was resized or deleted.
High oplog churn caused the agent to lose the tail of the oplog.

Cluster Mongos Is Missing¶: Sends an alert if Ops Manager cannot reach a mongos for the cluster.

Bind Error¶

Available only as a global alert.

Sends an alert if a backup job fails to bind to a Backup Daemon. A job might fail to bind if, for example:

No primary is found for the backed-up replica set. At the time the binding occurred, the Monitoring Agent did not detect a primary. Ensure that the replica set is healthy.
Not enough space is available on any Backup Daemon.

In both cases, resolve the issue and then restart the initial sync of the backup.

As an alternative, you can manually bind jobs to daemons through the Admin interface. See Jobs Page for more information.

Backup has reached a high number of retries¶

Available only as a global alert.

Sends an alert if the same task fails repeatedly. This could happen, for example, during maintenance. Check the corresponding job log for an error message explaining the problem. Contact MongoDB Support if you need help interpreting the error message.

Backup is in an unexpected state¶

Available only as a global alert.

Sends an alert when something unexpected happened and the Backup state for the replica set is broken. You must resync the backed-up replica set, as described in the Resync a Backup procedure.

In case of a Backup is in an unexpected state alert, check the corresponding job log for an error message explaining the problem. Contact MongoDB Support if you need help interpreting the error message.

Late Snapshot¶

Available only as a global alert.

Sends an alert if a snapshot has failed to complete before the next snapshot is scheduled to begin. Check the job log in the Ops Manager Admin interface for any obvious errors.

Bad Clustershot Count is¶

Sends an alert if Ops Manager fails a consecutive number of times to successfully take a cluster snapshot. Ops Manager sends notification when the number exceeds the number specified in the alert configuration.

The alert text should contain the reason for the problem. Common problems include the following:

There was no reachable mongos. To resolve this issue, ensure that there is at least one mongos showing on the Ops Manager Deployment page.
The balancer could not be stopped. To resolve this issue, check the log files for the first config server to determine why the balancer will not stop.
Could not insert a token in one or more shards. To resolve this issue, ensure connectivity between the Backup Agent and all shards.

Sync Slice Has Not Progressed in¶

Available only as a global alert.

Sends an alert when an initial sync has started but then subsequently stalled. A number of issues can cause this, including:

processes that are down (agents, ingest, backing databases)
network issues
incorrect authentication credentials

Job is busy for¶

Available only as a global alert.

Sends an alert when a backup job has taken longer than the time specified. This could occur if you have an overloaded Backup Daemon or blockstore. Check the corresponding job log for error messages. Contact MongoDB Support if you need help interpreting the error message.

User Alerts¶

These alert conditions apply to the Ops Manager Users.

Added to Group¶: Sends an alert when a new user joins the group.

Removed from Group¶: Sends an alert when a user leaves the group.

Changed Roles¶: Sends an alert when a user’s roles have been changed.

Group Alerts¶

These alert conditions apply to group membership, group security, and the group’s subscription.

Users awaiting approval to join group¶: Sends an alert if there are users who have asked to join the group. A user can ask to join a group when first registering for Ops Manager.

Users do not have two factor authentication enabled¶: Sends an alert if the group has users who have not set up two-factor authentication.

Service suspended due to unpaid invoice(s) more than 30 days old¶

Sends an alert if the group is suspended because of non-payment. A suspended group:

denies users access,
stops backups,
terminates stored snapshots 90 days after suspension.

← Manage Alert Configurations Global Alerts →