Troubleshooting¶

On this page

Getting Started Checklist
Monitoring
Authentication

This document provides advice for troubleshooting problems with MMS. Begin your troubleshooting using the Getting Started Checklist to check for common, easily fixed problems.

For answers to questions not covered here, see FAQ documentation.

Getting Started Checklist¶

To begin troubleshooting, complete these tasks to check for common, easily fixed problems:

Authentication Errors
Check Agent Output or Log
Confirm Only One Agent is Actively Monitoring
Ensure Connectivity Between Agent and Monitored Hosts
Ensure Connectivity Between Agent and MMS Server
Allow Agent to Discover Hosts and Collect Initial Data

Authentication Errors¶

If your MongoDB instances run with authentication enabled, ensure MMS has these credentials. For new hosts, click the Add Host button on the Deployment page then specify credentials for every host with authentication enabled. For hosts already listed in MMS, click the gear icon to the right of a host name on the Deployment page then select Edit Host to provide credentials.

Please consult the :ref:` Authentication Requirements documentation <security-access-control-authentication>` for details about how to use authentication.

Check Agent Output or Log¶

If you continue to encounter problems, check the agent’s output or logs for errors.

Confirm Only One Agent is Actively Monitoring¶

If you run multiple Monitoring Agents, make sure they are all the same version and that only one if actively monitoring. When you upgrade a Monitoring Agent, do not forget to delete any old standby agents.

When you run multiple agents, one runs as the primary agent and the others as standby agents. Standby agents poll MMS periodically to get the configuration but do not send data.

To determine which agent is the primary agent, look at the Last Ping value on the Monitoring Agents page. If there is no Last Ping value for a listed agent, the agent is a standby agent.

See Monitoring FAQs and Add Hosts to MMS Monitoring for more information.

Ensure Connectivity Between Agent and Monitored Hosts¶

Ensure the system running the agent can resolve and connect to the MongoDB instances. To confirm, log into the system where the agent is running and issue a command in the following form:

copy

mongo [hostname]:[port]

Replace [hostname] with the hostname and [port] with the port that the database is listening on.

Ensure Connectivity Between Agent and MMS Server¶

Verify that the Monitoring Agent can connect on TCP port 443 (outbound) to the MMS server (i.e. “mms.mongodb.com”.)

Allow Agent to Discover Hosts and Collect Initial Data¶

Allow the agent to run for 5-10 minutes to allow host discovery and initial data collection.

Installation¶

Why doesn’t the monitoring server startup successfully?¶

Confirm the URI or IP address for the MMS On Prem service is stored correctly in this property in the <install_dir>/conf/conf-mms.properties file:

copy

mongo.mongoUri=<SetToValidUri>
mongo.replicaSet=<ValidRSIfUsed>

If you don’t set this property, MMS will fail while trying to connect to the default 127.0.0.1:27017 URL.

If the URI or IP address of your service changes, you must update the property with the new address. For example, update the address if you deploy on a system without a static IP address, or if you deploy on EC2 without a fixed IP and then restart the EC2 instance.

If the URI or IP address changes, then each user who access the service must also update the address in the URL used to connect and in the client-side monitoring-agent.config files.

If you use the MMS <install_dir>/bin/credentialstool to encrypt the password used in the mongo.mongoUri value, also add the mongo.encryptedCredentials key to the <install_dir>/conf/conf-mms.properties file and set the value for this property to true:

copy

mongo.encryptedCredentials=true

For more details, see MongoDB Access Control Considerations.

Monitoring¶

Alerts¶

For information on creating and managing alerts, see Create an Alert Configuration and Manage Alerts.

Cannot Turn Off Email Notifications¶

There are at least two ways to turn off alert notifications:

Remove the host from your MMS account. Click the Deployment tab and then select the Deployment page. Select the host’s gear icon and select Remove Host.
Disable or delete the alert in MMS. Click the Activity tab then click Alert Settings. To the right of an alert, select the gear icon and select Disable or Delete.

Receive Duplicate Alerts¶

If the notification email list contains multiple email-groups, one or more people may receive multiple notifications of the same alert.

Receive “Host has low open file limits” or “Too many open files” error messages¶

These error messages appear on the Deployment page, under a host’s name. They appear if the number of available connections does not meet an MMS-defined minimum value. These errors are not generated by the mongos instance and, therefore, will not appear in mongos log files.

On a host by host basis, the Monitoring Agent compares the number of open file descriptors and connections to the maximum connections limit. The max open file descriptors ulimit parameter directly affects the number of available server connections. The agent calculates whether or not enough connections exist to meet an MMS-defined minimum value.

In ping documents, for each node and its serverStatus.connections values, if the sum of the current value plus the available value is less than the maxConns configuration value set for a monitored host, the Monitoring Agent will send a Host has low open file limits or Too many open files message to MMS.

Ping documents are data sent by Monitoring Agents to MMS. To view ping documents, click the Deployment page, then click the host’s name, and then click Last Ping.

To prevent this error, we recommend you set ulimit open files to 64000. We also recommend setting the maxConns command in the mongo shell to at least the recommended settings.

See the MongoDB ulimit reference page and the the MongoDB maxConns reference page for details.

Hosts¶

Hosts are not Visible¶

Problems with the Monitoring Agent detecting hosts can be caused by a few factors.

Host not added to MMS: In MMS, select the Deployment tab, then the Deployment page, and then click the Add Host button. In the New Host window, specify the host type, internal hostname, and port. If appropriate, add the database username and password and whether or not MMS should use SSL to connect with your Monitoring Agent. Note it is not necessary to restart your Monitoring Agent when adding (or removing) a host.

Accidental duplicate mongods If you add the host after a crash and restart the Monitoring Agent, you might not see the hostname on the MMS Deployment page. MMS detects the host as a duplicate and suppresses its data. To reset, select the Settings tab, then Group Settings, and then the Reset Duplicates button.

Too many Monitoring Agents installed: Only one Monitoring Agent is needed to monitor all hosts within a single network. You can use a single Monitoring Agent if your hosts exist across multiple data centers and can be discovered by a single agent. Check you have only one Monitoring Agent and remove old agents after upgrading the Monitoring Agent.

A second Monitoring Agent can be set up for redundancy. However, the MMS Monitoring Agent is robust. MMS sends an Agent Down alert only when there are no available Monitoring Agents available. See Monitoring FAQ and Monitoring Architecture for more information.

Cannot Delete a Host¶

In rare cases, the mongod is brought down and the replica set is reconfigured. The down host cannot be deleted and returns an error message, “This host cannot be deleted because it is enabled for backup.” Contact MMS Support for help in deleting these hosts.

For more information on deleted hosts, see Remove Hosts.

Groups¶

For information on creating and managing groups, see Groups.

Cannot Delete a Group¶

Please contact your MMS administrator to remove a company or group from your MMS account.

How to Add a Group¶

Create a group to monitor additional segregated systems or environments for servers, agents, users, and other resources. For example, your deployment might have two or more environments separated by firewalls. In this case, you would need two or more separate MMS groups.

API and shared secret keys are unique to each group. Each group requires its own agent with the appropriate API and shared secret keys. Within each group, the agent needs to be able to connect to all hosts it monitors in the group.

See User and Environment Management for more details.

1

In MMS, select the Users tab.¶

2

Click the Add New Group button.¶

3

Add the group.¶

In the Group Name box, type a name for the new group and then click Add New Group. For security and auditing reasons, you cannot use a name used earlier. Once you name a group, the group’s name cannot be changed.

4

Open the group.¶

To access the new group, select the Group box at the top of the MMS interface, type the group’s name, and select the group. You are the first user added to the new group.

5

Assign hosts.¶

In the Deployment section, click Get Started. Follow the prompts to download the agent, if you have not already, and to assign hosts to the group.

Munin¶

Install and configure the munin-node daemon on the monitored MongoDB server(s) before starting MMS monitoring. The MMS agent README file provides guidelines to install munin-node. However, new versions of Linux, specifically Red Hat Linux (RHEL) 6, can generate error messages. See Configure Hardware Monitoring with munin-node for details about monitoring hardware with munin-node.

Restart munin-node after creating links for changes to take effect.

“No package munin-node is available” Error¶

To correct this error, install the most current version of the Linux repos. Type these commands:

copy

sudo yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

Then type this command to install munin-node and all dependencies:

copy

sudo yum install munin-node

Non-localhost IP Addresses are Blocked¶

By default, munin blocks incoming connections from non-localhost IP addresses. The /var/log/munin/munin-node.log file will display a “Denying connection” error for your non-localhost IP address.

To fix this error, open the munin-node.conf configuration file and comment out these two lines:

copy

allow ^127\.0\.0\.1$
allow ^::1$

Then add this line to the munin-node.conf configuration file with a pattern that matches your subnet:

copy

cidr_allow 0.0.0.0/0

Restart munin-node after editing the configuration file for changes to take effect.

Verifying iostat and Other Plugins/Services Returns “# Unknown service” Error¶

The first step is to confirm there is a problem. Open a telnet session and connect to iostat, iostat_ios, and cpu:

copy

telnet HOSTNAME 4949 <default/required munin port>
fetch iostat
fetch iostat_ios
fetch cpu

The iostat_ios plugin creates the iotime chart, and the cpu plugin creates the cputime chart.

If any of these telnet fetch commands returns an “# Unknown Service” error, create a link to the plugin or service in /etc/munin/plugins/ by typing these commands:

copy

cd /etc/munin/plugins/
sudo ln -s /usr/share/munin/plugins/<service> <service>

Replace <service> with the name of the service that generates the error.

Disk names are not listed by Munin¶

In some cases, Munin will omit disk names with a dash between the name and a numerical prefix, for example, dm-0 or dm-1. There is a documented fix for Munin’s iostat plugin.

Authentication¶

Two-Factor Authentication¶

Missed SMS Authentication Tokens¶

Unfortunately SMS is not a 100% reliable delivery mechanism for messages, especially across international borders. The Google authentication option is 100% reliable. Unless you must use SMS for authentication, use the Google Authenticator application for two-factor authentication.

If you do not receive the SMS authentication tokens:

Refer to the Settings page for more details about using two-factor authentication. This page includes any limitations which may affect SMS delivery times.
Enter the SMS phone number with country code first followed by the area code and the phone number. Also try 011 first followed by the country code, then area code, and then the phone number.

If you do not receive the authentication token in a reasonable amount of time contact MMS Support to rule out SMS message delivery delays.

How to Delete or Reset Two-Factor Authentication¶

Contact your system administrator to remove or reset two-factor authentication on your account.

For administrative information on two-factor authentication, see Manage Two-Factor Authentication for On Prem MMS.

LDAP¶

Cannot Enable LDAP¶

You must enable LDAP before you sign up the first user through the MMS interface, as required in Authentication Requirements. If you cannot enable LDAP because you have created a user through MMS, you must reinstall MMS, being sure to enable LDAP before creating users and hosts. Please see Authentication Requirements.

Forgot to Change MONGODB-CR Error¶

If your MongoDB deployment uses LDAP for authentication, and you find the following error message:

copy

You forget to change "MONGODB-CR" to "LDAP (PLAIN)" since they both
take username/password.

Then make sure that you specified the LDAP (PLAIN) as is the authentication mechanism for both the Monitoring Agent and the Backup Agent. See LDAP.

Backup¶

Logs Display MongodVersionException¶

The MongodVersionException can occur if the Backup Daemon’s host cannot access the internet to download the version or versions of MongoDB required for the backed-up databases. Each database requires a version of MongoDB that matches the database’s version.

If the Daemon runs without access to the internet, you must manually download the required MongoDB versions, as described here:

Go to the MongoDB downloads page and download the appropriate versions for your environment.
Copy the download to the Daemon’s host.

Decompress the download into the directory specified in the mongodb.release.directory setting in the Daemon’s conf-daemon.properties file.

Within the directory specified in the mongodb.release.directory setting, the folder structure for MongoDB should look like the following:

copy

<path-to-mongodb-release-directory>/
|-- mongodb-<platform>
|  |-- THIRD-PARTY-NOTICES
|  |-- README
|  |-- GNU-AGPL-3.0
|  |-- bin
|  |  |-- bsondump
|  |  |-- mongo
|  |  |-- mongod
|  |  |-- mongodump
|  |  |-- mongoexport
|  |  |-- mongofiles
|  |  |-- mongoimport
|  |  |-- mongooplog
|  |  |-- mongoperf
|  |  |-- mongorestore
|  |  |-- mongos
|  |  |-- mongostat
|  |  |-- mongotop

System¶

Logs Display OutOfMemoryError¶

If your logs display OutOfMemoryError, ensure you are running with sufficient ulimits and RAM.

Increase Ulimits¶

For the recommended ulimit setting, see the FAQ on Receive “Host has low open file limits” or “Too many open files” error messages.

MMS infers the host’s ulimit setting using the total number of available and current connections. For more information about ulimits in MongoDB, see the UNIX ulimit Settings reference page.

Ensure Sufficient RAM for All Components¶

Ensure that each server has enough RAM for the components it runs. If a server runs multiple components, its RAM must be at least the sum of the required amount of RAM for each component.

For the RAM requirements for the MMS Application Server and, separately, for the MMS Backup Daemon Server, see On Prem MMS Hardware and Software Requirements.

For the RAM requirements for the MMS Application Database and, separately, for the MMS Backup Blockstore Database, see Preparing Backing MongoDB Instances.

← Security Frequently Asked Questions →