Navigation

Configure a Highly Available Ops Manager Application

The Ops Manager Application provides high availability through use of multiple Ops Manager Application servers behind a load balancer and through use of a replica set to host the Ops Manager Application Database.

Considerations

Multiple Ops Manager Application Servers

The Ops Manager Application’s components are stateless between requests. Any Ops Manager Application server can handle requests as long as all the servers read from the same Ops Manager Application Database. If one Ops Manager Application becomes unavailable, another fills requests.

To take advantage of this for high availability, configure a load balancer of your choice to balance between the pool of Ops Manager Application hosts. In Ops Manager, set the URL to Access Ops Manager property to the load balancer URL. Set the Load Balancer Remote IP Header property to the HTTP header field the load balancer uses to identify the originating client’s IP address (such as X-Forwarded-For). The Ops Manager Application uses the client’s IP address for auditing, logging, and whitelisting for the API.

After the load balancer is configured and started, you cannot log in to the Ops Manager Application from its individual host URLs.

Example

If you have two Ops Manager hosts serving the following URLs:

  • ops1.example.com
  • ops2.example.com

and put them behind a load balancer at the following URL:

  • opsmanager.example.com

After you configure and start that load balancer, you cannot log in to ops1.example.com. You can only log in to opsmanager.example.com.

Replica Set for the Ops Manager Application Database

Deploy a replica set rather than a standalone to host the Ops Manager Application Database. Replica sets have automatic failover if the primary becomes unavailable.

If the replica set has members in multiple facilities, ensure that a single facility has enough votes to elect a primary if needed. Choose the facility that hosts the core application systems. Place a majority of voting members and all the members that can become primary in this facility. Otherwise, network partitions could prevent the set from being able to form a majority. For details on how replica sets elect primaries, see Replica Set Elections.

You can back up the replica set using file system snapshots. File system snapshots use system-level tools to create copies of the device that holds replica set’s data files.

To deploy the replica set that hosts the Ops Manager Application Database, see backing MongoDB instance.

The gen.key File

gen.key file is a 24-byte binary file used to encrypt and decrypt Ops Manager’s backing databases and user credentials. An identical gen.key file must be stored on every server that is part of a highly available Ops Manager deployment.

The gen.key file can be generated automatically or manually.

To have Ops Manager generate the file:
Start one Ops Manager server. Ops Manager will create a gen.key file if none exists.
To create the file manually:

Generate a 24-byte binary file.

Example

The following creates the gen.key file using openssl:

openssl rand 24 > /<keyPath>/gen.key

Protect the gen.key file like any sensitive file. Change the owner to the user running Ops Manager and set the file permission to read and write for the owner only.

Once you have the gen.key file (either created automatically or manually), before starting the other Ops Manager servers, copy the file to the appropriate directory on the current server and to the appropriate directory on the other Ops Manager servers:

  • /etc/mongodb-mms/ for RPM or Ubuntu installations
  • <installPath>/.mongodb-mms/ for an archive (.tar) file installations
  • <installPath>\MMSData\Secrets for Microsoft Windows Server installations

Important

  • Any shared storage resource that stores the gen.key file should be configured for high availability so as not to introduce a potential single point of failure.
  • Any Ops Manager server that does not have the gen.key file installed cannot connect to the backing databases and become part of an HA Ops Manager instance.
  • Once you have generated the gen.key for your Ops Manager instance on the first Ops Manager server, back up the gen.key file to a secure location.

Prerequisites

Deploy the replica set that serves the Ops Manager Application Database. To deploy a replica set, see Deploy a Replica Set in the MongoDB manual.

Procedure

The following procedure assumes you generated the first gen.key using one of the Ops Manager Application hosts. If you instead create your own gen.key, distribute it to the Ops Manager hosts before starting any of the Ops Manager Applications.

Important

The load balancer placed in front of the Ops Manager Application servers must not return cached content. The load balancer must have caching disabled.

To configure multiple Ops Manager Applications with load balancing:

1

Configure a load balancer with the pool of Ops Manager Application hosts.

Configure the load balancer to perform a health check on each Ops Manager health API endpoint:

http://<OpsManagerHost>:<OpsManagerPort>/monitor/health

Ops Manager responds with one of two HTTP codes:

HTTP Status Code Health Status
200 Ops Manager host and application database appear healthy.
500

Ops Manager host or application database appear unhealthy.

See also

If this endpoint returns HTTP 500 often, review the Troubleshooting section.

The load balancer must not return cached content.

2

Configure Ops Manager to use the load balancer.

  1. In Ops Manager, click Admin, then the General tab, and then Ops Manager Config.
  2. Set the URL to Access Ops Manager property to point to the load balancer URL.
  3. Set the Load Balancer Remote IP Header property to the name of the HTTP header field the load balancer uses to identify the client’s IP address.
3

Update each Ops Manager Application host with the replication hosts information.

On each host, edit the conf-mms.properties file to set the mongo.mongoUri property to the connection string of the Ops Manager Application Database. You must specify at least 3 hosts in the mongo.mongoUri connection string.

mongo.mongoUri=mongodb://<mms0.example.net>:<27017>,<mms1.example.net>:<27017>,<mms2.example.net>:<27017>/?maxPoolSize=100
4

Change the Ops Manager URL to the Load Balancer URL in all Agent configuration files.

Complete all of the following steps on each Ops Manager host:

  1. Edit the Automation Agent configuration file.

    The location of the Automation Agent configuration file depends on your platform:

    Platform Installation Method Default Config File Path
    RHEL, CentOS, Amazon Linux and Ubuntu package manager /etc/mongodb-mms/automation-agent.config
    macOS or other Linux distributions tar /path/to/install/local.config
    Windows msi C:\MMSData\Automation\automation-agent.config
  2. Change the mmsBaseUrl property to point to the load balancer.

    mmsBaseUrl=<LOAD-BALANCER-URL>:<PORT>
    

Optional Steps when Automation does not manage Agents

Perform the following steps only when Automation does not manage the Monitoring and Backup Agents.

  1. Edit the Monitoring Agent configuration file.

    The location of the Monitoring Agent configuration file depends on your platform:

    Platform Installation Method Config File Path
    RHEL, CentOS, Amazon Linux and Ubuntu package manager /etc/mongodb-mms/monitoring-agent.config
    macOS or other Linux distributions tar <install-directory>/monitoring-agent.config
    Windows msi C:\MMSData\Monitoring\monitoring-agent.config
  2. Change the mmsBaseUrl property to point to the load balancer.

    mmsBaseUrl=<LOAD-BALANCER-URL>:<PORT>
    
  3. Edit the Backup Agent configuration file.

    The location of the Backup Agent configuration file depends on your platform:

    Platform Installation Method Config File Path
    RHEL, CentOS, Amazon Linux and Ubuntu package manager /etc/mongodb-mms/backup-agent.config
    macOS or other Linux distributions tar /path/to/install/local.config
    Windows msi C:\MMSData\Backup\local.config
  4. Change the mothership property to point to the load balancer.

    mothership=<LOAD-BALANCER-URL>:<PORT>
    
5

Start one of the Ops Manager Applications.

Example

If you installed the Ops Manager Application with an rpm or deb package, issue the following:

service mongodb-mms start
6

Copy the gen.key file to each Ops Manager host.

The gen.key file is located in /etc/mongodb-mms/ for installations from a package manager and in ${HOME}/.mongodb-mms/ for installations from an archive.

Copy the gen.key file from the running Ops Manager Application’s host to the appropriate directory on the other Ops Manager Application hosts.

9

Start the remaining Ops Manager Applications.

Additional Information

For information on making Ops Manager Backup highly available, see Configure a Highly Available Ops Manager Backup Service.