Navigation

Restore a Sharded Cluster from a Snapshot

When you restore a cluster from a snapshot, Ops Manager provides you with restore files for the selected restore point.

To learn about the restore process, see Restore Overview.

Changed in Ops Manager 3.6: Point-in-Time Restores

Prior to 3.6, the Backup Daemon created the complete point- in-time restore on its host. With 3.6, you download a client-side tool along with your snapshot. This tool downloads and applies the oplog to a snapshot on your client system. This reduces network and storage needs for your Ops Manager deployment.

Considerations

Review change to BinData BSON sub-type

The BSON specification changed the default subtype for the BSON binary datatype (BinData) from 2 to 0. Some binary data stored in a snapshot may be BinData subtype 2. The Backup automatically detects and converts snapshot data in BinData subtype 2 to BinData subtype 0. If your application code expects BinData subtype 2, you must update your application code to work with BinData subtype 0.

See also

The notes on the BSON specification explain the particular specifics of this change.

Restore using settings given in restoreInfo.txt

The backup restore file includes a metadata file named restoreInfo.txt. This file captures the options the database used when the snapshot was taken. The database must be run with the listed options after it has been restored. This file contains:

  • Group name

  • Replica Set name

  • Cluster ID (if applicable)

  • Snapshot timestamp (as Timestamp at UTC)

  • Restore timestamp (as a BSON Timestamp at UTC)

  • Last Oplog applied (as a BSON Timestamp at UTC)

  • MongoDB version

  • Storage engine type

  • mongod startup options used on the database when the snapshot was taken

  • Encryption (Only appears if encryption is enabled on the snapshot)

  • Master Key UUID (Only appears if encryption is enabled on the snapshot)

    If restoring from an encrypted backup, you must have a certificate provisioned for this Master Key.

Snapshots when Agent Cannot Stop Balancer

Ops Manager displays a warning next to cluster snapshots taken while the balancer is enabled. If you restore from such a snapshot, you run the risk of lost or orphaned data. For more information, see Snapshots when Agent Cannot Stop Balancer.

Backup Considerations for FCV 4.2

All FCV 4.2 databases must fulfill the appropriate backup considerations.

Prerequisites

Restore from Encrypted Backup Requires Same Master Key

To restore from an encrypted backup, you need the same master key used to encrypt the backup and either the same certificate as is on the Backup Daemon host or a new certificate provisioned with that key from the KMIP host.

If the snapshot is encrypted, the restore panel displays the KMIP master key id and the KMIP server information. You can also find the information when you view the snapshot itself as well as in the restoreInfo.txt file.

Disable Client Requests to MongoDB during Restore

You must ensure that the MongoDB deployment does not receive client requests during restoration. You must either:

  • Restore to new systems with new hostnames and reconfigure your application code once the new deployment is running, or
  • Ensure that the MongoDB deployment will not receive client requests while you restore data.

Restore a Snapshot

To have Ops Manager automatically restore the snapshot:

1

Click Backup, then the Overview tab.

2

Click the deployment, then click Restore or Download.

3

Select the restore point.

  1. Choose the point from which you want to restore your backup.

    Restore Type Description Action
    Snapshot Allows you to choose one stored snapshot. Select an existing snapshot to restore.
    Point In Time

    Allows you to choose a date and time as your restore time objective for your snapshot. By default, the Oplog Store stores 24 hours of data.

    Example

    If you select 12:00, the last operation in the restore is 11:59:59 or earlier.

    Important

    • If you are restoring a sharded cluster that runs FCV of 4.0 or earlier, you must enable cluster checkpoints to perform a PIT restore on a sharded cluster.

      If no checkpoints that include your date and time are available, Ops Manager asks you to choose another point in time.

    • You cannot perform a PIT restore that covers any time prior to the latest backup resync. For the conditions that cause a resync, see Resync a Backup.

    Select a Date and Time.
  2. Click Next.

  3. If you are restoring a sharded cluster that runs FCV of 4.0 or earlier and you chose Point In Time:

    1. A list of Checkpoints closest to the time you selected appears.
    2. To start your point in time restore, you may:
      • Choose one of the listed checkpoints, or
      • Click Choose another point in time to remove the list of checkpoints and select another date and time from the menus.
4

Choose to restore the files to another cluster.

  1. Click Choose Cluster to Restore to.

  2. Complete the following fields:

    Field Action
    Project Select a project to which you want to restore the snapshot.
    Cluster to Restore to

    Select a cluster to which you want to restore the snapshot.

    Ops Manager must manage the target sharded cluster.

    Warning

    Automation removes all existing data from the cluster. All backup data and snapshots for the existing cluster are preserved.

  1. Click Restore.

    Ops Manager notes how much storage space the restore requires in its console.

5

Click Restore.

1

Click Backup, then the Overview tab.

2

Click the deployment, then click Restore or Download.

3

Select the restore point.

  1. Choose the point from which you want to restore your backup.

    Restore Type Description Action
    Snapshot Allows you to choose one stored snapshot. Select an existing snapshot to restore.
    Point In Time

    Creates a custom snapshot that includes all operations up to but not including the selected time. By default, the Oplog Store stores 24 hours of data.

    Example

    If you select 12:00, the last operation in the restore is 11:59:59 or earlier.

    Important

    You cannot perform a PIT restore that covers any time prior to the latest backup resync. For the conditions that cause a resync, see Resync a Backup.

    Select a Date and Time.
    Oplog Timestamp

    Creates a custom snapshot that includes all operations up to and including the entered Oplog timestamp. The Oplog Timestamp is represented as two fields:

    Timestamp Timestamp in the number of seconds that have elapsed since the UNIX epoch
    Increment Order of operation applied in that second as a 32-bit ordinal.

    Type an Oplog Timestamp and Increment.

    Run a query against local.oplog.rs on your replica set to find the desired timestamp.

  2. Click Next.

  3. If you are restoring a sharded cluster that runs FCV of 4.0 or earlier and you chose Point In Time:

    1. A list of Checkpoints closest to the time you selected appears.
    2. To start your point in time restore, you may:
      • Choose one of the listed checkpoints, or
      • Click Choose another point in time to remove the list of checkpoints and select another date and time from the menus.
    3. Once you have selected a checkpoint, apply the oplog to this snapshot to bring your snapshot to the date and time you selected. The oplog is applied for all operations up to but not including the selected time.
4

Click Download to restore the files manually.

5

Configure the snapshot download.

  1. Configure the following download options:

    Pull Restore Usage Limit Select how many times the link can be used. If you select No Limit, the link is re-usable until it expires.
    Restore Link Expiration (in hours) Select the number of hours until the link expires. The default value is 1. The maximum value is the number of hours until the selected snapshot expires.
  2. Click Finalize Request.

  3. If you use 2FA, Ops Manager prompts you for your 2FA code. Enter your 2FA code, then click Finalize Request.

6

Retrieve the snapshots.

Ops Manager creates links to the snapshot. By default, these links are available for an hour and can be used just once.

To download the snapshots:

  1. If you closed the restore panel, click Backup, then Restore History.
  2. When the restore job completes, click (get link) for each shard and for one of the config servers appears.
  3. Click:
    • The copy button to the right of the link to copy the link to use it later, or
    • Download to download the snapshot immediately.

Extra step for point-in-time restores

For point-in-time and oplog timestamp restores, additional instructions are shown. The final step shows the full command you must run using the mongodb-backup-restore-util. It includes all of the necessary options to ensure a full restore.

Select and copy the mongodb-backup-restore-util command provided under Run Binary with PIT Options.

7

Restore the snapshot data files to the destination host.

Extract the snapshot archive for the config server and for each shard to a temporary location.

Example

tar -xvf <backupSnapshot>.tar.gz
mv <backupSnapshot> <temp-database-path>
8

Run the MongoDB Backup Restore Utility (Point-in-Time Restore Only).

  1. Download the MongoDB Backup Restore Utility to your host.

    Note

    If you closed the restore panel, click Backup, then More and then Download MongoDB Backup Restore Utility.

  2. Start a mongod instance without authentication enabled using the extracted snapshot directory as the data directory.

    Example

    mongod --port <port number> \
      --dbpath <temp-database-path> \
      --setParameter ttlMonitorEnabled=false
    

    Warning

    The MongoDB Backup Restore Utility doesn’t support authentication, so you can’t start this temporary database with authentication.

  3. Run the MongoDB Backup Restore Utility on your destination host. Run it once for the config server and each shard.

    Pre-configured mongodb-backup-restore-util command

    Ops Manager provides the mongodb-backup-restore-util with the appropriate options for your restore on the restore panel under Run Binary with PIT Options.

    You should copy the mongodb-backup-restore-util command provided in the Ops Manager Application.

    ./mongodb-backup-restore-util --https --host <targetHost> \
      --port <targetPort> \
      --opStart <opLogStartTimeStamp> \
      --opEnd <opLogEndTimeStamp> \
      --logFile <logPath> \
      --oplogSourceAddr <oplogSourceAddr> \
      --apiKey <apiKey> \
      --groupId <groupId> \
      --rsId <rsId> \
      --whitelist <database1.collection1, database2, etc.> \
      --blacklist <database1.collection1, database2, etc.> \
      --seedReplSetMember \
      --oplogSizeMB <size> \
      --seedTargetPort <port> \
      --ssl \
      --sslCAFile </path/to/ca.pem> \
      --sslPEMKeyFile </path/to/pemkey.pem>
      --sslClientCertificateSubject <distinguishedName> \
      --sslRequireValidServerCertificates <true|false> \
      --sslServerClientCertificate </path/to/client.pem> \
      --sslServerClientCertificatePassword <password> \
      --sslRequireValidMMSBackupServerCertificate <true|false> \
      --sslTrustedMMSBackupServerCertificate </path/to/mms-certs.pem> \
      --httpProxy <proxyURL>
    

    The mongodb-backup-restore-util command uses the following options:

    Option Necessity Description check circle icon
    --host Required Provide the hostname, FQDN, IPv4 address, or IPv6 address for the host that serves the mongod to which the oplog should be applied. If you copied the mongodb-backup-restore-util command provided in the Ops Manager Application, this field is pre-configured. check circle icon
    --port Required Provide the port for the host that serves the mongod to which the oplog should be applied. check circle icon
    --opStart Required

    Provide the BSON timestamp for the first oplog entry you want to include in the restore.

    Note

    This value must be less than or equal to the --opEnd value.

    check circle icon
    --opEnd Required

    Provide the BSON timestamp for the last oplog entry you want to include in the restore.

    Note

    This value cannot be greater than the end of the oplog.

    check circle icon
    --logFile Optional Provide a path, including file name, where the MBRU log is written.  
    --oplogSourceAddr Required Provide the URL for the Ops Manager resource endpoint. check circle icon
    --apiKey Required Provide your Ops Manager Agent API Key. check circle icon
    --groupId Required Provide the group ID. check circle icon
    --rsId Required Provide the replica set ID. check circle icon
    --whitelist Optional Provide a list of databases and/or collections to which you want to limit the restore.  
    --blacklist Optional Provide a list of databases and/or collections to which you want to exclude from the restore.  
    --seedReplSetMember Optional

    Use if you need a replica set member to re-create the oplog collection and seed it with the correct timestamp.

    Requires --oplogSizeMB and --seedTargetPort.

     
    --oplogSizeMB Conditional

    Provide the oplog size in MB.

    Required if --seedReplSetMember is set.

     
    --seedTargetPort Conditional

    Provide the port for the replica set’s primary. This may be different from the ephemeral port used.

    Required if --seedReplSetMember is set.

     
    --ssl Optional

    Use if you need TLS/SSL to apply the oplog to the mongod.

    Requires --sslCAFile and --sslPEMKeyFile.

     
    --sslCAFile Conditional

    Provide the path to the Certificate Authority file.

    Required if --ssl is set.

     
    --sslPEMKeyFile Conditional

    Provide the path to the PEM certificate file.

    Required if --ssl is set.

     
    --sslPEMKeyFilePwd Conditional

    Provide the password for the PEM certificate file specified in --sslPEMKeyFile.

    Required if --ssl is set and that PEM key file is encrypted.

     
    --sslClientCertificateSubject Optional

    Provide the Client Certificate Subject or Distinguished Name (DN) for the target MongoDB process.

    Required if --ssl is set.

     
    --sslRequireValidServerCertificates Optional Set a flag indicating if the tool should validate certificates that the target MongoDB process presents.  
    --sslServerClientCertificate Optional Provide the absolute path to Client Certificate file to use for connecting to the Ops Manager host.  
    --sslServerClientCertificatePassword Conditional

    Provide the absolute path to Client Certificate file password to use for connecting to the Ops Manager host.

    Required if --sslServerClientCertificate is set and that certificate is encrypted.

     
    --sslRequireValidMMSBackupServerCertificate Optional Set a flag indicating if valid certificates are required when contacting the Ops Manager host. Default value is true.  
    --sslTrustedMMSBackupServerCertificate Optional

    Provide the absolute path to the trusted Certificate Authority certificates in PEM format for the Ops Manager host. If this flag is not provided, the system Certificate Authority is used.

    If Ops Manager is using a self-signed SSL certificate, this setting is required.

     
    --httpProxy Optional Provide the URL of an HTTP proxy server the tool can use.  
9

Copy the completed snapshots to restore to other hosts.

  • For the config server, copy the restored config server database to the working database path of each replica set member.
  • For each shard, copy the restored shard database to the working database path of each replica set member.
10

Unmanage the Sharded Cluster.

Before attempting to restore the data manually, remove the sharded cluster from Automation.

11

Restore the Sharded Cluster Manually.

Follow the tutorial from the MongoDB Manual to restore the sharded cluster.

12

Reimport the Sharded Cluster.

To manage the sharded cluster with automation again, import the sharded cluster back into Ops Manager.

13

Start the Sharded Cluster Balancer.

Once a restore completes, the sharded cluster balancer is turned off. To start the balancer:

  1. Click Deployment.
  2. Click ellipsis h icon on the card for your desired sharded cluster.
  3. Click Manager Balancer.
  4. Toggle to Yes.
  5. Click pencil icon to the right of Set the Balancer State.
  6. Toggle to Yes.
  7. Click Save.
  8. Click Review & Deploy to save the changes.

To have Ops Manager automatically restore the snapshot:

1

Click Backup, then the Overview tab.

2

Click the deployment, then click Restore or Download.

3

Select the restore point.

  1. Choose the point from which you want to restore your backup.

    Restore Type Description Action
    Snapshot Allows you to choose one stored snapshot. Select an existing snapshot to restore.
    Point In Time

    Creates a custom snapshot that includes all operations up to but not including the selected time. By default, the Oplog Store stores 24 hours of data.

    Example

    If you select 12:00, the last operation in the restore is 11:59:59 or earlier.

    Important

    You cannot perform a PIT restore that covers any time prior to the latest backup resync. For the conditions that cause a resync, see Resync a Backup.

    Select a Date and Time.
    Oplog Timestamp

    Creates a custom snapshot that includes all operations up to and including the entered Oplog timestamp. The Oplog Timestamp is represented as two fields:

    Timestamp Timestamp in the number of seconds that have elapsed since the UNIX epoch
    Increment Order of operation applied in that second as a 32-bit ordinal.

    Type an Oplog Timestamp and Increment.

    Run a query against local.oplog.rs on your replica set to find the desired timestamp.

  2. Click Next.

4

Choose to restore the files to another cluster.

  1. Click Choose Cluster to Restore to.

  2. Complete the following fields:

    Field Action
    Project Select a project to which you want to restore the snapshot.
    Cluster to Restore to

    Select a cluster to which you want to restore the snapshot.

    Ops Manager must manage the target sharded cluster.

    Warning

    Automation removes all existing data from the cluster. All backup data and snapshots for the existing cluster are preserved.

  1. Click Restore.

    Ops Manager notes how much storage space the restore requires in its UI.

5

Click Restore.

Consider Automatic Restore

This procedure involves a large number steps. Some of these steps have severe security implications. If you don’t need to restore to a deployment that Ops Manager doesn’t manage, consider an automated restore.

To restore a snapshot yourself:

1

Click Backup, then the Overview tab.

2

Click the deployment, then click Restore or Download.

3

Select the Restore Point.

  1. Choose the point from which you want to restore your backup.

    Restore Type Description Action
    Snapshot Allows you to choose one stored snapshot. Select an existing snapshot to restore.
    Point In Time

    Creates a custom snapshot that includes all operations up to but not including the selected time. By default, the Oplog Store stores 24 hours of data.

    Example

    If you select 12:00, the last operation in the restore is 11:59:59 or earlier.

    Important

    You cannot perform a PIT restore that covers any time prior to the latest backup resync. For the conditions that cause a resync, see Resync a Backup.

    Select a Date and Time.
    Oplog Timestamp

    Creates a custom snapshot that includes all operations up to and including the entered Oplog timestamp. The Oplog Timestamp is represented as two fields:

    Timestamp Timestamp in the number of seconds that have elapsed since the UNIX epoch
    Increment Order of operation applied in that second as a 32-bit ordinal.

    Type an Oplog Timestamp and Increment.

    Run a query against local.oplog.rs on your replica set to find the desired timestamp.

  2. Click Next.

4

Click Download to Restore the Files Manually.

5

Configure the snapshot download.

  1. Configure the following download options:

    Pull Restore Usage Limit Select how many times the link can be used. If you select No Limit, the link is re-usable until it expires.
    Restore Link Expiration (in hours) Select the number of hours until the link expires. The default value is 1. The maximum value is the number of hours until the selected snapshot expires.
  2. Click Finalize Request.

  3. If you use 2FA, Ops Manager prompts you for your 2FA code. Enter your 2FA code, then click Finalize Request.

6

Retrieve the Snapshots.

Ops Manager creates links to the snapshot. By default, these links are available for an hour and can be used just once.

To download the snapshots:

  1. If you closed the restore panel, click Backup, then Restore History.
  2. When the restore job completes, click (get link) for each shard and for one of the config servers appears.
  3. Click:
    • The copy button to the right of the link to copy the link to use it later, or
    • Download to download the snapshot immediately.
7

Restore the Snapshot Data Files to the Destination Host.

Extract the snapshot archive for the config server and for each shard to a temporary location.

Example

The following commands use </path/to/snapshot/> as a temporary path:

tar -xvf <backupSnapshot>.tar.gz
mv <backupSnapshot> </path/to/snapshot>
8

Unmanage the Sharded Cluster.

Before attempting to restore the data manually, remove the sharded cluster from Automation.

Note

Steps 8 to 16 use the CSRS files downloaded in Step 7.

9

Shut Down Running MongoDB Processes.

If restoring to an existing cluster, shut down the mongod or mongos process on the target host. Using a mongo shell, connect to a host running:

mongos

Run db.shutdownServer() from the admin database:

db.getSiblingDB("admin").shutdownServer()
mongod

Run db.isMaster():

ismaster returns Member is To Shut Down
false secondary

Run db.shutdownServer() from the admin database.

db.getSiblingDB("admin").shutdownServer()
true primary
  1. Use rs.status() to identify the other members.
  2. Connect to each secondary and shut down their mongod processes first.
  3. After the primary detects that a majority of members are offline, it steps down.
  4. After the primary steps down (db.isMaster returns ismaster: false), shut down the primary’s mongod.
10

Copy the Completed Snapshots to Restore to Other Hosts.

  • For the config server, copy the restored config server database to the working database path of each replica set member.
  • For each shard, copy the restored shard database to the working database path of each replica set member.
11

Drop the Minimum Valid Timestamp.

Issue the following command:

db.getSiblingDB("local").replset.minvalid.drop()
12

Verify Hardware and Software Requirements.

Storage Capacity The target host hardware needs enough free storage space for the restored data. If you want to keep any existing sharded cluster data on this host, make sure the host has enough free space for both data sets.
MongoDB Version The target host and source host must run the same MongoDB Server version. To check the MongoDB version, run mongod --version from a terminal or shell.

To learn more about installation, see /installation.

13

Prepare Data and Log Directories on the Target Host.

  1. Create a directory tree on the target host for the restored database files and logs.

    mkdir -p </path/to/datafiles>/log
    
  2. Grant the user that runs the mongod read, write, and execute permissions for everything in that directory.

    chown -R mongodb:mongodb </path/to/datafiles>
    chmod -R 770 </path/to/datafiles>
    
14

Create Configuration File.

  1. Create a mongod configuration file in your database directory using your preferred text editor.

    vi </path/to/datafiles>/mongod.conf
    

    Note

    If you have access to the original configuration file for the mongod, you can copy it to your database directory on the target host instead.

  2. Grant the user that runs the mongod read and write permissions on your configuration file.

    chown mongodb:mongodb </path/to/datafiles>/mongod.conf
    chmod 644 </path/to/datafiles>/mongod.conf
    
  3. Modify your configuration as you require for your deployment.

    Setting Required Value
    storage.dbPath Path to your data directory
    systemLog.path Path to your log directory
    net.bindIp IP address of the host machine
    replication.replSetName Same value across each member in any given replica set.
    sharding.clusterRole Same value across each member in any given replica set.
15

Restore the CSRS Primary mongod Data Files.

  1. Copy the mongod data files from the backup data location to the data directory you created:

    cp -a </path/to/snapshot/> </path/to/datafiles>
    

    The -a option recursively copies the contents of the source path to the destination path while preserving folder and file permissions.

  2. Open your replica set configuration file in your preferred text editor.

  3. Comment out or omit the following configuration file settings:

    #replication
    #  replSetName: <myCSRSName>
    #sharding
    #  clusterRole: configsvr
    
  4. Start the mongod, specifying:

    • The --config option and the full path to the configuration file, and

    • The disableLogicalSessionCacheRefresh server parameter.

      ./mongod --config </path/to/datafiles>/mongod.conf \
               --setParameter disableLogicalSessionCacheRefresh=true
      

      If you have mongod configured to run as a system service, start it using the recommended process for your platform’s service manager.

  5. After the mongod starts, connect to it using the mongo shell.

16
17

Add a New Replica Set Configuration.

Insert the following document into the system.replset collection in the local database. Change <replaceMeWithTheCSRSName> to the name of your Replica Set and <port> to the port of your Replica Set.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
db.getSiblingDB("local").system.replset.insert( {
  "_id" : "<replaceMeWithTheCSRSName>",
  "version" : NumberInt(1),
  "configsvr" : true,
  "protocolVersion" : NumberInt(1),
  "members" : [
    {
      "_id" : NumberInt(0),
      "host" : "localhost:<port>"
    }
  ],
  "settings" : {

  }
} )

A successful response should look like this:

WriteResult( { "nInserted" : 1 } )
18

Insert the Minimum Valid Timestamp.

Issue the following command:

db.getSiblingDB("local").replset.minvalid.insert( {
  "_id" : ObjectId(),
  "t" : NumberLong(-1),
  "ts" : Timestamp(0,1)
} );
19

Set the Restore Point to the Restore Timestamp value from the restoreInfo file.

Set the oplogTruncateAfterPoint document to the Restore timestamp given in the restoreInfo.txt file.

truncateAfterPoint = Timestamp(restoreTS.getTime(), restoreTS.getInc())
db.getSiblingDB("local").replset.oplogTruncateAfterPoint.insert( {
   "_id": "oplogTruncateAfterPoint",
   "oplogTruncateAfterPoint": truncateAfterPoint
} )

A successful response should look like this:

WriteResult( { "nInserted" : 1 } )

Note

Each member has its own restoreInfo.txt file, but the restoreTs value should be the same in each file.

20

Restart as a Single-Node Replica Set to Recover the Oplog.

Start the mongod. The mongod replays the oplog up to the Restore timestamp.

./mongod --dbpath </path/to/datafiles> \
         --port <ephemeralPort> \
         --replSet <replaceMeWithTheCSRSName> \
         --configsvr

Important

You must not set <ephemeralPort> in this command to the same <port> you set in Step 14.

21

Stop the Temporary Single-Node Config Server Replica Set.

./mongod --dbpath </path/to/datafiles> \
         --port <ephemeralPort> \
         --shutdown
22

Run the MongoDB Backup Restore Utility (Point-in-Time Restore Only).

  1. Download the MongoDB Backup Restore Utility to your host.

    Note

    If you closed the restore panel, click Backup, then More and then Download MongoDB Backup Restore Utility.

  2. Start a mongod instance without authentication enabled using the extracted snapshot directory as the data directory.

    Example

    mongod --port <port number> \
           --dbpath <temp-database-path> \
           --setParameter ttlMonitorEnabled=false
    

    Warning

    The MongoDB Backup Restore Utility doesn’t support authentication, so you can’t start this temporary database with authentication.

  3. Run the MongoDB Backup Restore Utility on your destination host. Run it once for the replica set.

    Pre-configured mongodb-backup-restore-util command

    Ops Manager provides the mongodb-backup-restore-util with the appropriate options for your restore on the restore panel under Run Binary with PIT Options.

    You should copy the mongodb-backup-restore-util command provided in the Ops Manager Application.

    ./mongodb-backup-restore-util --https --host <targetHost> \
      --port <targetPort> \
      --opStart <opLogStartTimeStamp> \
      --opEnd <opLogEndTimeStamp> \
      --logFile <logPath> \
      --oplogSourceAddr <oplogSourceAddr> \
      --apiKey <apiKey> \
      --groupId <groupId> \
      --rsId <rsId> \
      --whitelist <database1.collection1, database2, etc.> \
      --blacklist <database1.collection1, database2, etc.> \
      --seedReplSetMember \
      --oplogSizeMB <size> \
      --seedTargetPort <port> \
      --ssl \
      --sslCAFile </path/to/ca.pem> \
      --sslPEMKeyFile </path/to/pemkey.pem>
      --sslClientCertificateSubject <distinguishedName> \
      --sslRequireValidServerCertificates <true|false> \
      --sslServerClientCertificate </path/to/client.pem> \
      --sslServerClientCertificatePassword <password> \
      --sslRequireValidMMSBackupServerCertificate <true|false> \
      --sslTrustedMMSBackupServerCertificate </path/to/mms-certs.pem> \
      --httpProxy <proxyURL>
    

    The mongodb-backup-restore-util command uses the following options:

    Option Necessity Description check circle icon
    --host Required Provide the hostname, FQDN, IPv4 address, or IPv6 address for the host that serves the mongod to which the oplog should be applied. If you copied the mongodb-backup-restore-util command provided in the Ops Manager Application, this field is pre-configured. check circle icon
    --port Required Provide the port for the host that serves the mongod to which the oplog should be applied. check circle icon
    --opStart Required

    Provide the BSON timestamp for the first oplog entry you want to include in the restore.

    Note

    This value must be less than or equal to the --opEnd value.

    check circle icon
    --opEnd Required

    Provide the BSON timestamp for the last oplog entry you want to include in the restore.

    Note

    This value cannot be greater than the end of the oplog.

    check circle icon
    --logFile Optional Provide a path, including file name, where the MBRU log is written.  
    --oplogSourceAddr Required Provide the URL for the Ops Manager resource endpoint. check circle icon
    --apiKey Required Provide your Ops Manager Agent API Key. check circle icon
    --groupId Required Provide the group ID. check circle icon
    --rsId Required Provide the replica set ID. check circle icon
    --whitelist Optional Provide a list of databases and/or collections to which you want to limit the restore.  
    --blacklist Optional Provide a list of databases and/or collections to which you want to exclude from the restore.  
    --seedReplSetMember Optional

    Use if you need a replica set member to re-create the oplog collection and seed it with the correct timestamp.

    Requires --oplogSizeMB and --seedTargetPort.

     
    --oplogSizeMB Conditional

    Provide the oplog size in MB.

    Required if --seedReplSetMember is set.

     
    --seedTargetPort Conditional

    Provide the port for the replica set’s primary. This may be different from the ephemeral port used.

    Required if --seedReplSetMember is set.

     
    --ssl Optional

    Use if you need TLS/SSL to apply the oplog to the mongod.

    Requires --sslCAFile and --sslPEMKeyFile.

     
    --sslCAFile Conditional

    Provide the path to the Certificate Authority file.

    Required if --ssl is set.

     
    --sslPEMKeyFile Conditional

    Provide the path to the PEM certificate file.

    Required if --ssl is set.

     
    --sslPEMKeyFilePwd Conditional

    Provide the password for the PEM certificate file specified in --sslPEMKeyFile.

    Required if --ssl is set and that PEM key file is encrypted.

     
    --sslClientCertificateSubject Optional

    Provide the Client Certificate Subject or Distinguished Name (DN) for the target MongoDB process.

    Required if --ssl is set.

     
    --sslRequireValidServerCertificates Optional Set a flag indicating if the tool should validate certificates that the target MongoDB process presents.  
    --sslServerClientCertificate Optional Provide the absolute path to Client Certificate file to use for connecting to the Ops Manager host.  
    --sslServerClientCertificatePassword Conditional

    Provide the absolute path to Client Certificate file password to use for connecting to the Ops Manager host.

    Required if --sslServerClientCertificate is set and that certificate is encrypted.

     
    --sslRequireValidMMSBackupServerCertificate Optional Set a flag indicating if valid certificates are required when contacting the Ops Manager host. Default value is true.  
    --sslTrustedMMSBackupServerCertificate Optional

    Provide the absolute path to the trusted Certificate Authority certificates in PEM format for the Ops Manager host. If this flag is not provided, the system Certificate Authority is used.

    If Ops Manager is using a self-signed SSL certificate, this setting is required.

     
    --httpProxy Optional Provide the URL of an HTTP proxy server the tool can use.  

    check circle icon means that if you copied the mongodb-backup-restore-util command provided in Ops Manager Application, this field is pre-configured.

  4. Issue the following command:

    ./mongod --dbpath </path/to/datafiles> \
             --port <ephemeralPort> \
             --shutdown
    
23

Restart as a Standalone to Recover the Oplog.

Start the mongod with the following setParameter options set to true:

  • recoverFromOplogAsStandalone
  • takeUnstableCheckpointOnShutdown

The mongod replays the oplog up to the Restore timestamp.

./mongod --dbpath </path/to/datafiles> --port <ephemeralPort> \
         --setParameter recoverFromOplogAsStandalone=true \
         --setParameter takeUnstableCheckpointOnShutdown=true
24

Stop the Temporary CSRS Standalone Used to Recover the Oplog.

Issue the following command:

./mongod --dbpath </path/to/datafiles> \
         --port <ephemeralPort> \
         --shutdown
25

Restart as a Standalone to Clear Old Settings.

Start the mongod.

./mongod --dbpath </path/to/datafiles> --port <ephemeralPort>
26

Clean Documents in Config Database that Reference the Previous Sharded Cluster Configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
db.getSiblingDB("config").mongos.remove( {} );
db.getSiblingDB("config").lockpings.remove( {} );
db.getSiblingDB("config").databases.updateMany( {"primary": SOURCE_SHARD_0_NAME},{"$set" : {"primary": DEST_SHARD_0_NAME}} );
db.getSiblingDB("config").databases.updateMany( {"primary": SOURCE_SHARD_1_NAME},{"$set" : {"primary": DEST_SHARD_1_NAME}} );
db.getSiblingDB("config").databases.updateMany( {"primary": SOURCE_SHARD_2_NAME},{"$set" : {"primary": DEST_SHARD_2_NAME}} );
db.getSiblingDB("config").chunks.updateMany( {"shard": SOURCE_SHARD_0_NAME},{"$set" : {"shard": DEST_SHARD_0_NAME, "history": []}} );
db.getSiblingDB("config").chunks.updateMany( {"shard": SOURCE_SHARD_1_NAME},{"$set" : {"shard": DEST_SHARD_1_NAME, "history": []}} );
db.getSiblingDB("config").chunks.updateMany( {"shard": SOURCE_SHARD_2_NAME},{"$set" : {"shard": DEST_SHARD_2_NAME, "history": []}} );
db.getSiblingDB("config").collections.updateMany( {"primary": SOURCE_SHARD_0_NAME},{"$set" : {"primary": DEST_SHARD_0_NAME}} );
db.getSiblingDB("config").collections.updateMany( {"primary": SOURCE_SHARD_1_NAME},{"$set" : {"primary": DEST_SHARD_1_NAME}} );
db.getSiblingDB("config").collections.updateMany( {"primary": SOURCE_SHARD_2_NAME},{"$set" : {"primary": DEST_SHARD_2_NAME}} );
db.getSiblingDB("config").shards.remove( {} );
db.getSiblingDB("config").shards.insert( {
  "_id": DEST_SHARD_0_NAME,
  "host": DEST_SHARD_0_NAME + "/" + DEST_SHARD_0_HOSTNAME,
  "State": 1,
} );
db.getSiblingDB("config").shards.insert( {
  "_id": DEST_SHARD_1_NAME,
  "host": DEST_SHARD_1_NAME + "/" + DEST_SHARD_1_HOSTNAME,
  "State": 1,
} );
db.getSiblingDB("config").shards.insert( {
  "_id": DEST_SHARD_2_NAME,
  "host": DEST_SHARD_2_NAME + "/" + DEST_SHARD_2_HOSTNAME,
  "State": 1,
} )

Note

This example covers a three shard cluster. Replace the following values with those in your configuration:

  • SOURCE_SHARD_<X>_NAME
  • <DEST_SHARD_<X>_NAME
  • DEST_SHARD_<X>_HOSTNAME
27

Restart the mongod as a New Single-node Replica Set.

  1. Open the configuration file in your preferred text editor.

  2. Uncomment or add the following configuration file options:

    replication
      replSetName: myNewCSRSName
    sharding
      clusterRole: configsvr
    
  3. To change the replica set name, update the replSetName field with the new name before proceeding.

  4. Start the mongod with the updated configuration file:

    ./mongod --config </path/to/datafiles>/mongod.conf
    

    If you have mongod configured to run as a system service, start it using the recommended process for your platform’s service manager.

  5. After the mongod starts, connect to it using the mongo shell.

28

Initiate the New Replica Set.

Initiate the replica set using rs.initiate() with the default settings.

Once the operation completes, use rs.status() to check that the member has become the primary.

29

Add Additional Replica Set Members.

  1. For each replica set member in the CSRS, start the mongod on its host.

  2. Once you have started up all remaining members of the cluster successfully, connect a mongo shell to the primary replica set member.

  3. From the primary, use the rs.add() method to add each member of the replica set. Include the replica set name as the prefix, followed by the hostname and port of the member’s mongod process:

    rs.add("myNewCSRSName/config2.example.net:<port>")
    rs.add("myNewCSRSName/config3.example.net:<port>")
    
  4. If you want to add the member with specific replica member configuration settings, you can pass a document to rs.add() that defines the member hostname and any members[n] settings your deployment requires.

    rs.add(
     {
       "host" : "myNewCSRSName/config2.example.net:<port>",
       priority: <int>,
       votes: <int>,
       tags: <int>
     }
    )
    
  5. Each new member performs an initial sync to catch up to the primary. Depending on data volume, network, and host performance factors, initial sync might take a while to complete.

  6. The replica set might elect a new primary while you add additional members. You can only run rs.add() from the primary. To identify which member is the current primary, use rs.status().

30

Configure Any Additional Required Replication Settings.

The rs.reconfig() method updates the replica set configuration based on a configuration document passed in as a parameter.

  1. Run reconfig() against the primary member of the replica set.
  2. Reference the original configuration file output of the replica set and apply settings as needed.
31

Remove Replica Set-Related Collections from the local Database.

Note

These steps appear repetitive, but cover shards instead of the CSRS.

Invoke the following commands to remove the previous replica set configuration and other non-oplog, replication-related collections.

db.getSiblingDB("local").replset.minvalid.drop()
db.getSiblingDB("local").replset.oplogTruncateAfterPoint.drop()
db.getSiblingDB("local").replset.election.drop()
db.getSiblingDB("local").system.replset.remove( {} )

A successful response should look like this:

> db.getSiblingDB("local").replset.minvalid.drop()
true
> db.getSiblingDB("local").replset.oplogTruncateAfterPoint.drop()
true
> db.getSiblingDB("local").replset.election.drop()
true
> db.getSiblingDB("local").system.replset.remove( {} )
WriteResult( { "nRemoved" : 1 } )
32

Insert the Minimum Valid Timestamp.

Issue the following command:

db.getSiblingDB("local").replset.minvalid.insert( {
  "_id" : ObjectId(),
  "t" : NumberLong(-1),
  "ts" : Timestamp(0,1)
} );
33

Add a New Replica Set Configuration.

Insert the following document into the system.replset collection in the local database. Change <replaceMeWithTheShardName> to the name of your Replica Set and <port> to the port of your Replica Set.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
db.getSiblingDB("local").system.replset.insert( {
  "_id" : "<replaceMeWithTheShardName>",
  "version" : NumberInt(1),
  "protocolVersion" : NumberInt(1),
  "members" : [
    {
      "_id" : NumberInt(0),
      "host" : "localhost:<port>"
    }
  ],
  "settings" : {

  }
} )

A successful response should look like this:

WriteResult( { "nInserted" : 1 } )
34

Set the Restore Point to the Restore Timestamp value from the restoreInfo file.

Set the oplogTruncateAfterPoint document to the Restore timestamp given in the restoreInfo.txt file.

truncateAfterPoint = Timestamp(restoreTS.getTime(), restoreTS.getInc())
db.getSiblingDB("local").replset.oplogTruncateAfterPoint.insert( {
   "_id": "oplogTruncateAfterPoint",
   "oplogTruncateAfterPoint": truncateAfterPoint
} )

A successful response should look like this:

WriteResult( { "nInserted" : 1 } )
35

Restart as a Single-Node Replica Set to Recover the Oplog.

Start the mongod. The mongod replays the oplog up to the Restore timestamp.

./mongod --dbpath </path/to/datafiles> \
         --port <ephemeralPort> \
         --replSet <replaceMeWithTheShardName>

Important

You must not set <ephemeralPort> in this command to the same <port> you set in Step 14.

36

Stop the Temporary Single-Node Shard Replica Set.

./mongod --dbpath </path/to/datafiles> \
         --port <ephemeralPort> \
         --shutdown
37

Run the MongoDB Backup Restore Utility (Point-in-Time Restore Only).

  1. Download the MongoDB Backup Restore Utility to your host.

    Note

    If you closed the restore panel, click Backup, then More and then Download MongoDB Backup Restore Utility.

  2. Start a mongod instance without authentication enabled using the extracted snapshot directory as the data directory.

    Example

    mongod --port <port number> \
      --dbpath <temp-database-path> \
      --setParameter ttlMonitorEnabled=false
    

    Warning

    The MongoDB Backup Restore Utility doesn’t support authentication, so you can’t start this temporary database with authentication.

  3. Run the MongoDB Backup Restore Utility on your destination host. Run it once for the replica set.

    Pre-configured mongodb-backup-restore-util command

    Ops Manager provides the mongodb-backup-restore-util with the appropriate options for your restore on the restore panel under Run Binary with PIT Options.

    You should copy the mongodb-backup-restore-util command provided in the Ops Manager Application.

    ./mongodb-backup-restore-util --https --host <targetHost> \
      --port <targetPort> \
      --opStart <opLogStartTimeStamp> \
      --opEnd <opLogEndTimeStamp> \
      --logFile <logPath> \
      --oplogSourceAddr <oplogSourceAddr> \
      --apiKey <apiKey> \
      --groupId <groupId> \
      --rsId <rsId> \
      --whitelist <database1.collection1, database2, etc.> \
      --blacklist <database1.collection1, database2, etc.> \
      --seedReplSetMember \
      --oplogSizeMB <size> \
      --seedTargetPort <port> \
      --ssl \
      --sslCAFile </path/to/ca.pem> \
      --sslPEMKeyFile </path/to/pemkey.pem>
      --sslClientCertificateSubject <distinguishedName> \
      --sslRequireValidServerCertificates <true|false> \
      --sslServerClientCertificate </path/to/client.pem> \
      --sslServerClientCertificatePassword <password> \
      --sslRequireValidMMSBackupServerCertificate <true|false> \
      --sslTrustedMMSBackupServerCertificate </path/to/mms-certs.pem> \
      --httpProxy <proxyURL>
    

    The mongodb-backup-restore-util command uses the following options:

    Option Necessity Description check circle icon
    --host Required Provide the hostname, FQDN, IPv4 address, or IPv6 address for the host that serves the mongod to which the oplog should be applied. If you copied the mongodb-backup-restore-util command provided in the Ops Manager Application, this field is pre-configured. check circle icon
    --port Required Provide the port for the host that serves the mongod to which the oplog should be applied. check circle icon
    --opStart Required

    Provide the BSON timestamp for the first oplog entry you want to include in the restore.

    Note

    This value must be less than or equal to the --opEnd value.

    check circle icon
    --opEnd Required

    Provide the BSON timestamp for the last oplog entry you want to include in the restore.

    Note

    This value cannot be greater than the end of the oplog.

    check circle icon
    --logFile Optional Provide a path, including file name, where the MBRU log is written.  
    --oplogSourceAddr Required Provide the URL for the Ops Manager resource endpoint. check circle icon
    --apiKey Required Provide your Ops Manager Agent API Key. check circle icon
    --groupId Required Provide the group ID. check circle icon
    --rsId Required Provide the replica set ID. check circle icon
    --whitelist Optional Provide a list of databases and/or collections to which you want to limit the restore.  
    --blacklist Optional Provide a list of databases and/or collections to which you want to exclude from the restore.  
    --seedReplSetMember Optional

    Use if you need a replica set member to re-create the oplog collection and seed it with the correct timestamp.

    Requires --oplogSizeMB and --seedTargetPort.

     
    --oplogSizeMB Conditional

    Provide the oplog size in MB.

    Required if --seedReplSetMember is set.

     
    --seedTargetPort Conditional

    Provide the port for the replica set’s primary. This may be different from the ephemeral port used.

    Required if --seedReplSetMember is set.

     
    --ssl Optional

    Use if you need TLS/SSL to apply the oplog to the mongod.

    Requires --sslCAFile and --sslPEMKeyFile.

     
    --sslCAFile Conditional

    Provide the path to the Certificate Authority file.

    Required if --ssl is set.

     
    --sslPEMKeyFile Conditional

    Provide the path to the PEM certificate file.

    Required if --ssl is set.

     
    --sslPEMKeyFilePwd Conditional

    Provide the password for the PEM certificate file specified in --sslPEMKeyFile.

    Required if --ssl is set and that PEM key file is encrypted.

     
    --sslClientCertificateSubject Optional

    Provide the Client Certificate Subject or Distinguished Name (DN) for the target MongoDB process.

    Required if --ssl is set.

     
    --sslRequireValidServerCertificates Optional Set a flag indicating if the tool should validate certificates that the target MongoDB process presents.  
    --sslServerClientCertificate Optional Provide the absolute path to Client Certificate file to use for connecting to the Ops Manager host.  
    --sslServerClientCertificatePassword Conditional

    Provide the absolute path to Client Certificate file password to use for connecting to the Ops Manager host.

    Required if --sslServerClientCertificate is set and that certificate is encrypted.

     
    --sslRequireValidMMSBackupServerCertificate Optional Set a flag indicating if valid certificates are required when contacting the Ops Manager host. Default value is true.  
    --sslTrustedMMSBackupServerCertificate Optional

    Provide the absolute path to the trusted Certificate Authority certificates in PEM format for the Ops Manager host. If this flag is not provided, the system Certificate Authority is used.

    If Ops Manager is using a self-signed SSL certificate, this setting is required.

     
    --httpProxy Optional Provide the URL of an HTTP proxy server the tool can use.  

    check circle icon means that if you copied the mongodb-backup-restore-util command provided in Ops Manager Application, this field is pre-configured.

38

Stop the Temporary Shard Standalone.

Issue the following command:

./mongod --dbpath </path/to/datafiles> \
         --port <ephemeralPort> \
         --shutdown
39

Restart as a Standalone to Clear Old Settings.

Start the mongod.

./mongod --dbpath </path/to/datafiles> --port <ephemeralPort>
40

Clean Documents in the Admin and Config Databases.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
db.getSiblingDB("admin").system.version.remove( {"_id": "minOpTimeRecovery"} );

db.getSiblingDB("admin").system.version.updateOne(
  { "_id": "shardIdentity" },
  {
    $set: {
      "shardName": <ShardName>,
      configsvrConnectionString: NEW_CONFIG_NAME + "/" + NEW_CONFIG_HOSTNAME,
      clusterId: <clusterId>
    }
  }
);

db.getSiblingDB("config").cache.collections.drop()
db.getSiblingDB("config").cache.databases.drop()
db.getSiblingDB("config").cache.chunks.config.system.sessions.drop()

Note

Replace the following values with those in your configuration:

  • <ShardName>
  • <clusterId>
  • NEW_CONFIG_NAME
  • NEW_CONFIG_HOSTNAME
41

Stop the Temporary Document Cleaning Standalone.

./mongod --dbpath </path/to/datafiles> \
         --port <ephemeralPort> \
         --shutdown
42

Connect a mongo Shell to the mongod Instance.

From the host running this mongod process, start the mongo shell. To connect to the mongod listening to localhost on port <port>, invoke:

./mongo --port <port>

Depending on your path, you may need to specify the path to the mongo shell.

After the mongod starts accepting connections, continue.

43

Initiate the New Replica Set.

Run rs.initiate() on the replica set:

rs.initiate( {
  _id : <replaceMeWithTheShardName>,
  members: [ {
    _id : 0,
    host : <host:port>
  } ]
} )

MongoDB initiates a set that consists of the current member and that uses the default replica set configuration.

44

Shut Down the New Replica Set.

db.shutdownServer( {} );
45

Reimport the Sharded Cluster.

To manage the sharded cluster with Automation again, import the sharded cluster back into Ops Manager.

47

Restore the Shard Primary mongod Data Files.

  1. Copy the mongod data files from the backup data location to the data directory you created:

    cp -a </path/to/snapshot/> </path/to/datafiles>
    

    The -a option recursively copies the contents of the source path to the destination path while preserving folder and file permissions.

  2. Open your replica set configuration file in your preferred text editor.

  3. Comment out or omit the following configuration file settings:

    #replication
    #  replSetName: <myShardName>
    #sharding
    #  clusterRole: shardsvr
    
  4. Start the mongod, specifying:

    • The --config option and the full path to the configuration file, and
    • The disableLogicalSessionCacheRefresh server parameter.
    ./mongod --config </path/to/datafiles>/mongod.conf \
             --setParameter disableLogicalSessionCacheRefresh=true
    

    If you have mongod configured to run as a system service, start it using the recommended process for your platform’s service manager.

  5. After the mongod starts, connect to it using the mongo shell.

48

Create a Temporary User with the __system Role.

Important

Skip this step if the cluster does not enforce authentication.

Clusters that enforce authentication limit who can change the admin.system.version collection. Clusters limit permission to users with the __system role.

Warning

The __system role allows a user to take any action against any object in the database.

Do not keep this user active beyond the scope of this procedure. This procedure includes instructions for removing the user created in this step.

Consider creating this user with the clientSource authentication restriction configured such that only the specified hosts can authenticate as the privileged user.

  1. Authenticate as a user with either the userAdmin role on the admin database or the userAdminAnyDatabase role:

    db.getSiblingDB("admin").auth("<myUserAdmin>","<replaceMeWithAStrongPassword>")
    
  2. Create a user with the __system role:

    db.getSiblingDB("admin").createUser(
      {
        user: "<myTempSystemUserWithTotalAccess>",
        pwd: "<replaceMeWithAStrongPassword>",
        roles: [ "__system" ]
      }
    )
    

    Make these passwords random, long, and complex. Keep the system secure and prevent or delay malicious access.

  3. Authenticate as the privileged user:

    db.getSiblingDB("admin").auth("<myTempSystemUserWithTotalAccess>","<replaceMeWithAStrongPassword>")
    
49

Restart the mongod as a New Single-node Replica Set.

  1. Shut down the mongod.

  2. Open the configuration file in your preferred text editor.

  3. Uncomment or add the following configuration file options:

    replication
      replSetName: myNewShardName
    sharding
      clusterRole: shardsvr
    
  4. To change the replica set name, update the replSetName field with the new name before proceeding.

  5. Start the mongod with the updated configuration file:

    ./mongod --config </path/to/datafiles>/mongod.conf
    

    If you have mongod configured to run as a system service, start it using the recommended process for your platform’s service manager.

  6. After the mongod starts, connect to it using the mongo shell.

50

Initiate the New Replica Set.

Initiate the replica set using rs.initiate() with the default settings.

Once the operation completes, use rs.status() to check that the member has become the primary.

51

Add Additional Replica Set Members.

  1. For each replica set member in the shard replica set, start the mongod on its host.

  2. Once you have started up all remaining members of the cluster successfully, connect a mongo shell to the primary replica set member.

  3. From the primary, use the rs.add() method to add each member of the replica set. Include the replica set name as the prefix, followed by the hostname and port of the member’s mongod process:

    rs.add("myNewShardName/repl2.example.net:<port>")
    rs.add("myNewShardName/repl3.example.net:<port>")
    
  4. If you want to add the member with specific replica member configuration settings, you can pass a document to rs.add() that defines the member hostname and any members[n] settings your deployment requires.

    rs.add(
     {
       "host" : "myNewShardName/repl2.example.net:<port>",
       priority: <int>,
       votes: <int>,
       tags: <int>
     }
    )
    
  5. Each new member performs an initial sync to catch up to the primary. Depending on data volume, network, and host performance factors, initial sync might take a while to complete.

  6. The replica set might elect a new primary while you add additional members. You can only run rs.add() from the primary. To identify which member is the current primary, use rs.status().

52

Configure Any Additional Required Replication Settings.

The rs.reconfig() method updates the replica set configuration based on a configuration document passed in as a parameter.

  1. Run reconfig() against the primary member of the replica set.
  2. Reference the original configuration file output of the replica set and apply settings as needed.
53

Remove the Temporary Privileged User.

For clusters enforcing authentication, remove the privileged user created earlier in this procedure:

  1. Authenticate as a user with the userAdmin role on the admin database or userAdminAnyDatabase role:

    db.getSiblingDB("admin").auth("<myUserAdmin>","<replaceMeWithAStrongPassword>")
    
  2. Delete the privileged user:

    db.getSiblingDB("admin").removeUser("<myTempSystemUserWithTotalAccess>")
    
54

Restart Each mongos.

Restart each mongos in the cluster.

mongos --config </path/to/config/>mongos.conf

Include all other command line options as required by your deployment.

If the CSRS replica set name or any member hostname changed, update the mongos configuration file setting sharding.configDB with updated configuration server connection string:

sharding:
  configDB: "myNewCSRSName/config1.example.net:<port>,config2.example.net:<port>,config3.example.net:<port>"
55

Verify that You Can Access the Cluster.

  1. Connect a mongo shell to one of the mongos processes for the cluster.

  2. Use sh.status() to check the overall cluster status.

    If sh.status() indicates that the balancer is not running, use sh.startBalancer() to restart the balancer.

  3. To confirm that you can access all shards and they are communicating, insert test data into a temporary sharded collection.

  4. Confirm that data is being split and migrated between each shard in your cluster.

    You can connect a mongo shell to each shard primary and use db.collection.find() to validate that the data was sharded as expected.