• Back Up MongoDB Deployments >
• Restore MongoDB Deployments >
• Restore a Sharded Cluster from a Backup

Restore a Sharded Cluster from a Backup

Overview

You can restore a sharded cluster onto new hardware from the artifacts captured by Backup.

You can restore from a snapshot or checkpoint. You must enable checkpoints to use them. When you restore from a checkpoint, Ops Manager takes the snapshot previous to the checkpoint and applies the oplog to create a custom snapshot. Checkpoint recovery takes longer than recovery from a stored snapshot.

Ops Manager provides restore files as downloadable archive; Ops Manager can also scp files directly to your system. The scp delivery method requires additional configuration but provides faster delivery.

Ops Manager provides a separate backup artifacts for each shard and one file for the config servers.

Sequence

The sequence to restore a snapshot is to:

• select and download the restore files,
• distribute the restore files to their new locations,
• start the mongod instances,
• configure each shard’s replica set, and
• configure and start the cluster.

Considerations

Client Requests During Restoration

You must ensure that the MongoDB deployment does not receive client requests during restoration. You must either:

• restore to new systems with new hostnames and reconfigure your application code once the new deployment is running, or
• ensure that the MongoDB deployment will not receive client requests while you restore data.

Snapshots when Agent Cannot Stop Balancer

Ops Manager displays a warning next to cluster snapshots taken while the balancer is enabled. If you restore from such a snapshot, you run the risk of lost or orphaned data. For more information, see Snapshots when Agent Cannot Stop Balancer.

Procedures

Select and Download the Snapshot Files

1

Select the Backup tab and then select Sharded Cluster Status.

2

Click the name of the sharded cluster to restore.

Ops Manager displays your selection’s stored snapshots.

3

Click the Restore button and select the snapshot from which to restore.

Click the Restore button at the top of the page. On the resulting page, choose from restoring from a stored snapshot or creating a custom snapshot.

To select a stored snapshot, click the Restore this snapshot link next to the snapshot.

To create a custom snapshot from a checkpoint, select the Use Custom Point In Time checkbox and enter the point in time in the Date and Time fields, and click Next.

4

Select the checkpoint.

Ops Manager lists the checkpoints that are the closest match to the point in time that you selected. Select a checkpoint from which to create the snapshot, and click Next.

5

Select HTTP as the delivery method for the snapshot.

In the Delivery Method field, select Pull via Secure HTTP (HTTPS).

Optionally, you can instead choose SCP as the delivery method. See: Retrieve a Snapshot with SCP Delivery for the SCP delivery option’s configuration. If you choose SCP, you must provide the hostname and port of the server to receive the files and provide access to the server through a username and password or though an SSH key. Follow the instructions on the Ops Manager screen.

6

Select tar.gz as the download format.

In the Format drop-down list, select Archive (tar.gz).

7

Finalize the request.

Click Finalize Request and confirm your identify via two-factor verification. Then click Finalize Request again.

8

Retrieve the snapshot.

Ops Manager creates one-time links to tar files for the snapshot. The links are available for one download each, and each expires after an hour.

To download the tar files, select the Ops Manager Backup tab and then Restore Jobs. When the restore job completes, the download link appears for every config server and shard in the cluster. Click each link to download the tar files and copy each tar file to its server. For a shard, copy the file to every member of the shard’s replica set.

If you optionally chose SCP as the delivery method, the files are copied to the server directory you specfied. To verify that the files are complete, see the section on how to validate an SCP restore.

Restore Each Shard’s Primary

For all shards, restore the primary. You must have a copy of the snapshot on the server that provides the primary:

1

Shut down the entire replica set.

Shut down the replica set’s mongod processes using one of the following methods, depending on your configuration:

• Automated Deployment:

  If you use Ops Manager Automation to manage the replica set, you must shut down through the Ops Manager console. See Shut Down MongoDB Processes.

• Non-Automated Deployment on MongoDB 2.6 or Later:

  Connect to each member of the set and issue the following:

  copy

  use admin
  db.shutdownServer()
  

• Non-Automated Deployment on MongoDB 2.4 or earlier:

  Connect to each member of the set and issue the following:

  copy

  use admin
  db.shutdownServer( { force: true } )
  

2

Restore the snapshot data files to the primary.

Extract the data files to the location where the mongod instance will access them through the dbpath setting. If you are restoring to existing hardware, use a different data directory than used previously. The following are example commands:

copy

tar -xvf <backup-restore-name>.tar.gz
mv <backup-restore-name> /data

3

Start the primary with the new dbpath.

For example:

copy

mongod --dbpath /<path-to-data> --replSet <replica-set-name> --logpath /<path-to-data>/mongodb.log --fork

4

Connect to the primary and initiate the replica set.

For example, first issue the following to connect:

copy

mongo

And then issue rs.initiate():

copy

rs.initiate()

5

Restart the primary as a standalone, without the --replSet option.

Use the following sequence:

1. Shut down the process using one of the following methods:

   • Automated Deployment:

     Shut down through the Ops Manager console. See Shut Down MongoDB Processes.

   • Non-Automated Deployment on MongoDB 2.6 or Later:

     copy

     use admin
     db.shutdownServer()
     

   • Non-Automated Deployment on MongoDB 2.4 or earlier:

     copy

     use admin
     db.shutdownServer( { force: true } )
     

2. Restart the process as a standalone:

   copy

   mongod --dbpath /<path-to-data> --logpath /<path-to-data>/mongodb.log --fork
   

6

Connect to the primary and drop the oplog.

For example, first issue the following to connect:

copy

mongo

And then issue rs.drop() to drop the oplog.

copy

use local
db.oplog.rs.drop()

7

Run the seedSecondary.sh script on the primary.

The seedSecondary.sh script re-creates the oplog collection and seeds it with the timestamp of the snapshot’s creation. This will allow the secondary to come back up to time without requiring a full initial sync. This script is customized by Ops Manager for this particular snapshot and is included in the backup restore file.

To run the script, issue the following command at the system prompt, where <mongod-port> is the port of the mongod instance and <oplog-size-in-gigabytes> is the size of the replica set’s oplog:

copy

./seedSecondary.sh <mongod-port> <oplog-size-in-gigabytes>

8

Restart the primary as part of a replica set.

Use the following sequence:

1. Shut down the process using one of the following methods:

   • Automated Deployment:

     Shut down through the Ops Manager console. See Shut Down MongoDB Processes.

   • Non-Automated Deployment on MongoDB 2.6 or Later:

     copy

     use admin
     db.shutdownServer()
     

   • Non-Automated Deployment on MongoDB 2.4 or earlier:

     copy

     use admin
     db.shutdownServer( { force: true } )
     

2. Restart the process as part of a replica set:

   copy

   mongod --dbpath /<path-to-data> --replSet <replica-set-name>
   

Restore All Secondaries

After you have restored the primary for a shard you can restore all secondaries. You must have a copy of the snapshot on all servers that provide the secondaries:

1

Connect to the server where you will create the new secondary.

2

Restore the snapshot data files to the secondary.

Extract the data files to the location where the mongod instance will access them through the dbpath setting. If you are restoring to existing hardware, use a different data directory than used previously. The following are example commands:

copy

tar -xvf <backup-restore-name>.tar.gz
mv <backup-restore-name> /data

3

Start the secondary as a standalone, without the --replSet option.

Use the following sequence:

1. Shut down the process using one of the following methods:

   • Automated Deployment:

     Shut down through the Ops Manager console. See Shut Down MongoDB Processes.

   • Non-Automated Deployment on MongoDB 2.6 or Later:

     copy

     use admin
     db.shutdownServer()
     

   • Non-Automated Deployment on MongoDB 2.4 or earlier:

     copy

     use admin
     db.shutdownServer( { force: true } )
     

2. Restart the process as a standalone:

   copy

   mongod --dbpath /<path-to-data> --logpath /<path-to-data>/mongodb.log --fork
   

4

Run the seedSecondary.sh script on the secondary.

The seedSecondary.sh script re-creates the oplog collection and seeds it with the timestamp of the snapshot’s creation. This will allow the secondary to come back up to time without requiring a full initial sync. This script is customized by Ops Manager for this particular snapshot and is included in the backup restore file.

To run the script, issue the following command at the system prompt, where <mongod-port> is the port of the mongod instance and <oplog-size-in-gigabytes> is the size of the replica set’s oplog:

copy

./seedSecondary.sh <mongod-port> <oplog-size-in-gigabytes>

5

Restart the secondary as part of the replica set.

Use the following sequence:

1. Shut down the process using one of the following methods:

   • Automated Deployment:

     Shut down through the Ops Manager console. See Shut Down MongoDB Processes.

   • Non-Automated Deployment on MongoDB 2.6 or Later:

     copy

     use admin
     db.shutdownServer()
     

   • Non-Automated Deployment on MongoDB 2.4 or earlier:

     copy

     use admin
     db.shutdownServer( { force: true } )
     

2. Restart the process as part of a replica set:

   copy

   mongod --dbpath /<path-to-data> --replSet <replica-set-name>
   

6

Connect to the primary and add the secondary to the replica set.

Connect to the primary and use rs.add() to add the secondary to the replica set.

copy

rs.add("<host>:<port>")

Repeat this operation for each member of the set.

Restore Each Config Server

Perform this procedure separately for each config server. Each config server must have a copy of the tar file with the config server data.

1

Restore the snapshot to the config server.

Extract the data files to the location where the config server’s mongod instance will access them. This is the location you will specify as the storage.dbPath when running mongod for the config server.

copy

tar -xvf <backup-restore-name>.tar.gz
mv <backup-restore-name> /data

2

Start the config server.

The following example starts the config server using the new data:

copy

mongod --configsvr --dbpath /data

3

Update the sharded cluster metadata.

If the new shards do not have the same hostnames and ports as the original cluster, you must update the shard metadata. To do this, connect to each config server and update the data.

First connect to the config server with the mongo shell. For example:

copy

mongo

Then access the shards collection in the config database. For example:

copy

use config
db.shards.find().pretty()

The find() method returns the documents in the shards collection. The collection contains a document for each shard in the cluster. The host field for a shard displays the name of the shard’s replica set and then the hostname and port of the shard. For example:

copy

{ "_id" : "shard0000", "host" : "shard1/localhost:30000" }

To change a shard’s hostname and port, use the MongoDB update() command to modify the documents in the shards collection.

Start the mongos

Start the cluster’s mongos bound to your new config servers.

←   Restore Flows Restore a Replica Set from a Backup  →

© MongoDB, Inc 2008-present. MongoDB, Mongo, and the leaf logo are registered trademarks of MongoDB, Inc.