Navigation
You were redirected from a different version of the documentation. Click here to go back.

Seed a New Secondary from Snapshot

Overview

When a new secondary is added to a replica set, MongoDB synchronizes data to that new secondary as it would with any member of the replica set. If you need to synchronize a large database, the process may take more time and resources than desired for your deployment.

Instead of synchronizing the data, the new secondary can use data from an existing backup snapshot to seed itself first. This limits performance impact to the new secondary and allows the active MongoDB process to continue storing and retrieving data.

Considerations

Need to Suspend Automation

You must suspend Automation for the replica set when performing this procedure. Suspending Automation lets you shut down a replica set member without Automation starting it back up again.

Differences in Backup Format

If your format of the data in your snapshot differs from the format you want for your restored database, use mongodump to back up the data from the existing database and mongorestore to restore the data to the new database.

Example

You want to restore a snapshot of a deployment that uses the MMAPv1 storage engine to a deployment that uses the WiredTiger storage engine. Use mongodump to backup the database then mongorestore to restore it to a new database.

seedSecondary Script is Not Provided

The seedSecondary script is omitted from the snapshot archive if either of the following is true:

  • If you specified a namespace filter when creating the snapshot.
  • If you use legacy mirrored config servers (config servers that are not in a replica set).

If you had received the script in the snapshot archive and then tried to apply it, the secondary member would become inconsistent with the other members.

Length of the Oplog

To seed a new secondary completely, you need both the snapshot and the oplog from the primary. The oplog must includes all database operations going back to the time when the snapshot was taken. If the oplog length, or window, is not longer than the time since the last snapshot was taken, not all data can be restored.

To check the oplog length before attempting to restore from a snapshot, run the rs.printReplicationInfo() command in the mongo shell on the primary member.

See also

Review Check the Size of the Oplog on the Troubleshoot Replica Sets page of the MongoDB Manual for guidelines on oplog size.

When you run the seedSecondary script, you must provide the replica set’s oplog size in gigabytes.

Note

The rs.printReplicationInfo() method returns the oplog size in megabytes (MB) and the seedSecondary script takes the value as gigabytes (GB). Divide the oplog size in MB by 1,000 to get the value in GB.

Prerequisites

To seed a secondary from a snapshot, you must have the snapshot archive. This archive includes:

  • The snapshot data files to be restored.

  • The seedSecondary.sh (Linux/Mac OS X) and seedSecondary.bat (Windows) scripts to perform the seeding activities.

  • A human-readable restore metadata file, restoreInfo.txt, to provide the data about the snapshot to be restored.

    This file contains:

    • Group name

    • Replica Set name

    • Cluster Id (if applicable)

    • Snapshot timestamp (as Timestamp at UTC)

    • Last Oplog applied (as a BSON Timestamp at UTC)

    • MongoDB version

    • Storage engine type

    • mongod startup options used on the database when the snapshot was taken

    • Encryption (Only appears if encryption is enabled on the snapshot)

    • Master Key UUID (Only appears if encryption is enabled on the snapshot)

      If restoring from an encrypted backup, you must have a certificate provisioned for this Master Key.

    Example

    The following is a sample restoreInfo.txt file content:

    Restore Information
    Group Name: test1
    Replica Set: config
    Cluster Id: bar
    Snapshot timestamp: Thu Dec 15 20:03:59 GMT 2016
    Last Oplog Applied: Thu Dec 15 20:04:45 GMT 2016 (1463688285, 2)
    MongoDB Version: 3.4.0
    Storage Engine: wiredTiger
    Startup Options: { "storage.directoryPerDB" : true }
    Encryption: enabled
    Master Key UUID: 7253
    

Procedure

1

Suspend Automation for the replica set.

Suspend automation using the procedure described in Suspend Automation.

Important

If you do not suspend Automation first, Ops Manager restarts the member after you shut it down.

2

Remove the secondary member to be replaced from your replica set.

Connect to the primary of your replica set and run the rs.remove() command in the mongo shell to remove the secondary to be replaced from the replica set.

rs.remove("<secondaryHost>:<secondaryPort>")
3

Start a new standalone instance.

  1. Connect to the server that hosts the new secondary.

  2. Create the data directory if it does not exist.

    Platform Command
    Linux / Mac OS X
    mkdir -m 755 /data
    
    Windows
    md c:\data
    
  3. Move the snapshot to the data directory.

    Platform Command
    Linux / Mac OS X
    mv <snapshot>.tar.gz /data
    
    Windows
    move <snapshot>.tar.gz c:\data
    
  4. Extract the snapshot.

    Platform Command
    Linux / Mac OS X
    tar zxvf <snapshot>.tar.gz
    
    Windows Install a package to extract the tar file then extract the file.
  5. Start a temporary standalone as the mongod user.

    <alternatePort> is a different port than the usual port on which your secondary runs.

    Platform Command
    Linux / Mac OS X
    sudo -u mongod mongod --port <alternatePort>
    --dbpath /data/<snapshot>
    
    Windows
    mongod.exe --port <alternatePort> --dbpath c:\data\<snapshot>
    
4

Run the seedSecondary script to re-create oplog collection and seed it with the correct timestamp.

Use the appropriate script for your operating system:

UNIX-based seedSecondary.sh
Windows seedSecondary.bat

This script recreates the oplog collection and seeds it with the timestamp of the snapshot’s creation. This tells the secondary to recreate all operations to the current time without requiring a full initial sync. Ops Manager customizes this script for this particular snapshot and includes it in the snapshot archive.

To run the script, issue the following command, where:

<alternatePort> The port of the mongod process
<oplogSizeInGigabytes> The size of the replica set’s oplog
<replicaSetName> The name of the replica set
<primaryHost:primaryPort> The hostname:port combination for the replica set’s primary

For UNIX-based systems:

./seedSecondary.sh <alternatePort> <oplogSizeInGigabytes> <replicaSetName> <primaryHost:primaryPort>
./seedSecondary.sh 27018 2 rs-1 primaryHost.example.com:27017

For Windows-based systems:

.\seedSecondary.bat <alternatePort> <oplogSizeInGigabytes> <replicaSetName> <primaryHost:primaryPort>
.\seedSecondary.bat 27018 2 rs-1 primaryHost.example.com:27017
5

Shut down the new secondary running as a standalone on the alternate port.

  1. Connect to the new secondary with the mongo shell:

    Platform Command
    Linux / Mac OS X
    mongo --port <alternatePort>
    
    Windows
    mongod.exe --port <alternatePort>
    
  2. Connect to the admin database to shut down the new secondary.

    use admin
    db.runCommand({ shutdown: 1 })
    
6

Start up the new secondary as part of the replica set.

Platform Command
Linux / Mac OS X
mongod --port <secondaryPort> --dbpath /data
--replSet <replicaSetName>
Windows
mongod.exe --port <secondaryPort> --dbpath /data
--replSet <replicaSetName>
7

Add the new secondary to the replica set on the primary host.

  1. Connect to the primary using the mongo shell.

    Platform Command
    Linux / Mac OS X
    mongo --host <primaryHost> --port <primaryPort>
    
    Windows
    mongo.exe --host <primaryHost> --port <primaryPort>
    
  2. Run the rs.add() command to add the new secondary to the replica set.

    rs.add("<secondaryHost>:<secondaryPort>")
    

Important

If you added the new secondary successfully, a message that appears to be an error (code 103) is returned.

demo:PRIMARY> rs.add("<secondaryHost>:<secondaryPort>")
{
  "ok" : 0,
  "errmsg" : "Found two member configurations with same host field,
  members.1.host == members.3.host == <secondaryHost>:<secondaryPort>",
  "code" : 103
}

If you see this message, you can ignore it.

If you want to verify that the new secondary is functioning, use rs.status().

rs.status()
8

Resume Automation for the replica set.

Use the procedure to Resume Automation.