Navigation
You were redirected from a different version of the documentation. Click here to go back.

Backup Process

Ops Manager backups, once started, are an ongoing and continuous process. Data is continually backed up as long as the backup remains synchronized with the database. This process works like replica set data synchronization

The backup process:

  1. Performs an initial sync to back up all of your existing data in its current state. In sharded clusters, this occurs on each shard and on the config servers.

  2. Takes snapshots of the data directory in a deployment as often as your snapshot schedule specifies and then transfers the snapshots to a storage system.

    Sharded Clusters also can enable checkpoints to permit restores at moments between snapshots. To learn how sharded clusters use checkpoints, see checkpoints.

  3. Monitors the oplog constantly and adds new database operations to the latest backup to keep the local Ops Manager copy of the data current.

The backup process works in this manner regardless of how snapshots are stored.

Backup Definition and Operational States

Each backup is defined as a job. Each job defines how much and how often data is backed up. Backup jobs are defined on a per-group basis.

Operational States

The following table lists the states of a backup job:

State Retain Old Snapshots Create New Snapshots Apply Oplogs
Active Yes Yes Yes
Stopped Yes No No
Inactive No No No

Change Operational States

Once backup jobs are active for a group, they run without further intervention until they are stopped or terminated. The operator can change the state of a backup in the following ways:

Initial State Desired State Method
Inactive Active after Initial Sync Click Start.
Active Stopped Click Stop.
Stopped Active after Initial Sync Click Restart.
Stopped Inactive

Click Terminate.

Warning

Terminate deletes all retained backups.

Important

You may receive a Backup requires a resync alert for your backup jobs. This may require you to Resync a Backup. This is not a different state, but a triggering of a new Backup Process Flows. The backup job, once Initial Sync completes, becomes Active again.

Backup Process Flows

Initial Backup

Once created, a backup job goes through the following process flow:

Diagram showing the flow of data for Ops Manager's backup components when using snapshot storage.
  1. The Backup Agent connects to, and authenticates with, the databases associated with the backup job.
  2. The initial sync begins and enters its starting phase. Initial sync is a transition state between Inactive and Active. Initial Sync goes through a series of phases that are displayed on the Backup page to show progress. The Backup Agent streams the existing data to Ops Manager in 10 MB compressed bundles of documents called slices. The Backup Agent creates slices at the point in time when the backup was created. Data inserted to the instance after that point in time is captured separately.
  3. The transferring phase begins as the slices are streamed and stored in the Oplog Store temporarily on the Backup Daemon’s behalf. The Backup Daemon service cannot dedicate itself to processing the large stream of initial sync slices at the expense of processing other backup jobs. The Oplog Store stores the slices until the Backup Daemon can fetch them. The Oplog Store is created when the first Snapshot Store is created.
  4. While the Backup Agent is streaming the data, it tails the oplog. This tailing collects any differences between the state of the deployment database when the backup began and the deployment database’s current state. The oplog entries are sent in 10 MB compressed bundles of documents called oplog slices. These two streams of slices are collected in parallel to reduce the time needed to construct a complete backup.
  5. The building phase begins once Ops Manager has received all of the slices. In this phase, Ops Manager creates a local version of the backed up database called a head database on the host running the Backup Daemon service.
  6. Ops Manager, through the Backup Daemon service, inserts the documents stored in the Oplog Store into the head database.
  7. The applying oplogs phase begins as Ops Manager applies the tailed oplog entries into the head database.
  8. During the fetching missing documents phase, Ops Manager queries the deployment database for documents missed during document insertion. Ops Manager inserts the missing documents found in the deployment database into the head database.
  9. After inserting the missing documents, the creating indexes phase begins as Ops Manager creates all of the indexes found in the deployment databases in the head database. When the indexes finish, the initial sync ends and the phase changes to complete.
  10. Depending upon which snapshot store you chose to store your backup, a snapshot can be written out:
    1. As blocks to a blockstore.
    2. As blocks to an AWS S3 bucket. The metadata for those blocks is written to a MongoDB database on the Ops Manager host.
    3. As snapshot files to a file system store.

Note

The characteristics of each storage method is covered in Backup Configuration Options.

Subsequent Backups

The head database works as a full copy of the deployment database. It needs oplogs applied to it on a regular basis to keep its data synchronized with the deployment database. Snapshots are generated from the data stored in the head database according to your snapshot schedule.

Once the first full backup is completed, each active backup job follows this process:

  1. The Backup Agent tails the deployment’s oplog.
  2. The Backup Agent routinely batches new oplog entries in oplog slices and transfers them to Ops Manager.
  3. Ops Manager stores the oplog entries in the Oplog Store.
  4. Ops Manager applies the new oplog entries from the oplog slices to the head database that stores the deployment backup.
  5. Ops Manager creates a new snapshot and stores it in the snapshot store as specified in your snapshot schedule.