- Back Up and Restore Deployments >
- Back up MongoDB Deployments >
- Backup Process
Backup Process¶
Ops Manager backups, once started, are an ongoing and continuous process. Data is continually backed up as long as the backup remains synchronized with the database. This process works like replica set data synchronization
The backup process:
Performs an initial sync to back up all of your existing data in its current state. In sharded clusters, this occurs on each shard and on the config servers.
Takes snapshots of the
data
directory in a deployment as often as your snapshot schedule specifies and then transfers the snapshots to a storage system.Sharded Clusters also can enable checkpoints to permit restores at moments between snapshots. To learn how sharded clusters use checkpoints, see checkpoints.
Monitors the oplog constantly and adds new database operations to the latest backup to keep the local Ops Manager copy of the data current.
The backup process works in this manner regardless of how snapshots are stored.
Backup Definition and Operational States¶
Each backup is defined as a job. Each job defines how much and how often data is backed up. Backup jobs are defined on a per-project basis.
Operational States¶
The following table lists the states of a backup job:
State | Retain Old Snapshots | Create New Snapshots | Apply Oplogs |
---|---|---|---|
Active |
Yes | Yes | Yes |
Stopped |
Yes | No | No |
Inactive |
No | No | No |
Change Operational States¶
Once backup jobs are active for a project, they run without further intervention until they are stopped or terminated. The operator can change the state of a backup in the following ways:
Initial State | Desired State | Method |
---|---|---|
Inactive |
Active after Initial Sync |
Click Start. |
Active |
Stopped | Click Stop. |
Stopped |
Active after Initial Sync |
Click Restart. |
Stopped |
Inactive | Click Terminate. Warning Terminate deletes all retained backups. |
Important
You may receive a Backup requires a resync
alert
for your backup jobs. This may require you to Resync a Backup.
This is not a different state, but a triggering of
a new Backup Process Flows. The backup job, once Initial
Sync
completes, becomes Active again.
Backup Process Flows¶
Initial Backup¶
Once created, a backup job goes through the following process flow:
- The Backup Agent connects to, and authenticates with, the databases associated with the backup job.
- The initial sync begins and enters its
starting
phase. Initial sync is a transition state between Inactive and Active. Initial Sync goes through a series of phases that are displayed on the Backup page to show progress. The Backup Agent streams the existing data to Ops Manager in 10 MB compressed bundles of documents called slices. The Backup Agent creates slices at the point in time when the backup was created. Data inserted to the instance after that point in time is captured separately. - The
transferring
phase begins as the slices are streamed and stored in the Oplog Store temporarily on the Backup Daemon’s behalf. The Backup Daemon service cannot dedicate itself to processing the large stream of initial sync slices at the expense of processing other backup jobs. The Oplog Store stores the slices until the Backup Daemon can fetch them. The Oplog Store is created when the first Snapshot Store is created. - While the Backup Agent is streaming the data, it tails the oplog. This tailing collects any differences between the state of the deployment database when the backup began and the deployment database’s current state. The oplog entries are sent in 10 MB compressed bundles of documents called oplog slices. These two streams of slices are collected in parallel to reduce the time needed to construct a complete backup.
- The
building
phase begins once Ops Manager has received all of the slices. In this phase, Ops Manager creates a local version of the backed up database called a head database on the host running the Backup Daemon service. - Ops Manager, through the Backup Daemon service, inserts the documents stored in the Oplog Store into the head database.
- The
applying oplogs
phase begins as Ops Manager applies the tailed oplog entries into the head database. - During the
fetching missing documents
phase, Ops Manager queries the deployment database for documents missed during document insertion. Ops Manager inserts the missing documents found in the deployment database into the head database. - After inserting the missing documents, the
creating indexes
phase begins as Ops Manager creates all of the indexes found in the deployment databases in the head database. When the indexes finish, the initial sync ends and the phase changes tocomplete
. - Depending upon which snapshot store you chose to store your backup,
a snapshot can be written out:
- As blocks to a blockstore.
- As blocks to an AWS S3 bucket. The metadata for those blocks is written to a MongoDB database on the Ops Manager host.
- As snapshot files to a file system store.
Note
The characteristics of each storage method is covered in Backup Configuration Options.
Subsequent Backups¶
The head database works as a full copy of the deployment database. It needs oplogs applied to it on a regular basis to keep its data synchronized with the deployment database. Snapshots are generated from the data stored in the head database according to your snapshot schedule.
Once the first full backup is completed, each active backup job follows this process:
- The Backup Agent tails the deployment’s oplog.
- The Backup Agent routinely batches new oplog entries in oplog slices and transfers them to Ops Manager.
- Ops Manager stores the oplog entries in the Oplog Store.
- Ops Manager applies the new oplog entries from the oplog slices to the head database that stores the deployment backup.
- Ops Manager creates a new snapshot and stores it in the snapshot store as specified in your snapshot schedule.