Navigation
This version of the documentation is archived and no longer supported. It will be removed on EOL_DATE. To learn how to upgrade your version of MongoDB Ops Manager, refer to the upgrade documentation.
You were redirected from a different version of the documentation. Click here to go back.
This version of the manual is no longer supported. It will be removed on EOL_DATE.

Manage S3 Blockstore Snapshot Storage

Ops Manager can back up MongoDB databases as snapshots to one or more of the following storage options:

  • Another MongoDB database, called a Blockstore,
  • As files stored on a local or network-attached file system, and/or
  • An AWS S3 bucket.

This tutorial covers backing up your MongoDB databases as snapshots stored in S3 and S3-compatible buckets. The metadata for S3 snapshot stores is stored in a MongoDB database.

Note

You may have issues that require you to use more than one snapshot store like needing more capacity, localizing data, or meeting privacy regulations. To learn how to assign snapshot stores to different data centers, see Assign Snapshot Stores to Specific Data Centers.

Considerations

S3 Blockstore Can’t Be Moved

After you create an S3 blockstore, you cannot move it to another S3 bucket. If you need to use a different S3 bucket to host your S3 blockstore, you must create a new S3 blockstore in that S3 bucket.

Storage API Support

MongoDB supports endpoints for:

IBM and Dell EMC support a subset of the full AWS S3 API.

You can use other S3-compatible endpoints. Ops Manager attempts to validate these endpoints when you save the S3 blockstore setup. If validation passes, Ops Manager saves the configuration. If validation fails, Ops Manager displays an error and doesn’t save the configuration.

Prerequisites

Metadata Storage Prerequisites

AWS S3 Storage Prerequisites

  1. Make sure you have an IAM user on AWS.

  2. Create your own AWS access keys for your IAM user. This allows you to create S3 buckets and store snapshot files in them. MongoDB does not create or issue AWS access keys.

  3. Create your own S3 bucket to store your S3 snapshot store snapshots.

    Note

    The IAM user for which you created the AWS access keys must own /AmazonS3/latest/user-guide/set-bucket-permissions.html> the AWS S3 Bucket.

IBM Cloud Object Storage Prerequisites

  1. Create an Access Key and Secret Key using IBM credential tools.
  2. Create your own S3-compatible bucket.

DellEMC Elastic Cloud Storage Prerequisites

  1. Create an Access Key and Secret Key from your ECS User ID.
  2. Create your own S3-compatible bucket.

Other S3-Compatible Storage

Other S3-compatible endpoints can be used. Ops Manager attempts to validate these endpoints when you save the configuration. If validation passes, the configuration is saved. If validation fails, Ops Manager displays an error and the configuration is not saved.

Add an S3 Blockstore

1
2

Click Create New S3 Blockstore.

3

Provide the S3 blockstore details.

Field Necessity Contents
Name Required Type the name for the S3 blockstore.
S3 Bucket Name Required Type the name of the S3 bucket where you want to host the the S3 blockstore.
S3 Endpoint Required

Type the URL for this AWS S3 or S3-compatible bucket.

What URL you write depends upon:

  1. Your S3 Bucket Name and
  2. If you checked Path Style Access.

Example

You created an S3 bucket called mybucket on AWS in the US East region.

Path Style Access Checked Not Checked
S3 Endpoint s3.us-east-2.amazonaws.com mybucket.s3.us-east-2.amazonaws.com
Requests to Bucket https://s3.us-east-2.amazonaws.com/mybucket https://mybucket.s3.us-east-2.amazonaws.com
S3 Max Connections Required Type a positive integer indicating the maximum number of connections to this AWS S3 or S3-compatible bucket.
Path Style Access Optional

Select if you want your AWS S3 or S3-compatible bucket to use a path-style URL endpoint (s3.amazonaws.com/<bucket>) instead of a virtual-host-style URL endpoint (<bucket>.s3.amazonaws.com).

To review the S3 bucket URL conventions, see the AWS S3 documentation

Server Side Encryption Optional Select to enable server-side encryption. Clear to disable server-side encryption.
Datastore Type Required Select Standalone, Replica Set or Sharded Cluster. This MongoDB database stores the metadata for the blockstore.
MongoDB Host List Conditional

Type a comma-separated list of mongod instances (for a Replica Set) or mongos instances (for a Sharded Cluster) in the <hostname:port> format that comprise the blockstore metadata database.

Example

host1.example.com:27017, host2.example.com:27017, host2.example.com:27018

This field displays only if you set Datastore Type to Replica Set or Sharded Cluster.

MongoDB Hostname Conditional

Type the hostname of the S3 blockstore metadata database.

This field displays only if you set Datastore Type to Standalone.

MongoDB Port Conditional

Type the port number of the S3 blockstore metadata database.

This field displays only if you set Datastore Type to Standalone.

Username Optional
Auth Mechanism Action Configure Instructions
Username and Password (SCRAM-SHA-1) Type the name of the user authorized to access the S3 blockstore metadata database. SCRAM
x.509 (X.509) Type the RFC2253-formatted subject from the client certificate of the user authorized to access the S3 blockstore metadata database. x.509
Kerberos (GSSAPI) Type the UPN of the user authorized to access the S3 blockstore metadata database. Kerberos
LDAP (PLAIN) Type the name of the LDAP user authorized to access the S3 blockstore metadata database. LDAP
Password Optional

Warning

If you did not use the credentialstool to encrypt this password, it is stored as plaintext in the database.

Auth Mechanism Action Configure Instructions
Username and Password (SCRAM-SHA-1) Type the password associated with the username that can access the S3 blockstore metadata database. SCRAM
x.509 (X.509) Leave it blank. x.509
Kerberos (GSSAPI) Kerberos retrieves the password from its keytab file. You do not need to type a password into this field. Kerberos
LDAP (PLAIN) Type the password of the LDAP user authorized to access the S3 blockstore metadata database. LDAP
Connection Options Optional

Type any additional configuration file options for the MongoDB instance.

This field supports unescaped values only.

For proper syntax, see Connection String URI Format in the MongoDB manual.

Encrypted Credentials Optional Select if the credentials for the database were encrypted using the credentialstool. The credentials include the Username, Password, AWS Access Key ID and AWS Secret Key.
Use TLS/SSL Optional

Select if the S3 blockstore metadata database only accepts connection encrypted using TLS.

Beyond this checkbox, to connect this S3 blockstore using TLS, you must enable:

  • TLS on the S3 blockstore database.

  • Ops Manager to use TLS to connect to this S3 blockstore.

    Note

    All backing databases share the same TLS certificates. If you have enabled TLS for another backing database, you need only select this checkbox to enable a TLS connection to this S3 blockstore.

New Assignment Enabled Optional Select if you want to enable this S3 blockstore after creating it. This is selected by default so the S3 blockstore can be assigned backup jobs. If you clear this checkbox, the S3 blockstore is created but you cannot assign backups to this S3 blockstore.
4

Click Create.

Edit an Existing S3 Blockstore

Once created, S3 snapshot stores are listed directly on the Snapshot Storage page in a table. Each row contains the settings for each S3 blockstore.

1

Navigate to the Snapshot Storage page.

  1. Click the Admin link.
  2. Click the Backup tab.
  3. (Optional) If you have not previously set the head directory, set it in the Head Directory box.
  4. Click the Snapshot Storage page.
2

Go to the row for the blockstore you want to edit.

3

In the MongoDB Connection column, update any values that need to be changed in the following fields:

Field Necessity Contents
S3 Bucket Name Required Type the name of the S3 bucket where you want to host the the S3 blockstore.
S3 Endpoint Required

Type the URL for this AWS S3 or S3-compatible bucket.

What URL you write depends upon:

  1. Your S3 Bucket Name and
  2. If you checked Path Style Access.

Example

You created an S3 bucket called mybucket on AWS in the US East region.

Path Style Access Checked Not Checked
S3 Endpoint s3.us-east-2.amazonaws.com mybucket.s3.us-east-2.amazonaws.com
Requests to Bucket https://s3.us-east-2.amazonaws.com/mybucket https://mybucket.s3.us-east-2.amazonaws.com
S3 Max Connections Required Type a positive integer indicating the maximum number of connections to this AWS S3 or S3-compatible bucket.
Path Style Access Optional

Click if you want your AWS S3 or S3-compatible bucket to use a path-style URL endpoint (s3.amazonaws.com/<bucket>) instead of a virtual-host-style URL endpoint (<bucket>.s3.amazonaws.com).

To review the S3 bucket URL conventions, see the AWS S3 documentation

Server Side Encryption Optional Click to enable server-side encryption. Clear to disable server-side encryption.
<hostname>:<port> Required

Type in one or more hosts that comprise the S3 blockstore metadata database in the <hostname:port> format.

Important

If these hosts are changed, the blockstore they host must have the same data as the original blockstore. Changing the host to a new blockstore results in data loss.

  • If the S3 blockstore metadata database is a Replica Set or Sharded Cluster, type a comma-separated list of mongod instances (for a Replica Set) or mongos instances (for a Sharded Cluster).

    Example

    host1.example.com:27017, host2.example.com:27017, host2.example.com:27018

  • If the S3 blockstore metadata database is a standalone MongoDB instance, type the hostname:port of the instance.

MongoDB Auth Username Optional
Auth Mechanism Action Configure Instructions
Username and Password (SCRAM-SHA-1) Type the name of the user authorized to access the S3 blockstore metadata database. SCRAM
x.509 (X.509) Type the RFC2253-formatted subject from the client certificate of the user authorized to access the S3 blockstore metadata database. x.509
Kerberos (GSSAPI) Type the UPN of the user authorized to access the S3 blockstore metadata database. Kerberos
LDAP (PLAIN) Type the name of the LDAP user authorized to access the S3 blockstore metadata database. LDAP
MongoDB Auth Password Optional

Warning

If you did not use the credentialstool to encrypt this password, it is stored as plaintext in the database.

Auth Mechanism Action Configure Instructions
Username and Password (SCRAM-SHA-1) Type the password associated with the username that can access the S3 blockstore metadata database. SCRAM
x.509 (X.509) Leave it blank. x.509
Kerberos (GSSAPI) Kerberos retrieves the password from its keytab file. You do not need to type a password into this field. Kerberos
LDAP (PLAIN) Type the password of the LDAP user authorized to access the S3 blockstore metadata database. LDAP

Note

The existing MongoDB Auth Password is not displayed.

Encrypted Credentials Optional Select if the credentials for the database were encrypted using the credentialstool. The credentials include the Username, Password, AWS Access Key ID and AWS Secret Key.
Use TLS/SSL Optional

Select if the blockstore database only accepts connection encrypted using TLS.

Beyond this checkbox, to connect this S3 blockstore using TLS, you must enable:

  • TLS on the S3 blockstore database.

  • Ops Manager to use TLS to connect to this S3 blockstore.

    Note

    All backing databases share the same TLS certificates. If you have enabled TLS for another backing database, you need only select this checkbox to enable a TLS connection to this S3 blockstore.

Connection Options Optional

Type any additional configuration file options for the MongoDB instance. This field supports unescaped values only.

For proper syntax, see Connection String URI Format in the MongoDB manual.

Assignment Labels Optional Type a comma-separated list of labels to assign the S3 blockstores to specific projects.
Load Factor Optional

Type any positive integer that expresses how much backup work you want this snapshot store to perform compared to another snapshot store.

Important

If you have only one snapshot store, skip this setting.

Backup work includes running backups, restoring snapshots or grooming blockstores. The term of backup work ratio assigned to a single snapshot store is called its Load Factor.

By default, Ops Manager assigns each snapshot store a Load Factor of 1. This means each snapshot store would perform the same amount of backup work.

As a snapshot store’s Load Factor increases, it performs more backup work compared to another snapshot store. If the Load Factor of snapshot store A is set to 2 and the Load Factor of snapshot store B is set to 1, A performs two times the backup work of B.

Example

How to estimate Load Factor

Consider a five-shard sharded cluster with the following backup storage configuration:

  • File system store (F) manages the backup work for one shard in the cluster. F is running on a single two-core physical server.
  • Blockstore (B) manages the backup work for four shards in the cluster. B is running as a four-node sharded cluster on four physical servers with two cores on each server.

In this example, B has four times the capability of F. B should receive the greater part of the ratio of backup work: 4:1. For every one backup task F performs, B performs 4.

Set the Load Factors of B to 4 and F to 1.

Snapshot stores with greater compute or storage performance should be given a greater Load Factor:

  • A file system store with 16-cores and 128 GB of RAM can back up more databases in less time than a file system store with only 2 cores and 8 GB of RAM.
  • A blockstore backed by a 10-node sharded cluster can back up more databases and groom more databases than a blockstore backed by a single replica set.

Load Factor can be set to 0. When one snapshot store’s Load Factor is set to 0, it performs no backup work at all. If a snapshot store’s Load Factor is changed while backup work is in progress, all jobs or tasks running on that snapshot store are allowed to finish. All future backup work then is re- distributed among the remaining snapshot stores with a Load Factor of 1 or greater and have Assignment Enabled selected.

Write Concern Required

Select your preferred Write Concern:

Default The write is not acknowledged.
Journaled

A primary or standalone MongoDB instance acknowledged the write and wrote that write to their on-disk journals.

Note

This is the default for standalone deployments.

Acknowledged A primary or standalone MongoDB instance acknowledged the write.
Replica Acknowledged

Two replica set members acknowledged the write.

Note

This is the default for replica sets or sharded clusters.

Majority A majority of the replica set members acknowledged the write.
4

Select the checkbox in the Assignment Enabled column.

Select if you want to enable this S3 blockstore after creating it. This is selected by default so the S3 blockstore can be assigned backup jobs. If you clear this checkbox, the S3 blockstore is created but you cannot assign backups to this S3 blockstore.

5

Click Save.

6

If you change any connection string values or the Write Concern, restart all the Ops Manager instances including those running Backup Daemons.

Warning

Modifying the connection string values or the Write Concern for an existing blockstore requires all Ops Manager components, including those only running the Backup Daemon, to be restarted to apply those changes. Connection parameters include:

  • <hostname>:<port>
  • MongoDB Auth Username
  • MongoDB Auth Password
  • Encrypted Credentials
  • Use TLS/SSL
  • Connection Options
  • Write Concern

If you change to another blockstore host, the data on the existing blockstore is not copied automatically to the other blockstore.

See also

For more details on the MongoDB connection string URI, see Connection String URI Format in the MongoDB Manual.

Delete an S3 Blockstore

1

Navigate to the Snapshot Storage page.

  1. Click the Admin link.
  2. Click the Backup tab.
  3. (Optional) If you have not previously set the head directory, set it in the Head Directory box.
  4. Click the Snapshot Storage page.
2