<commit_before>## MongoDB Consistent Backup Tool - mongodb-consistent-backup

### About

Creates cluster-consistent point-in-time backups of MongoDB via wrapping 'mongodump'. Backups are remotely-pulled and outputted onto the host running the tool.

### Features

- Works on a single replset (2+ members) or a sharded cluster
- Auto-discovers healthy members for backup by considering replication lag, replication 'priority' and by preferring 'hidden' members.
- Creates cluster-consistent backups across many separate shards
- Archives and compresses backups (*inline compression with mongodump 3.2+*)
- Transparent restore process (*just add --oplogReplay flag to your mongorestore command*)
- AWS S3 Secure/HTTPS Multipart backup uploads (*optional*)
- [Nagios NSCA](https://sourceforge.net/p/nagios/nsca) push notification support (*optional*)
- Multi-threaded, single executable

### Current Limitations
- The host running 'mongodb-consistent-backup' must have enough disk, network and cpu resources to backup all shards in parallel 
- When MongoDB authentication is used, the same user/password/authdb and role(s) must exist on all hosts

### Requirements:
- Backup consistency depends on consistent server time across all hosts. Server time **must be synchronized on all nodes** using ntpd and a consistent time source
- Must have 'mongodump' installed and specified if not at default: */usr/bin/mongodump*. Even if you do not run MongoDB 3.2+, it is strongly recommended to use MongoDB 3.2+ binaries due to inline compression, parallelism, etc
- Must have Python 2.7 installed

### Build/Install

To build on CentOS/RedHat, you wil need the following packages (see command):

```
yum install python python-devel python-virtualenv gcc make libffi-devel openssl-devel
```

To install to default '*/usr/local/bin/mongodb-consistent-backup*':

```
cd path/to/mongo_backup 
make
make install
```

Use the PREFIX= variable to change the installation path (*default: /usr/local*), ie: ```make PREFIX=/usr install``` to install to: '*/usr/bin/mongodb-consistent-backup*'.

### Run a Backup

```
$ mongodb-consistent-backup -H mongos1.example.com -P 27018 -u mongodb-consistent-backup -p s3cr3t -n prodwebsite -l /opt/mongobackups
...
...
$ ls /opt/mongobackups
prodwebsite
```

### Restore a Backup

```
$ tar xfvz <shardname>.tar.gz
...
$ mongorestore --host mongod12.example.com --port 27017 -u admin -p 123456 --oplogReplay /path/to/backup
```

### Run as Docker Container

*Note: you need to use persistent volumes to store backups long-term on disk when using Docker. Data in Docker containers is destroyed when the container is deleted.*

#### Via Docker Hub
```
$ docker run -i timvaillancourt/mongodb_consistent_backup <mongodb_consistent_backup-flags>
```

#### Build Local Docker Image
```
$ cd /path/to/mongodb_consistent_backup
$ make docker
$ docker run -t mongodb_consistent_backup <mongodb_consistent_backup-flags>
```

### Roadmap

- "Distributed Mode" for running/storing backup on remote hosts (*via ssh + magic*)
- Single collection and/or database backup feature<commit_msg>README.md update<commit_after>## MongoDB Consistent Backup Tool - mongodb-consistent-backup

### About

Creates cluster-consistent point-in-time backups of MongoDB via wrapping 'mongodump'. Backups are remotely-pulled and outputted onto the host running the tool.

### Features

- Works on a single replset (2+ members) or a sharded cluster
- Auto-discovers healthy members for backup by considering replication lag, replication 'priority' and by preferring 'hidden' members.
- Creates cluster-consistent backups across many separate shards
- Archives and compresses backups (*inline compression with mongodump 3.2+*)
- Transparent restore process (*just add --oplogReplay flag to your mongorestore command*)
- AWS S3 Secure/HTTPS Multipart backup uploads (*optional*)
- [Nagios NSCA](https://sourceforge.net/p/nagios/nsca) push notification support (*optional*)
- Multi-threaded, single executable

### Current Limitations
- The host running 'mongodb-consistent-backup' must have enough disk, network and cpu resources to backup all shards in parallel 
- When MongoDB authentication is used, the same user/password/authdb and role(s) must exist on all hosts

### Requirements:
- Backup consistency depends on consistent server time across all hosts. Server time **must be synchronized on all nodes** using ntpd and a consistent time source
- Must have 'mongodump' installed and specified if not at default: */usr/bin/mongodump*. Even if you do not run MongoDB 3.2+, it is strongly recommended to use MongoDB 3.2+ binaries due to inline compression, parallelism, etc
- Must have Python 2.7 installed

### Build/Install

To build on CentOS/RedHat, you wil need the following packages (see command):

```
yum install python python-devel python-virtualenv gcc make libffi-devel openssl-devel
```

To install to default '*/usr/local/bin/mongodb-consistent-backup*':

```
cd path/to/mongo_backup 
make
make install
```

Use the PREFIX= variable to change the installation path (*default: /usr/local*), ie: ```make PREFIX=/usr install``` to install to: '*/usr/bin/mongodb-consistent-backup*'.

### Run a Backup

```
$ mongodb-consistent-backup -H mongos1.example.com -P 27018 -u mongodb-consistent-backup -p s3cr3t -n prodwebsite -l /opt/mongobackups
...
...
$ ls /opt/mongobackups
prodwebsite
```

### Restore a Backup

```
$ tar xfvz <shardname>.tar.gz
...
$ mongorestore --host mongod12.example.com --port 27017 -u admin -p 123456 --oplogReplay /path/to/backup
```

### Run as Docker Container

*Note: you need to use persistent volumes to store backups long-term on disk when using Docker. Data in Docker containers is destroyed when the container is deleted.*

#### Via Docker Hub
```
$ docker run -i timvaillancourt/mongodb_consistent_backup <mongodb_consistent_backup-flags>
```

#### Build and Run Docker Image
```
$ cd /path/to/mongodb_consistent_backup
$ make docker
$ docker run -t mongodb_consistent_backup <mongodb_consistent_backup-flags>
```

### Roadmap

- "Distributed Mode" for running/storing backup on remote hosts (*via ssh + magic*)
- Single collection and/or database backup feature
{'fraction_non_alphanumeric': 0.09995155821088325, 'fraction_numerical': 0.01001130308412724, 'mean_word_length': 4.560143626570915, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3557472', 'n_tokens_mistral': 1924, 'n_tokens_neox': 1739, 'n_words': 711}