<commit_before># radogost
[![Stories in Ready](https://badge.waffle.io/goodylabs/radogost.svg?label=ready&title=Ready)](http://waffle.io/goodylabs/radogost)
[![Stories in Progress](https://badge.waffle.io/goodylabs/radogost.svg?label=in%20progress&title=In%20Progress)](http://waffle.io/goodylabs/radogost)

[![Throughput Graph](http://graphs.waffle.io/goodylabs/radogost/throughput.svg)](https://waffle.io/goodylabs/radogost/metrics)

[Radogost](http://en.wikipedia.org/wiki/Radegast_%28god%29) 
is an old god of hospitality in the Slavic mythology.

This project is a set of scripts to:
* fetch backup from remote machines using [scp](http://en.wikipedia.org/wiki/Secure_copy) 

## Download (both on source server and on backup machine)

``` 
cd $HOME
git clone https://github.com/goodylabs/radogost.git
```

## Configure the backup procedure on source server (optional)

Example for MySQL for smaller databases with default table locking and UTF-8 with emojis for data consistency (leave last 3 backups):

```bash
{ crontab -l ; echo '30 5 * * * CURR_DATE=`date +\%Y\%m\%d\%H\%M\%S` && /bin/mkdir -p /root/backup && /usr/bin/mysqldump --lock-tables=true --default-character-set=utf8mb4 -u root project_x > /root/backup/backup_${CURR_DATE}_project_x.sql && /bin/gzip -9r /root/backup/backup_${CURR_DATE}_project_x.sql && /root/radogost/scripts/rotate_latest_backups.sh /root/backup *.sql.gz 3'; } | crontab -
```
Example for MySQL for larger databases with UTF-8 encoding with emojis but without default table locking that can cause a database speed downgrade on a heavily used production server (leave last 3 backups):

```bash
{ crontab -l ; echo '30 5 * * * CURR_DATE=`date +\%Y\%m\%d\%H\%M\%S` && /bin/mkdir -p /root/backup && /usr/bin/mysqldump --lock-tables=false --default-character-set=utf8mb4 -u root project_x > /root/backup/backup_${CURR_DATE}_project_x.sql && /bin/gzip -9r /root/backup/backup_${CURR_DATE}_project_x.sql && /root/radogost/scripts/rotate_latest_backups.sh /root/backup *.sql.gz 3'; } | crontab -
```


Example for Redis (leave last 3 backups):

```bash
{ crontab -l ; echo '30 5 * * * CURR_DATE=`date +\%Y\%m\%d\%H\%M\%S` && /bin/mkdir -p /root/backup && /bin/cp /var/lib/redis/dump.rdb /root/backup/backup_${CURR_DATE}_dump.rdb && /bin/gzip -9r /root/backup/backup_${CURR_DATE}_dump.rdb && /root/radogost/scripts/rotate_latest_backups.sh /root/backup *.rdb.gz 3'; } | crontab -
```

## Install in crontab on backup machine

The following will install a crontab entry with fetching:
* files that match regexp: backup_*.sql.gz
* from a remote directory /home/some_remote_user/backup
* from a remote machine somehost.com:22 

and put it in $HOME/project_x into local user.
It will autorotate old backups and only latest 3 will remain in that directory.

```bash<commit_msg>name of paragraph about DB backup procedures<commit_after># radogost
[![Stories in Ready](https://badge.waffle.io/goodylabs/radogost.svg?label=ready&title=Ready)](http://waffle.io/goodylabs/radogost)
[![Stories in Progress](https://badge.waffle.io/goodylabs/radogost.svg?label=in%20progress&title=In%20Progress)](http://waffle.io/goodylabs/radogost)

[![Throughput Graph](http://graphs.waffle.io/goodylabs/radogost/throughput.svg)](https://waffle.io/goodylabs/radogost/metrics)

[Radogost](http://en.wikipedia.org/wiki/Radegast_%28god%29) 
is an old god of hospitality in the Slavic mythology.

This project is a set of scripts to:
* fetch backup from remote machines using [scp](http://en.wikipedia.org/wiki/Secure_copy) 

## Download (both on source server and on backup machine)

``` 
cd $HOME
git clone https://github.com/goodylabs/radogost.git
```

## Configure the database backup procedure on source server (optional)

Example for MySQL for smaller databases with default table locking and UTF-8 with emojis for data consistency (leave last 3 backups):

```bash
{ crontab -l ; echo '30 5 * * * CURR_DATE=`date +\%Y\%m\%d\%H\%M\%S` && /bin/mkdir -p /root/backup && /usr/bin/mysqldump --lock-tables=true --default-character-set=utf8mb4 -u root project_x > /root/backup/backup_${CURR_DATE}_project_x.sql && /bin/gzip -9r /root/backup/backup_${CURR_DATE}_project_x.sql && /root/radogost/scripts/rotate_latest_backups.sh /root/backup *.sql.gz 3'; } | crontab -
```
Example for MySQL for larger databases with UTF-8 encoding with emojis but without default table locking that can cause a database speed downgrade on a heavily used production server (leave last 3 backups):

```bash
{ crontab -l ; echo '30 5 * * * CURR_DATE=`date +\%Y\%m\%d\%H\%M\%S` && /bin/mkdir -p /root/backup && /usr/bin/mysqldump --lock-tables=false --default-character-set=utf8mb4 -u root project_x > /root/backup/backup_${CURR_DATE}_project_x.sql && /bin/gzip -9r /root/backup/backup_${CURR_DATE}_project_x.sql && /root/radogost/scripts/rotate_latest_backups.sh /root/backup *.sql.gz 3'; } | crontab -
```


Example for Redis (leave last 3 backups):

```bash
{ crontab -l ; echo '30 5 * * * CURR_DATE=`date +\%Y\%m\%d\%H\%M\%S` && /bin/mkdir -p /root/backup && /bin/cp /var/lib/redis/dump.rdb /root/backup/backup_${CURR_DATE}_dump.rdb && /bin/gzip -9r /root/backup/backup_${CURR_DATE}_dump.rdb && /root/radogost/scripts/rotate_latest_backups.sh /root/backup *.rdb.gz 3'; } | crontab -
```

## Install in crontab on backup machine

The following will install a crontab entry with fetching:
* files that match regexp: backup_*.sql.gz
* from a remote directory /home/some_remote_user/backup
* from a remote machine somehost.com:22 

and put it in $HOME/project_x into local user.
It will autorotate old backups and only latest 3 will remain in that directory.

```bash
{'fraction_non_alphanumeric': 0.15030131159163418, 'fraction_numerical': 0.01240694789081886, 'mean_word_length': 4.478640776699029, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '388428', 'n_tokens_mistral': 2088, 'n_tokens_neox': 1961, 'n_words': 498}