<commit_before>bdsrotater
==========

A Python module for managing a rotation of removable backup disks that
hold a PHD Virtual Backup Appliance Archive block de-duplication store.

Introduction
------------
This script was designed to be run on a Linux NFS server with the
backups disk attached.
When issued **_start_** the script mounts the removable disk, exports the
VBABACKUPS folder and powers on the VBA.
Conversely, when issued **_stop_** it shuts down the VBA, unexports the
VBABACKUPS folder and un-mounts the removable disk.
It will email alerts on failures.
It has only been tested on Debian 7 (Wheezy) with USB3 Removable disk.

Prerequisites
-------------

#### Udev Rules For Removeable Disks

Partition and format the removable disks with EXT4 (other file systems
are untested).
The options I use when formatting are:

    mkfs.ext4 /dev/sdxx -E lazy_itable_init=0 -m 0.4 -O ^has_journal

The removable disks need to have `/etc/fstab` entries the simplest way to
do this by creating a consistent device node with udev.  Replace `$DISK`
variable with your removable disk.
    
    DISK=/dev/sdx
    echo "SUBSYSTEM==\"block\", $(udevadm info -a -p $(udevadm info -q path -n $DISK) | grep ATTRS{serial} | head -n 1 | sed 's/    //'), KERNEL==\"sd?1\", SYMLINK+=\"backupdisk\", GROUP=\"BACKUP\"" >> /etc/udev/rules.d/10-backup-disk.rules
    udevadm trigger

This appends a rule for the removable disk to:

    /etc/udev/rules.d/10-backup-disk.rules

After running `udevadm trigger` you should see a device node
`/dev/backupdisk` which is a symlink to your actual removeable disk.
Repeat the process for every disk until they are all added.

In `/etc/fstab` you can then create a mount point for all the disks e.g.

    /dev/backupdisk /mnt/backup ext4 rw,noauto,noatime,nodiratime,data=writeback 0 0

#### VBABACKUPS Folder

A dedicated **VBABACKUPS** folder must exist on the root of each
removeable disk.

    mkdir /mnt/backup/VBABACKUPS

#### Python Modules

On Debian Wheezy you will need the following Python Packages.

    apt-get install python-pip python-mailer python2.7

Once installed you also need pysphere which can be installed as follows

    pip install -U pysphere

Installation
------------

Download and place the script in your path and make executable, e.g.

     wget -O /usr/local/bin/bdsrotator.py "https://raw.github.com/jelloir/bdsrotater/master/bdsrotater.py"
     chmod +x /usr/local/bin/bdsrotator.py

Review the help and pay attention to the defaults:<commit_msg>readme changes<commit_after>bdsrotater
==========

A Python module for managing a rotation of removable backup disks that
hold a PHD Virtual Backup Appliance Archive block de-duplication store.

Introduction
------------
This script was designed to be run on a Linux NFS server with the
backups disk attached.
When issued **_start_** the script mounts the removable disk, exports the
VBABACKUPS folder and powers on the VBA.
Conversely, when issued **_stop_** it shuts down the VBA, unexports the
VBABACKUPS folder and un-mounts the removable disk.
It will email alerts on failures.
It has only been tested on Debian 7 (Wheezy) with USB3 Removable disk.

Prerequisites
-------------

#### Udev Rules For Removeable Disks

Partition and format the removable disks with EXT4 (other file systems
are untested).
The options I use when formatting are:

    mkfs.ext4 /dev/sdxx -E lazy_itable_init=0 -m 0.4 -O ^has_journal

The removable disks need to have `/etc/fstab` entries the simplest way to
do this by creating a consistent device node with udev.  Replace `$DISK`
variable with your removable disk.
    
    DISK=/dev/sdx
    echo "SUBSYSTEM==\"block\", $(udevadm info -a -p $(udevadm info -q path -n $DISK) | grep ATTRS{serial} | head -n 1 | sed 's/    //'), KERNEL==\"sd?1\", SYMLINK+=\"backupdisk\", GROUP=\"BACKUP\"" >> /etc/udev/rules.d/10-backup-disk.rules
    udevadm trigger

This appends a rule for the removable disk to:

    /etc/udev/rules.d/10-backup-disk.rules

After running `udevadm trigger` you should see a device node
`/dev/backupdisk` which is a symlink to your actual removeable disk.
Repeat the process for every disk until they are all added.

In `/etc/fstab` you can then create a mount point for all the disks e.g.

    /dev/backupdisk /mnt/backup ext4 rw,noauto,noatime,nodiratime,data=writeback 0 0

#### VBABACKUPS Folder

A dedicated **VBABACKUPS** folder must exist on the root of each
removeable disk.  You can customise this but will need to use the
`--backupdir` argument when running the script.

    mkdir /mnt/backup/VBABACKUPS

#### Python Modules

On Debian Wheezy you will need the following Python Packages.

    apt-get install python-pip python-mailer python2.7

Once installed you also need pysphere which can be installed as follows

    pip install -U pysphere

Installation
------------

Download and place the script in your path and make executable, e.g.

     wget -O /usr/local/bin/bdsrotator.py "https://raw.github.com/jelloir/bdsrotater/master/bdsrotater.py"
     chmod +x /usr/local/bin/bdsrotator.py

Review the help and pay attention to the defaults:
{'fraction_non_alphanumeric': 0.09640203363316387, 'fraction_numerical': 0.007039499413375049, 'mean_word_length': 3.5588235294117645, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '389805', 'n_tokens_mistral': 1667, 'n_tokens_neox': 1560, 'n_words': 666}