<commit_before># Backup Integrity Checker [![Build Status](https://travis-ci.org/elliottslaughter/integrity-checker.svg?branch=master)](https://travis-ci.org/elliottslaughter/integrity-checker) [![Build Status](https://ci.appveyor.com/api/projects/status/w3mn421fl8l6r95a/branch/master?svg=true)](https://ci.appveyor.com/project/elliottslaughter/integrity-checker/branch/master)

This tool is an integrity checker for backups and filesystems.

## What it Does

  * Given a directory, the tools constructs a database of metadata
    (hashes, sizes, timestamps, etc.) of the contents. The database
    itself is of course checksummed as well.

  * Given two databases (or a database and a directory) the tool
    iterates the entries and prints a *helpful* summary of the
    differences between them. For example, the tool highlights
    suspicious patterns, such as files which got truncated (had
    non-zero size, and now have zero size) or have other patterns that
    could indicate corruption (e.g. the presence of NUL bytes, if the
    file originally had none). Surfacing useful data while minimizing
    false positives is an ongoing effort.

## What's it Good For?

Here are a couple sample use cases:

  * Backup integrity checking: Record a database when you make a
    backup. When restoring the backup, compare against the database to
    make sure the backup restore function has worked properly. (Or
    better, perform this check periodically to ensure that the backups
    are functioning properly.)

  * Continuous sync sanity checking: Suppose you use a tool like
    Dropbox. In theory, your files are "backed up" on a continuous
    basis. In practice, you have no assurance that the tool isn't
    modifying files behind your back. By recording databases
    periodically, you can sanity check that directories that shouldn't
    change often are in fact not changing. (Note: For this to be
    useful, the tool has to be very good at minimizing false positives.)

    This also applies to any live filesystem. Consider that a typical
    user will maintain continuity of data across possibly decades of
    hardware and filesystem upgrades. Every transition is an
    opportunity for silent data corruption. This tool can provide
    peace of mind that integrity is preserved for long-lived data.

The tool is designed around an especially stable database format so
that if something were to happen, it would be relatively
straightforward to recover the contained metadata.

## Install

For users running macOS or Linux on x86(-64), run:

    cargo install integrity-checker --features=asm

Other users run:

    cargo install integrity-checker

The `asm` feature enables an optimization in the `sha2` crate which
makes the SHA2 hash implementation faster.<commit_msg>Add usage to README.<commit_after># Backup Integrity Checker [![Build Status](https://travis-ci.org/elliottslaughter/integrity-checker.svg?branch=master)](https://travis-ci.org/elliottslaughter/integrity-checker) [![Build Status](https://ci.appveyor.com/api/projects/status/w3mn421fl8l6r95a/branch/master?svg=true)](https://ci.appveyor.com/project/elliottslaughter/integrity-checker/branch/master)

This tool is an integrity checker for backups and filesystems.

## What it Does

  * Given a directory, the tools constructs a database of metadata
    (hashes, sizes, timestamps, etc.) of the contents. The database
    itself is of course checksummed as well.

  * Given two databases (or a database and a directory) the tool
    iterates the entries and prints a *helpful* summary of the
    differences between them. For example, the tool highlights
    suspicious patterns, such as files which got truncated (had
    non-zero size, and now have zero size) or have other patterns that
    could indicate corruption (e.g. the presence of NUL bytes, if the
    file originally had none). Surfacing useful data while minimizing
    false positives is an ongoing effort.

## What's it Good For?

Here are a couple sample use cases:

  * Backup integrity checking: Record a database when you make a
    backup. When restoring the backup, compare against the database to
    make sure the backup restore function has worked properly. (Or
    better, perform this check periodically to ensure that the backups
    are functioning properly.)

  * Continuous sync sanity checking: Suppose you use a tool like
    Dropbox. In theory, your files are "backed up" on a continuous
    basis. In practice, you have no assurance that the tool isn't
    modifying files behind your back. By recording databases
    periodically, you can sanity check that directories that shouldn't
    change often are in fact not changing. (Note: For this to be
    useful, the tool has to be very good at minimizing false positives.)

    This also applies to any live filesystem. Consider that a typical
    user will maintain continuity of data across possibly decades of
    hardware and filesystem upgrades. Every transition is an
    opportunity for silent data corruption. This tool can provide
    peace of mind that integrity is preserved for long-lived data.

The tool is designed around an especially stable database format so
that if something were to happen, it would be relatively
straightforward to recover the contained metadata.

## Install

For users running macOS or Linux on x86(-64), run:

    cargo install integrity-checker --features=asm

Other users run:

    cargo install integrity-checker

The `asm` feature enables an optimization in the `sha2` crate which
makes the SHA2 hash implementation faster.

## Usage

To build a database `db.json.gz` from the directory at `path`, run:

    ick build db path

There are several operations one can perform on a database. To check
it directly against the contents of a directory, run:

    ick check db path

To diff two databases, run:

    ick diff db db2

To verify the validity of the checksums on a database, run:

    ick selfcheck db
{'fraction_non_alphanumeric': 0.0545972198961648, 'fraction_numerical': 0.004856807904873556, 'mean_word_length': 3.8950819672131147, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '274484', 'n_tokens_mistral': 1563, 'n_tokens_neox': 1486, 'n_words': 790}