<commit_before>Dumb Content-Addressed-Datastore
================================

Dumbcas is mainly a backup tool.

The idea of this yet-another-backup-tool is that you can rsync the data around
and merge multiple backups together with rsync without problem. Also a
single-bit corruption will only affect a single file. To be used as a backup
solution that is faster than raw rsync (supports file rename/move efficiently
because it's content-addressed) but permits deleting old backups unlike
[bup](http://github.com/apenwarr/bup).

Dumbcas defines an on-disk CAS (Content-Addressed-Storage) that is somewhat
inspired by git objects. It could use a remote CAS backup like
[camlistore](http://camlistore.org) but that's not implemented.

The tool is itself really simple and is a exercise of design. For example, all
the unit tests are run in parallel with test case locale logs that are printed
out on test case failure. This was certainly challenging for the implementation
of the subcommand support.


Installation
------------

First install [Go](http://golang.org), then:

    go get -u github.com/maruel/dumbcas
    dumbcas help


Backup and serve over the web
-----------------------------

    # List all files or directories to archive.
    # - One entry per line.
    # - Environment variables are supported.
    # - Can be absolute paths or relative to the toArchive file.
    echo ${HOME}> toArchive.txt
    echo /random/path> toArchive.txt

    # Archive the files to /path/to/storage.
    dumbcas archive -out=/path/to/storage -comment="My first backup" toArchive.txt

    # Verify the archive. Verifies all the sha-1 are valids.
    dumbcas fsck -out=/path/to/storage

    # Serve over http://localhost:8010/
    dumbcas web -out=/path/to/storage

You can set `$DUMBCAS_ROOT` environment variable to use a default value for
-out.


Delete a backup set
-------------------

    rm /path/to/storage/nodes/<month>/<name>
    dumbcas gc -out=/path/to/storage<commit_msg>-out was renamed to -root a long time ago.<commit_after>Dumb Content-Addressed-Datastore
================================

Dumbcas is mainly a backup tool.

The idea of this yet-another-backup-tool is that you can rsync the data around
and merge multiple backups together with rsync without problem. Also a
single-bit corruption will only affect a single file. To be used as a backup
solution that is faster than raw rsync (supports file rename/move efficiently
because it's content-addressed) but permits deleting old backups unlike
[bup](http://github.com/apenwarr/bup).

Dumbcas defines an on-disk CAS (Content-Addressed-Storage) that is somewhat
inspired by git objects. It could use a remote CAS backup like
[camlistore](http://camlistore.org) but that's not implemented.

The tool is itself really simple and is a exercise of design. For example, all
the unit tests are run in parallel with test case locale logs that are printed
out on test case failure. This was certainly challenging for the implementation
of the subcommand support.


Installation
------------

First install [Go](http://golang.org), then:

    go get -u github.com/maruel/dumbcas
    dumbcas help


Backup and serve over the web
-----------------------------

    # List all files or directories to archive.
    # - One entry per line.
    # - Environment variables are supported.
    # - Can be absolute paths or relative to the toArchive file.
    echo ${HOME}> toArchive.txt
    echo /random/path> toArchive.txt

    # Archive the files to /path/to/storage.
    dumbcas archive -root=/path/to/storage -comment="My first backup" toArchive.txt

    # Verify the archive. Verifies all the sha-1 are valids.
    dumbcas fsck -root=/path/to/storage

    # Serve over http://localhost:8010/
    dumbcas web -root=/path/to/storage

You can set `$DUMBCAS_ROOT` environment variable to use a default value for
-root.


Delete a backup set
-------------------

    rm /path/to/storage/nodes/<month>/<name>
    dumbcas gc -root=/path/to/storage
{'fraction_non_alphanumeric': 0.11542313470205308, 'fraction_numerical': 0.0025037556334501754, 'mean_word_length': 3.1441908713692945, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1235833', 'n_tokens_mistral': 1145, 'n_tokens_neox': 1083, 'n_words': 485}