<commit_before>s3helper
========

A few methods to simplify using Amazon S3 as a filestore.

## Dependencies

    $ gem install aws-sdk
    $ export S3_ACCESS_KEY_ID=" ... "
    $ export S3_SECRET_ACCESS_KEY=" ... "
    $ export S3_BUCKET=" ... "
    $ export S3_HOST=" ... "


For running the tests:

    $ gem install rspec
    $ gem install faker

## Usage

Include module `S3helper` for the following useful functions:

### Filestore::ls(dirpath = nil, filesmatch = '*')

Get a directory listing. Returns an array of filenames.

- simple globbing (\* and ?) is supported on **filesmatch**
- `dirpath` must not have a leading slash
- `filesmatch` must not contain any slashes
- only lists simple files; to list subdirectories use **lsdir**
<commit_msg>minor formatting changes<commit_after>s3helper
========

A few methods to simplify using Amazon S3 as a filestore.

## Dependencies

    $ gem install aws-sdk
    $ export S3_ACCESS_KEY_ID=" ... "
    $ export S3_SECRET_ACCESS_KEY=" ... "
    $ export S3_BUCKET=" ... "
    $ export S3_HOST=" ... "


For running the tests:

    $ gem install rspec
    $ gem install faker

## Usage

Include module **S3helper** for the following useful functions:

### Filestore::ls(dirpath = nil, filesmatch = '\*')

Get a directory listing. Returns an array of filenames.

- simple globbing (\* and ?) is supported on **filesmatch**
- **dirpath** must not have a leading slash
- **filesmatch** must not contain any slashes
- only lists simple files; to list subdirectories use **lsdir**

{'fraction_non_alphanumeric': 0.12107329842931937, 'fraction_numerical': 0.009162303664921465, 'mean_word_length': 3.3936781609195403, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4364255', 'n_tokens_mistral': 485, 'n_tokens_neox': 471, 'n_words': 170}