<commit_before>GrabLib
=======

Python tool and library for downloading library files (eg. Javascript and CSS) so they don't clog up your repo.

Definition files can either be JSON or Python (see `/examples`). So the versions of libraries used in your project can be defined in version control without the need to include them all and clog things up.

The formats define the sames things, the following values can be set:
* Libs: a dictionary of files to download and locations to put them, zip files may also be defined with multiple files extracted to different locations.
* [OPTIONAL] Target: the location to put the files, defaults to the working directory.
* [OPTIONAL] Verbosity: Level 0 (nothing except errors), 1 (less),  2 (default), 3 (everything)
* [OPTIONAL] Overwrite: bool, whether or not to overwrite files that already exist, default is not to download existing
* [OPTIONAL] file_permissions: explitly set files permissions

All options settings can also be overwritten at the command line, in python format or using the API a special output function may also be defined.

### TODO

* dry-run mode where links are check, zip files are opened but files are not actually put into place
* try out coloured term with https://pypi.python.org/pypi/colorterm
* add javascript/css concatination and compression<commit_msg>add TODO<commit_after>GrabLib
=======

Python tool and library for downloading library files (eg. Javascript and CSS) so they don't clog up your repo.

Definition files can either be JSON or Python (see `/examples`). So the versions of libraries used in your project can be defined in version control without the need to include them all and clog things up.

The formats define the sames things, the following values can be set:
* Libs: a dictionary of files to download and locations to put them, zip files may also be defined with multiple files extracted to different locations.
* [OPTIONAL] Target: the location to put the files, defaults to the working directory.
* [OPTIONAL] Verbosity: Level 0 (nothing except errors), 1 (less),  2 (default), 3 (everything)
* [OPTIONAL] Overwrite: bool, whether or not to overwrite files that already exist, default is not to download existing
* [OPTIONAL] file_permissions: explitly set files permissions

All options settings can also be overwritten at the command line, in python format or using the API a special output function may also be defined.

### TODO

* dry-run mode where links are check, zip files are opened but files are not actually put into place
* try out coloured term with https://pypi.python.org/pypi/colorterm
* add example of getting local file with file:///home/samuel/...
* add javascript/css concatination and compression
{'fraction_non_alphanumeric': 0.060294117647058824, 'fraction_numerical': 0.0029411764705882353, 'mean_word_length': 4.656964656964657, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1283782', 'n_tokens_mistral': 694, 'n_tokens_neox': 661, 'n_words': 406}