<commit_before>produce
=======

Produce is an incremental build system for the command line, like make or redo.
It strives to be simple and user-friendly. It is less geared towards compiling
code, and more towards processing data and running sets of machine learning
experiments. Specifically, it works well with filenames that have not just one
but many variable parts, e.g. to indicate experimental parameters.

Produce recently graduated from first working prototype to "dogfood": the
developer is using it daily in his main project.

Dependencies
------------

* A Unix-like operating system such as Linux or Mac OS X. Cygwin under Windows
  may also work.
* Python 3

Installation
------------

Produce consists of a single executable Python script called "produce". Just
make sure it is in your PATH, e.g. by copying it to /usr/local/bin or by
linking to it from your $HOME/bin directory.

Usage
-----

When invoked, Produce will look for a file called produce.ini in the current
working directory. Its format is not documented yet, but there is 
[an example project](https://github.com/texttheater/produce/tree/master/doc/samples/tokenization)
that should be enlightening.

Also have a look at the
[PyGrunn 2014 slides](https://texttheater.github.io/produce-pygrunn2014).

Getting in touch
----------------
<commit_msg>Slightly more explicit installation instructions.<commit_after>produce
=======

Produce is an incremental build system for the command line, like Make or redo.
It strives to be simple and user-friendly. It is less geared towards compiling
code, and more towards processing data and running sets of machine learning
experiments. Specifically, it works well with filenames that have not just one
but many variable parts, e.g. to indicate experimental parameters.

Requirements
------------

* A Unix-like operating system such as Linux or Mac OS X. Cygwin under Windows
  may also work.
* Python 3
* Git (for downloading Produce)

Obtaining Produce
-----------------

Donload Produce by running the following command in a convenient location:

    git clone https://github.com/texttheater/produce

This will create a directory called `produce`. To update to the latest version
of Produce later, you can just go into that directory and run:

    git pull

Installing Produce
------------------

The `produce` directory contains an executable Python script also called
`produce`. This is all you need to run Produce. Just make sure it is in your
`PATH`, e.g. by copying it to `/usr/local/bin` or by linking to it from your
`$HOME/bin` directory.

Usage
-----

When invoked, Produce will look for a file called produce.ini in the current
working directory. Its format is not documented yet, but there is 
[an example project](https://github.com/texttheater/produce/tree/master/doc/samples/tokenization)
that should be enlightening.

Also have a look at the
[PyGrunn 2014 slides](https://texttheater.github.io/produce-pygrunn2014).

Getting in touch
----------------

{'fraction_non_alphanumeric': 0.09440910612654838, 'fraction_numerical': 0.006026113157013726, 'mean_word_length': 3.589861751152074, 'pattern_counts': {'":': 1, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '845612', 'n_tokens_mistral': 818, 'n_tokens_neox': 770, 'n_words': 398}