<commit_before>pips
====

Simplistic library to build type-safe data pipelines for processing streams of data items `DataPoint<V>`. Single units (`IDataProcessor<IN,OUT>`)
having input and output channels can be connected to form data processing networks where data points flow from one processor
to the next. Encapsulating specific processing logic in single data processors allows to reuse and recombine logic very easily.

Pips aims at ease of use of its components and is not (yet) optimized for high performance / data volumes. The underlying data structures are
mostly standard JDK classes. This might change in the future depending on user requirements. Performance should still be quite sufficient for
many scenarios where data collection / processing is not the software's core functionality. Pips was initially created to support easy collection of runtime
information for the macro benchmarking library [lab](http://github.com/bennidi/lab).

See the [javadoc](http://bennidi.github.io/lab)
<commit_msg>fixed pom, linked javadoc<commit_after>pips
====

Simplistic library to build type-safe data pipelines for processing streams of data items `DataPoint<V>`. Single units (`IDataProcessor<IN,OUT>`)
having input and output channels can be connected to form data processing networks where data points flow from one processor
to the next. Encapsulating specific processing logic in single data processors allows to reuse and recombine logic very easily.

Pips aims at ease of use of its components and is not (yet) optimized for high performance / data volumes. The underlying data structures are
mostly standard JDK classes. This might change in the future depending on user requirements. Performance should still be quite sufficient for
many scenarios where data collection / processing is not the software's core functionality. Pips was initially created to support easy collection of runtime
information for the macro benchmarking library [lab](http://github.com/bennidi/lab).

See the [javadoc](http://bennidi.github.io/pips) for more details on the API

{'fraction_non_alphanumeric': 0.05079825834542816, 'fraction_numerical': 0.0, 'mean_word_length': 5.228915662650603, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3646809', 'n_tokens_mistral': 487, 'n_tokens_neox': 458, 'n_words': 277}