<commit_before>go-pipeliner
============

Tool for creating automation-related pipelines.

This is a work in progress but the general idea is already here. Do contact me if you are interested on helping.

How to write your module (plugin).
----------------------------------

There are currently 3 types of plugins:

1. Input plugins: These plugins will always be at the start of the pipeline. From the pipeline point of view, they do not accept any input and generate output. At least one input plugin is required for a valid pipeline. Its job is to generate data to be consumed by the pipeline.
2. Filter plugins: These plugins accept input and generate output. They are completelly optional and a valid pipeline does not require any filter plugin to be present. Its job is to look at any input it receives and decide if each item should continue in the pipeline or be dropped from it (so it will never reach any output plugins).
3. Output plugins: These plugins are always at the end of a pipeline. They accept input and generate no output. At least one output plugin must be present for a valid pipeline. its job is to consume somehow the items that reach the end of the pipeline.

Note that one pipeline can have multiple plugins in each category (input, filter and output). Multiple input plugins are executed concurrently and their outputs are are multiplexed in a single channel that connects to the next phase of the pipeline. Multipe filter plugins are executed serially, meaning that if an item is filtered by filter1, filter2 will never see it. Multiple outputs are executed concurrently and each one get one copy of any item (using a built-in demultiplexer) that reach this pipeline stage.

When writting a plugin, you first decide which type it is. Once you did, you need to make sure your plugin implements the correct interface as defined in https://github.com/brunoga/go-pipeliner/blob/master/pipeline/pipeline.go (InputNode, FilterNode or OutputNode). For any types, the generic module interface defined in https://github.com/brunoga/go-modules/blob/master/modules.go (to simplify things, you can use the GenericModule defined in https://github.com/brunoga/go-modules/blob/master/generic-module.go using struct embedding (see existing plugins to see how it works) and only override methods that need to be overriden for your plugin.

Here are some of the most inportant methods that need to be implemented:
<commit_msg>Added simple instructions on how to run Pipeliner.<commit_after>go-pipeliner
============

Tool for creating automation-related pipelines.

This is a work in progress but the general idea is already here. Do contact me if you are interested on helping.

How to use it.
--------------

1. Fetch the source code and compile it.

        go get github.com/brunoga/go-pipeliner

2. Edit $GOPATH/src/github.com/brunoga/go-pipeliner/config.yml to configure your pipeline(s).
3. Run your pipelines.

        $GOPATH/bin/go-pipeliner -config $GOPATH/src/github.com/brunoga/go-pipeliner/config.yml

How to write your module (plugin).
----------------------------------

There are currently 3 types of plugins:

1. Input plugins: These plugins will always be at the start of the pipeline. From the pipeline point of view, they do not accept any input and generate output. At least one input plugin is required for a valid pipeline. Its job is to generate data to be consumed by the pipeline.
2. Filter plugins: These plugins accept input and generate output. They are completelly optional and a valid pipeline does not require any filter plugin to be present. Its job is to look at any input it receives and decide if each item should continue in the pipeline or be dropped from it (so it will never reach any output plugins).
3. Output plugins: These plugins are always at the end of a pipeline. They accept input and generate no output. At least one output plugin must be present for a valid pipeline. its job is to consume somehow the items that reach the end of the pipeline.

Note that one pipeline can have multiple plugins in each category (input, filter and output). Multiple input plugins are executed concurrently and their outputs are are multiplexed in a single channel that connects to the next phase of the pipeline. Multipe filter plugins are executed serially, meaning that if an item is filtered by filter1, filter2 will never see it. Multiple outputs are executed concurrently and each one get one copy of any item (using a built-in demultiplexer) that reach this pipeline stage.

When writting a plugin, you first decide which type it is. Once you did, you need to make sure your plugin implements the correct interface as defined in https://github.com/brunoga/go-pipeliner/blob/master/pipeline/pipeline.go (InputNode, FilterNode or OutputNode). For any types, the generic module interface defined in https://github.com/brunoga/go-modules/blob/master/modules.go (to simplify things, you can use the GenericModule defined in https://github.com/brunoga/go-modules/blob/master/generic-module.go using struct embedding (see existing plugins to see how it works) and only override methods that need to be overriden for your plugin.

Here are some of the most inportant methods that need to be implemented:

{'fraction_non_alphanumeric': 0.06270249666476081, 'fraction_numerical': 0.002858776443682104, 'mean_word_length': 3.95561850802644, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '388971', 'n_tokens_mistral': 1261, 'n_tokens_neox': 1223, 'n_words': 782}