<commit_before># SciPipe

An experimental library for writing Scientific (Batch) Workflows in vanilla [Go(lang)](http://golang.org),
based on an idea for a flow-based like pattern in pure Go, as presented in
[this Gopher Academy blog post](http://blog.gopheracademy.com/composable-pipelines-pattern).

See a short demo about how it might look, working with SciPipe in the terminal:

![SciPipe Demo](http://i.imgur.com/Egzo1VO.gif)

## Benefits

Some benefits of SciPipe, that are not always available in other systems available:

- **Easy-to-grasp behaviour:** Data flowing through a network.
- **Parallel:** Apart from the inherent pipeline parallelism, SciPipe processes also spawn multiple parallel tasks when the same process has multiple inputs.
- **Concurrent:** Each process runs in an own light-weight thread, and is not blocked by
  operations in other processes, except when waiting for inputs from upstream processes.
- **Inherently simple:** Uses Go's concurrency primitives (go-routines and channels)
  to create an "implicit" scheduler, which means very little additional infrastructure code.
  This means that the code is easy to modify and extend.
- **Flexible:** Processes that wrap command-line programs and scripts can be combined with
  processes coded directly in Golang.
- **Custom file naming:** SciPipe gives you full control over how file names are produced,
  making it easy to understand and find your way among the output files of your computations.
- **Highly Debuggable(!):** Since everything in SciPipe is plain Go(lang), you can easily use the [gdb debugger](http://golang.org/doc/gdb) (preferrably
  with the [cgdb interface](https://www.youtube.com/watch?v=OKLR6rrsBmI) for easier use) to step through your program at any detail, as well as all
  the other excellent debugging tooling for Go (See eg [delve](https://github.com/derekparker/delve) and [godebug](https://github.com/mailgun/godebug)),
  or just use `println()` statements at any place in your code. In addition, you can easily
  turn on very detailed debug output from SciPipe's execution itself, by just turning on debug-level
  logging with `scipipe.InitLogDebug()` in your `main()` method.
- **Efficient:** Workflows are compiled into static compiled code, that runs fast.<commit_msg>Better GIF in README<commit_after># SciPipe

An experimental library for writing Scientific (Batch) Workflows in vanilla [Go(lang)](http://golang.org),
based on an idea for a flow-based like pattern in pure Go, as presented in
[this Gopher Academy blog post](http://blog.gopheracademy.com/composable-pipelines-pattern).

See a short demo about how it might look, working with SciPipe in the terminal:

![SciPipe Demo](http://i.imgur.com/bFPFWjZ.gif)

## Benefits

Some benefits of SciPipe, that are not always available in other systems available:

- **Easy-to-grasp behaviour:** Data flowing through a network.
- **Parallel:** Apart from the inherent pipeline parallelism, SciPipe processes also spawn multiple parallel tasks when the same process has multiple inputs.
- **Concurrent:** Each process runs in an own light-weight thread, and is not blocked by
  operations in other processes, except when waiting for inputs from upstream processes.
- **Inherently simple:** Uses Go's concurrency primitives (go-routines and channels)
  to create an "implicit" scheduler, which means very little additional infrastructure code.
  This means that the code is easy to modify and extend.
- **Flexible:** Processes that wrap command-line programs and scripts can be combined with
  processes coded directly in Golang.
- **Custom file naming:** SciPipe gives you full control over how file names are produced,
  making it easy to understand and find your way among the output files of your computations.
- **Highly Debuggable(!):** Since everything in SciPipe is plain Go(lang), you can easily use the [gdb debugger](http://golang.org/doc/gdb) (preferrably
  with the [cgdb interface](https://www.youtube.com/watch?v=OKLR6rrsBmI) for easier use) to step through your program at any detail, as well as all
  the other excellent debugging tooling for Go (See eg [delve](https://github.com/derekparker/delve) and [godebug](https://github.com/mailgun/godebug)),
  or just use `println()` statements at any place in your code. In addition, you can easily
  turn on very detailed debug output from SciPipe's execution itself, by just turning on debug-level
  logging with `scipipe.InitLogDebug()` in your `main()` method.
- **Efficient:** Workflows are compiled into static compiled code, that runs fast.
{'fraction_non_alphanumeric': 0.08524590163934426, 'fraction_numerical': 0.0006557377049180328, 'mean_word_length': 4.821882951653944, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4053943', 'n_tokens_mistral': 1264, 'n_tokens_neox': 1210, 'n_words': 590}