<commit_before># SciPipe

<img src="images/fbp_factory.png" style="width: 30%; float: right; margin: 0 .4em;">

## Scientific Workflow Library written in Go

<big>Run complex series of shell commands, or pipelines, in a flexible, robust & reproducible manner</big>

## Why SciPipe?

- **Intuitive:** SciPipe operates by flowing data (files) through a
  network of channels and processes
- **Flexible:** Wrapped command-line programs or scripts can be combined with
  processes coded in Go
- **Convenient:** Full control over how your files are named
- **Portable:** Workflows can be distributed either as Go code to be run with
  `go run`, or as stand-alone executable files
- **Easy to debug:** Use available debugging tools or even just `println()`
- **Supports streaming:** Can stream outputs via UNIX named pipes
- **Efficient:** Workflows consist of statically compiled code that runs fast
- **Parallel:** Pipeline paralellism between processes as well as task
  parallelism for multiple inputs, making efficient use of multiple CPU cores
<commit_msg>Re-order why-points<commit_after># SciPipe

<img src="images/fbp_factory.png" style="width: 30%; float: right; margin: 0 .4em;">

## Scientific Workflow Library written in Go

<big>Run complex series of shell commands, or pipelines, in a flexible, robust & reproducible manner</big>

## Why SciPipe?

- **Intuitive:** SciPipe operates by flowing data (files) through a
  network of channels and processes
- **Flexible:** Wrapped command-line programs or scripts can be combined with
  processes coded in Go
- **Convenient:** Full control over how your files are named
- **Efficient:** Workflows consist of statically compiled code that runs fast
- **Parallel:** Pipeline paralellism between processes as well as task
  parallelism for multiple inputs, making efficient use of multiple CPU cores
- **Supports streaming:** Can stream outputs via UNIX named pipes
- **Easy to debug:** Use available debugging tools or even just `println()`
- **Portable:** Workflows can be distributed either as Go code to be run with
  `go run`, or as stand-alone executable files

{'fraction_non_alphanumeric': 0.09159584513692162, 'fraction_numerical': 0.003777148253068933, 'mean_word_length': 4.886111111111111, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '52312', 'n_tokens_mistral': 582, 'n_tokens_neox': 545, 'n_words': 281}