<commit_before># SciPipe

<img src="images/fbp_factory.png" style="width: 30%; float: right; margin: 0 .4em;">

## Scientific Workflow Library written in Go

<big>Run complex series of shell commands, or pipelines, in a flexible, robust & reproducible manner</big>

## Why SciPipe?

- **Intuitive:** SciPipe operates by flowing data (files) through a
  network of channels and processes
- **Flexible:** Wrapped command-line programs or scripts can be combined with
  processes coded in Go
- **Convenient:** Full control over how your files are named
- **Efficient:** Workflows consist of statically compiled code that runs fast
- **Parallel:** Pipeline paralellism between processes as well as task
  parallelism for multiple inputs, making efficient use of multiple CPU cores
- **Supports streaming:** Can stream outputs via UNIX named pipes
- **Easy to debug:** Use available debugging tools or even just `println()`
- **Portable:** Workflows can be distributed either as Go code to be run with
  `go run`, or as stand-alone executable files

<strong><small>Project links: [GitHub repo](http://github.com/scipipe/scipipe) | [Issue Tracker](https://github.com/scipipe/scipipe/issues) | [Chat](https://gitter.im/scipipe/scipipe)</small></strong>

[![Build Status](https://img.shields.io/circleci/project/github/scipipe/scipipe.svg)](https://circleci.com/gh/scipipe/scipipe)
[![Test Coverage](https://img.shields.io/codecov/c/github/scipipe/scipipe.svg)](https://codecov.io/gh/scipipe/scipipe)
[![Codebeat Grade](https://codebeat.co/badges/96e93624-2ac8-42c9-9e94-2d6e5325d8ff)](https://codebeat.co/projects/github-com-scipipe-scipipe-master)
[![Go Report Card](https://goreportcard.com/badge/github.com/scipipe/scipipe)](https://goreportcard.com/report/github.com/scipipe/scipipe)
[![GoDoc](https://godoc.org/github.com/scipipe/scipipe?status.svg)](https://godoc.org/github.com/scipipe/scipipe)
[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/scipipe/scipipe)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1157941.svg)](https://doi.org/10.5281/zenodo.1157941)

<div style="width:37%; float: right; font-size: 0.8em; padding: 1em; margin: 0 0 1em 1em;">
<h4>Project updates</h4>
<strong>NEW: Scientific study using SciPipe:</strong><br><a href="https://doi.org/10.3389/fphar.2018.01256">Predicting off-target binding profiles with confidence using Conformal Prediction</a><br>
<strong>NEW: Slides:</strong> <a href="https://pharmb.io/blog/saml-gostockholm2018/)">Presentation on SciPipe and more at Go Stockholm Conference</a><br>
<strong>Preprint paper on SciPipe:</strong><br><a href="https://www.biorxiv.org/content/early/2018/08/01/380808) - explaining the background, motivations and technical aspects in depth. Freely available via bioRxiv.</strong>">SciPipe - A workflow library for agile development of complex and dynamic bioinformatics pipelines</a><br>
<strong>Blog post:</strong> <a href="http://bionics.it/posts/provenance-reports-in-scientific-workflows) - going into details about how SciPipe is addressing provenance">Provenance reports in Scientific Workflows</a><br>
<strong>Blog post:</strong> <a href="http://bionics.it/posts/first-production-workflow-run-with-scipipe">First production workflow run with SciPipe</a>
</div>

## Introduction

When you need to run many commandline programs that depend on each other in
complex ways, SciPipe helps by making the process of running these programs
flexible, robust and reproducible. SciPipe also lets you restart an interrupted
run without over-writing already produced output and produces an audit report
of what was run, among many other things.

SciPipe is built on the proven principles of [Flow-Based Programming](https://en.wikipedia.org/wiki/Flow-based_programming)
(FBP) to achieve maximum flexibility, productivity and agility when designing
workflows.  Compared to plain dataflow, FBP provides the benefits that
processes are fully self-contained, so that a library of re-usable components
can be created, and plugged into new workflows ad-hoc.

Similar to other FBP systems, SciPipe workflows can be likened to a network of
assembly lines in a factory, where items (files) are flowing through a network
of conveyor belts, stopping at different independently running stations
(processes) for processing, as depicted in the picture above.

SciPipe was initially created for problems in bioinformatics and
cheminformatics, but works equally well for any problem involving pipelines of
commandline applications.

**Project status:** SciPipe is still alpha software and minor breaking API
changes still happens as we try to streamline the process of writing workflows.
Please follow the commit history closely for any API updates if you have code
already written in SciPipe (Let us know if you need any help in migrating code
to the latest API).

## Known limitations

- There are still a number of missing good-to-have features for workflow
  design. See the [issue tracker](https://github.com/scipipe/scipipe/issues)
  for details.
- There is not (yet) support for the [Common Workflow Language](http://common-workflow-language.github.io).

## Hello World example<commit_msg>Add install section on frontpage<commit_after># SciPipe

<img src="images/fbp_factory.png" style="width: 30%; float: right; margin: 0 .4em;">

## Scientific Workflow Library written in Go

<big>Run complex series of shell commands, or pipelines, in a flexible, robust & reproducible manner</big>

## Why SciPipe?

- **Intuitive:** SciPipe operates by flowing data (files) through a
  network of channels and processes
- **Flexible:** Wrapped command-line programs or scripts can be combined with
  processes coded in Go
- **Convenient:** Full control over how your files are named
- **Efficient:** Workflows consist of statically compiled code that runs fast
- **Parallel:** Pipeline paralellism between processes as well as task
  parallelism for multiple inputs, making efficient use of multiple CPU cores
- **Supports streaming:** Can stream outputs via UNIX named pipes
- **Easy to debug:** Use available debugging tools or even just `println()`
- **Portable:** Workflows can be distributed either as Go code to be run with
  `go run`, or as stand-alone executable files

<strong><small>Project links: [GitHub repo](http://github.com/scipipe/scipipe) | [Issue Tracker](https://github.com/scipipe/scipipe/issues) | [Chat](https://gitter.im/scipipe/scipipe)</small></strong>

[![Build Status](https://img.shields.io/circleci/project/github/scipipe/scipipe.svg)](https://circleci.com/gh/scipipe/scipipe)
[![Test Coverage](https://img.shields.io/codecov/c/github/scipipe/scipipe.svg)](https://codecov.io/gh/scipipe/scipipe)
[![Codebeat Grade](https://codebeat.co/badges/96e93624-2ac8-42c9-9e94-2d6e5325d8ff)](https://codebeat.co/projects/github-com-scipipe-scipipe-master)
[![Go Report Card](https://goreportcard.com/badge/github.com/scipipe/scipipe)](https://goreportcard.com/report/github.com/scipipe/scipipe)
[![GoDoc](https://godoc.org/github.com/scipipe/scipipe?status.svg)](https://godoc.org/github.com/scipipe/scipipe)
[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/scipipe/scipipe)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1157941.svg)](https://doi.org/10.5281/zenodo.1157941)

<div style="width:37%; float: right; font-size: 0.8em; padding: 1em; margin: 0 0 1em 1em;">
<h4>Project updates</h4>
<strong>NEW: Scientific study using SciPipe:</strong><br><a href="https://doi.org/10.3389/fphar.2018.01256">Predicting off-target binding profiles with confidence using Conformal Prediction</a><br>
<strong>NEW: Slides:</strong> <a href="https://pharmb.io/blog/saml-gostockholm2018/)">Presentation on SciPipe and more at Go Stockholm Conference</a><br>
<strong>Preprint paper on SciPipe:</strong><br><a href="https://www.biorxiv.org/content/early/2018/08/01/380808) - explaining the background, motivations and technical aspects in depth. Freely available via bioRxiv.</strong>">SciPipe - A workflow library for agile development of complex and dynamic bioinformatics pipelines</a><br>
<strong>Blog post:</strong> <a href="http://bionics.it/posts/provenance-reports-in-scientific-workflows) - going into details about how SciPipe is addressing provenance">Provenance reports in Scientific Workflows</a><br>
<strong>Blog post:</strong> <a href="http://bionics.it/posts/first-production-workflow-run-with-scipipe">First production workflow run with SciPipe</a>
</div>

## Introduction

When you need to run many commandline programs that depend on each other in
complex ways, SciPipe helps by making the process of running these programs
flexible, robust and reproducible. SciPipe also lets you restart an interrupted
run without over-writing already produced output and produces an audit report
of what was run, among many other things.

SciPipe is built on the proven principles of [Flow-Based Programming](https://en.wikipedia.org/wiki/Flow-based_programming)
(FBP) to achieve maximum flexibility, productivity and agility when designing
workflows.  Compared to plain dataflow, FBP provides the benefits that
processes are fully self-contained, so that a library of re-usable components
can be created, and plugged into new workflows ad-hoc.

Similar to other FBP systems, SciPipe workflows can be likened to a network of
assembly lines in a factory, where items (files) are flowing through a network
of conveyor belts, stopping at different independently running stations
(processes) for processing, as depicted in the picture above.

SciPipe was initially created for problems in bioinformatics and
cheminformatics, but works equally well for any problem involving pipelines of
commandline applications.

**Project status:** SciPipe is still alpha software and minor breaking API
changes still happens as we try to streamline the process of writing workflows.
Please follow the commit history closely for any API updates if you have code
already written in SciPipe (Let us know if you need any help in migrating code
to the latest API).

## Known limitations

- There are still a number of missing good-to-have features for workflow
  design. See the [issue tracker](https://github.com/scipipe/scipipe/issues)
  for details.
- There is not (yet) support for the [Common Workflow Language](http://common-workflow-language.github.io).

## Installing

1. [Download](https://golang.org/dl/) and [install](https://golang.org/doc/install) Go
2. Run the following command, to install the scipipe Go library (don't miss the trailing dots!):

```bash
go get github.com/scipipe/scipipe/...
```

## Hello World example
{'fraction_non_alphanumeric': 0.11453039713909278, 'fraction_numerical': 0.018633540372670808, 'mean_word_length': 5.2993479549496145, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 79, 'https://': 44, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2040591', 'n_tokens_mistral': 3374, 'n_tokens_neox': 3115, 'n_words': 1071}