<commit_before>+++
Author = ["<NAME>"]
date = "2015-01-30T00:00:00-08:00"
series = [""]
title = "Pattern for pluggable pipeline components in Go"
+++

# Patterns for composable concurrent pipelines in Go

## Generator function in Python

The author of this post came into Go from the python-world, which is quite prevalent the field of bioinformatics. In python one can easily write composable lazy-evauated pipelines of string processing operations using the generator syntax built into the language, which has been shown to both use less memory and to be a bit faster than their eager counterparts.

The Generator functionality in python basically means that you create a function that, rather than returning a single data structure once (say for example a list of items), rather will return a generator object which can later be iterated over by repeadetly calling its next method, or by the shorthand `for item in ...` syntax.

What is special about a generator compared to other *iterables* in python such as lists (which also support the `for item in ...` syntax), is that the generator function will start evaluating itself and yielding its output objects one by one only after iteration has started. Thus, minimal intermediate data is created and stored in memory.

To give an example, say that we have a file, chr_y.fa containing a little bit of the familiar A, C, G, T DNA nucleotides from the human Y chromosome, in the ubiquotous [FASTA file format](http://en.wikipedia.org/wiki/FASTA_format):

**chr_y.fa:**
````fasta
>Homo_sapiens.GRCh37.67.dna_rm.chromosome.Y
GTGATTGTTGTTAGGTTCTTAGCTGCTTCTGAAAAATGGGGTGATAATCTTAGAAGGACT
TGCTTCATGGGATGTGGTCCATAAAACTTCCTCTGCCCCAGTTGTAGGGCAGAAGACAAT
TTCTGTTACTGTAGTTTGGCCTTTTTTTGCAGAGATTCAGACATCTGTTTACTGACCTTA
GTTAAATTGTGACACTATGCCTAAAGGAGCCTGCAAGCTTTTATTTTTGCTCACTATGAA
GTCATCATTCAATTGTAAAATTTCCTTTTTAAGTTTCAGGTTGACTTAATGTCTGTCAAA
GCACAGTCTTTGGCAATAACAAAACAAACATATGCTGAATGAAAATGTTTAAGAGATGGA
TGACTATTTACTACTAAAAGAAGAAAAATTGGAAGAGAATAAAATGAAAACATGCATCTC
CTAAACCATATGNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN<commit_msg>Fixing title for the pipeline blog post<commit_after>+++
Author = ["<NAME>"]
date = "2015-01-30T00:00:00-08:00"
series = [""]
title = "Patterns for composable concurrent pipelines in Go"
+++

# Patterns for composable concurrent pipelines in Go

## Generator function in Python

The author of this post came into Go from the python-world, which is quite prevalent the field of bioinformatics. In python one can easily write composable lazy-evauated pipelines of string processing operations using the generator syntax built into the language, which has been shown to both use less memory and to be a bit faster than their eager counterparts.

The Generator functionality in python basically means that you create a function that, rather than returning a single data structure once (say for example a list of items), rather will return a generator object which can later be iterated over by repeadetly calling its next method, or by the shorthand `for item in ...` syntax.

What is special about a generator compared to other *iterables* in python such as lists (which also support the `for item in ...` syntax), is that the generator function will start evaluating itself and yielding its output objects one by one only after iteration has started. Thus, minimal intermediate data is created and stored in memory.

To give an example, say that we have a file, chr_y.fa containing a little bit of the familiar A, C, G, T DNA nucleotides from the human Y chromosome, in the ubiquotous [FASTA file format](http://en.wikipedia.org/wiki/FASTA_format):

**chr_y.fa:**
````fasta
>Homo_sapiens.GRCh37.67.dna_rm.chromosome.Y
GTGATTGTTGTTAGGTTCTTAGCTGCTTCTGAAAAATGGGGTGATAATCTTAGAAGGACT
TGCTTCATGGGATGTGGTCCATAAAACTTCCTCTGCCCCAGTTGTAGGGCAGAAGACAAT
TTCTGTTACTGTAGTTTGGCCTTTTTTTGCAGAGATTCAGACATCTGTTTACTGACCTTA
GTTAAATTGTGACACTATGCCTAAAGGAGCCTGCAAGCTTTTATTTTTGCTCACTATGAA
GTCATCATTCAATTGTAAAATTTCCTTTTTAAGTTTCAGGTTGACTTAATGTCTGTCAAA
GCACAGTCTTTGGCAATAACAAAACAAACATATGCTGAATGAAAATGTTTAAGAGATGGA
TGACTATTTACTACTAAAAGAAGAAAAATTGGAAGAGAATAAAATGAAAACATGCATCTC
CTAAACCATATGNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
{'fraction_non_alphanumeric': 0.045691020131192035, 'fraction_numerical': 0.009952499434517077, 'mean_word_length': 6.637305699481865, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 8, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2221539', 'n_tokens_mistral': 1516, 'n_tokens_neox': 1373, 'n_words': 487}