<commit_before>+++
author = ["<NAME>"]
date = "2015-12-18T00:00:00-08:00"
title = "Composable Pipelines Improved"
series = ["Advent 2015"]
+++

I wrote a [post here on GopherAcademy](https://blog.gopheracademy.com/composable-pipelines-pattern) earlier this year,
about an idea for a "framework-less" pattern for [Flow-Based Programming](http://www.jpaulmorrison.com/fbp/) style programs in Go,
or let's just call it "composable concurrent pipelines". During the year, I have experimented more, and added some minor modifications to this pattern, which
I describe below.

## The old way

The basic idea described in that earlier post was the following: Rather to do [like in the generator "pattern"](https://talks.golang.org/2012/concurrency.slide#25),
where concurrent processes were packed into functions that returns the output channels on which their lazy-evaluated output will be provided,
we instead store each process into structs, with each in-port and out-port (I use the term port from Flow-based programming here, though technically in Go, it
is in this case just a channel stored in a named struct field) as separate struct fields, and the main code of the processes in a
go-routine, to make it concurrent.

The basic approach has the benefits that the code for routing out-ports to in-ports becomes a lot clearer
than with the generator pattern, since each port is handled separately, even on separate lines of code.

An unnecessary feat of that provided pattern though, was using methods for each out-ports, that would ensure that a channel
is created for that out-port, and then returning that channel. The pattern proposed that a process would be defined like so:

```go
type AProcess struct {
	In  string
	Out string
}

func (p *AProcess) OutChan() chan string {
	p.Out = make(chan []byte, 16)
	return p.Out
}

func (p *AProcess) Init() {
	go func() {
		for line := range p.In {
			// Do something with `line` ...
		}
	}()
}
```

Connecting such processes together would go something like:

```go
proc1 := &AProcess{}
proc2 := &AProcess{}
// Connect
proc2.In = proc1.OutChan()
```
<commit_msg>Re-write of first paragraph under "The old way"<commit_after>+++
author = ["<NAME>"]
date = "2015-12-18T00:00:00-08:00"
title = "Composable Pipelines Improved"
series = ["Advent 2015"]
+++

I wrote a [post here on GopherAcademy](https://blog.gopheracademy.com/composable-pipelines-pattern) earlier this year,
about an idea for a "framework-less" pattern for [Flow-Based Programming](http://www.jpaulmorrison.com/fbp/) style programs in Go,
or let's just call it "composable concurrent pipelines". During the year, I have experimented more, and added some minor modifications to this pattern, which
I describe below.

## The old way

The basic idea in that earlier post was to expand on the [generator pattern described in a slide by <NAME>](https://talks.golang.org/2012/concurrency.slide#25) by storing the concurrent processes in structs rather than just functions. This allows respresenting in-port and out-ports as struct fields, that can be used to connect in-ports and out-ports of multiple processes in a more fluent way.

I have realized some further simplifications of this pattern though. One unnecessary thing suggested in that post was to use a method for each out-port, that would ensure that a channel is created for that out-port before returning it. 

The pattern proposed that a process would be defined like so:

```go
type AProcess struct {
	In  string
	Out string
}

func (p *AProcess) OutChan() chan string {
	p.Out = make(chan []byte, 16)
	return p.Out
}

func (p *AProcess) Init() {
	go func() {
		for line := range p.In {
			// Do something with `line` ...
		}
	}()
}
```

Connecting such processes together would go something like:

```go
proc1 := &AProcess{}
proc2 := &AProcess{}
// Connect
proc2.In = proc1.OutChan()
```

{'fraction_non_alphanumeric': 0.09385532797511019, 'fraction_numerical': 0.017630282603059374, 'mean_word_length': 4.3509015256588075, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3952249', 'n_tokens_mistral': 1187, 'n_tokens_neox': 1093, 'n_words': 518}