<commit_before>Nodus
============================================================================================================================

_(WARNING: EXPERIMENTAL)_

Description
------------------------------------------------------------------------------------------------------------------------------

Framework for [parallel](http://en.wikipedia.org/wiki/Parallelization)
[data-flow](http://en.wikipedia.org/wiki/Dataflow) based applications.


It is influenced by and similar to:

 * [Kahn Process Networks](http://en.wikipedia.org/wiki/Kahn_process_networks)
 * [Algorithmic Skeletons](http://en.wikipedia.org/wiki/Algorithmic_skeleton) (see also [here](https://github.com/ParaPhrase/skel))
 * [Iteratee IO](http://okmij.org/ftp/Streams.html)
 * And of course the hundreds of other reinventions with names like
   `/(((data|signal|packet)?(stream|flow)|pipe(line)?|(flow|event|signal|reactive)-(processing|programming|architecture|computing|language)/`
   combined with parallelization, concurrency, clustering paradigms, etc. etc...


### In More Detail ##################################


- [x] Guiding assumptions
  - [x] Steady-state
  - [x] Composable
  - [x] Proportionate
- [ ] Simple single-stream sequential
  - [ ] URI-like notation
  - [ ] Token state accumulation
- [ ] Single-stream sequential with explicit sink
- [ ] Single-stream parallel
  - [ ] Branches
  - [ ] Forks
  - [ ] Waypoints
  - [ ] Pattern Branching
- [ ] Multi-stream
- [ ] Getting away with duck-typing



#### Guiding Assumptions

This library and associated commandline tools are most appropriate for these types of (overlapping and somewhat
redundant) problems & constraints:

|        |      |
| ------ | ---- |
| __Dataflow-Oriented__ | Problems where the easiest way to look at it is a (possibly branching) pipeline of operations on a stream of data. |
| __Steady-State__ | Where the processing nodes and overall application have upper bounds on their memory requirements in order to safely and reliably handle very long running streams. |
| __Functional__   | Most of the generator & processing nodes are usually assumed to be side-effect-free and pure (at least given all previous inputs). |
| __Composable__   | Easy to make nodes out of combinations and networks of other nodes |
| __Proportionate__| Very easy and fast to do a simple pipeline (for example some simple functions that mutate an integer from within the console), but easily scales up to more complex production-ready solutions/projects. It strives to maintain the following inequality:  `effort ≤ problem‐complexity`. |

It is additionally tuned for (but doesn't assume) problems with the following properties:


|        |      |
| ------ | ---- |
| __Parallel__          | For example, map-reduce type problems, or wherever strictly sequential simply isn't required.  |
| __Ordered__           | When there is a natural ordering to the tokens. e.g., a time-series, or bytes from most IO streams; |
| __State-Accumulation__| Where the tokens can accumulate state and where processing nodes can look at previous process results relative to that token as they pass through the graph. As opposed to destructive changes to the token at each process or lack of tokens altogether (like simple functions). |
| __Multi-Stream__      | Potentially create secondary streams that are unsynchronized with the original stream; |
| __Coordinated__       | Dataflow type problems where there is a need to synchronize/coordinate multiple orthogonal streams; |
| __Decoupled__         | A way to cache intermediate results (e.g., half way through the pipeline) so that multiple stream processing applications can be simultaneously writing results and other unrelated processes reading a stream of results- sometimes in sync with when they are being written by the decoupled process. (some good examples to come). |
| __Daemonized__        | Where certain stream sources and graphs of processors and sinks should be managed as a single long-lived system process when desired. |
| __Simulations/Reruns__| Persistent caching nodes (nexus decouplers) etc. allow one to easily simulate reruns of past parts of a stream- possibly generating a new version of subsequently persisted results. |

Components
------------------------------------------------------------------------------------------------------------------------------

### Data #############################################

| Thing     | Otherwise Known As |                                                    |
|-----------|--------------------|----------------------------------------------------------|
| Stream    | Flow, Signal                  | Related ordered data representation- bundled into tokens (chunks). A gradually consumed state |
| Token     | Chunk, Element, Event, Sample | Coherent instance of stream data being mutated and passing through the pipelines (potentially in parallel) - special values of EOF & Exceptionals |

#### Stream

Defined by:
  * The eventual property distribution of its constituent tokens (possibly given as a shorthand name/"token-type"),
    which, because the token's internal state ends up mirroring the operations that have occurred, means it is also a<commit_msg>Minor README polish<commit_after>Nodus
============================================================================================================================

_(WARNING: EXPERIMENTAL)_

Description
------------------------------------------------------------------------------------------------------------------------------

Framework for [parallel](http://en.wikipedia.org/wiki/Parallelization)
[data-flow](http://en.wikipedia.org/wiki/Dataflow) based applications.


It is influenced by and similar to:

 * [Kahn Process Networks](http://en.wikipedia.org/wiki/Kahn_process_networks)
 * [Algorithmic Skeletons](http://en.wikipedia.org/wiki/Algorithmic_skeleton) (see also [here](https://github.com/ParaPhrase/skel))
 * [Iteratee IO](http://okmij.org/ftp/Streams.html)
 * And of course the hundreds of other reinventions with names like
   `/(((data|signal|packet)?(stream|flow)|pipe(line)?|(flow|event|signal|reactive)-(processing|programming|architecture|computing|language)/`
   combined with parallelization, concurrency, clustering paradigms, etc. etc...


### In More Detail ##################################


- [x] Guiding assumptions
  - [x] Steady-state
  - [x] Composable
  - [x] Proportionate
- [ ] Simple single-stream sequential
  - [ ] URI-like notation
  - [ ] Token state accumulation
- [ ] Single-stream sequential with explicit sink
- [ ] Single-stream parallel
  - [ ] Branches
  - [ ] Forks
  - [ ] Waypoints
  - [ ] Pattern Branching
- [ ] Multi-stream
- [ ] Getting away with duck-typing



#### Guiding Assumptions

This library and associated commandline tools are most appropriate for these types of (overlapping and somewhat
redundant) problems & constraints:

|        |      |
| ------ | ---- |
| __Dataflow-Oriented__ | Problems where the easiest way to look at it is a (possibly branching) pipeline of operations on a stream of data. |
| __Steady-State__ | Where the processing nodes and overall application have upper bounds on their memory requirements in order to safely and reliably handle very long running streams. i.e., bounded online algorithms and [streaming-algorithms](http://en.wikipedia.org/wiki/Streaming_algorithm) |
| __Functional__   | Most of the generator & processing nodes are usually assumed to be side-effect-free and pure (at least given all previous inputs). |
| __Composable__   | Easy to make nodes out of combinations and networks of other nodes |
| __Proportionate__| Very easy and fast to do a simple pipeline (for example some simple functions that mutate an integer from within the console), but easily scales up to more complex production-ready solutions/projects. It strives to maintain the following inequality:  `effort ≤ problem‐complexity`. |

It is additionally tuned for (but doesn't assume) problems with the following properties:


|        |      |
| ------ | ---- |
| __Parallel__          | For example, map-reduce type problems, or wherever strictly sequential simply isn't required.  |
| __Ordered__           | When there is a natural ordering to the tokens. e.g., a time-series, or bytes from most IO streams; |
| __State-Accumulation__| Where the tokens can accumulate state and where processing nodes can look at previous process results relative to that token as they pass through the graph. As opposed to destructive changes to the token at each process or lack of tokens altogether (like simple functions). |
| __Multi-Stream__      | Potentially create secondary streams that are unsynchronized with the original stream; |
| __Coordinated__       | Dataflow type problems where there is a need to synchronize/coordinate multiple orthogonal streams; |
| __Decoupled__         | A way to cache intermediate results (e.g., half way through the pipeline) so that multiple stream processing applications can be simultaneously writing results and other unrelated processes reading a stream of results- sometimes in sync with when they are being written by the decoupled process. (some good examples to come). |
| __Daemonized__        | Where certain stream sources and graphs of processors and sinks should be managed as a single long-lived system process when desired. |
| __Simulations/Reruns__| Persistent caching nodes (nexus decouplers) etc. allow one to easily simulate reruns of past parts of a stream- possibly generating a new version of subsequently persisted results. |


#### _Example_: Single-Stream Sequential

    ┌───┐     ┌──────┐     ┌──────┐
    │ G │ ──> │ f(x) │ ──> │ g(x) │
    └───┘     └──────┘     └──────┘




---

Components
------------------------------------------------------------------------------------------------------------------------------

### Data #############################################

| Thing     | Otherwise Known As |                                                    |
|-----------|--------------------|----------------------------------------------------------|
| Stream    | Flow, Signal                  | Related ordered data representation- bundled into tokens (chunks). A gradually consumed state |
| Token     | Packet, Event, Chunk, Element, Sample | Coherent instance of stream data being mutated and passing through the pipelines (potentially in parallel) - special values of EOF & Exceptionals |

#### Stream

Defined by:
  * The eventual property distribution of its constituent tokens (possibly given as a shorthand name/"token-type"),
    which, because the token's internal state ends up mirroring the operations that have occurred, means it is also a
{'fraction_non_alphanumeric': 0.17720458143216314, 'fraction_numerical': 0.0, 'mean_word_length': 2.936950146627566, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2381274', 'n_tokens_mistral': 2591, 'n_tokens_neox': 2442, 'n_words': 1086}