<commit_before># Architecture Principles

- Keep it Simple, Stupid
- Don't Repeat Yourself
- Follow the Unix Philosophy:

  > Write programs that do one thing, and do it well. Write programs that work together.    
  > -- http://www.linfo.org/unix_philosophy.html

  Each program being a micro-service of the complete picture.
- Each micro-service:
    - Has its own git repo
    - Has its own versioning/release 
    - Exposes:
        - Metrics
        - Logging
        - API
    - Packaged as a single executable jar
    - Installable as you would with any other application, via an OS-dependant package, incl:
        - its configuration
        - its command script
        - its OS dependend "start as daemon" scripts (eg. unix rc.d scripts)

## Todo
- Determine universal dataformat and protocol for the micro-services to communicate, candidates are:
    - text streams
    - atom/json over http
    - ..?

# Design Roadmap

## v1
Starting with a single application that does it all in a single process, calling external commands from the `Runner`.
```
 +---------------------------------------------------+
 |  Client                                           |
 |---------------------------------------------------|
 |                        +----+                     |
 |  +------------+  uses  |    | uses  +----------+  |
 |  | Groovy DSL |<-------+Main+------>|  Output  |  |
 |  +--------+---+        +-+--+       +----------+  |
 |           |              |                        |
 |           |uses          |uses                    |
 |           |              |                        |
 |  +--------v--------------v---------------------+  |
 |  |                Command-Query API            |  |
 |  +---------------------------------------------+  |
 |                          ^                        |
 |                          |impl.                   |
 |                          |                        |
 |  +-----------------------+---------------------+  |
 |  |                     Runner                  |  |
 |  +---------------------------------------------+  |
 +---------------------------------------------------+
```

## v2
Start moving out the `Runner` to its own application, introducing a `Coordinator` to coordinate the runner processes.
```
 +----------------------------------------+
 |  Client                                |
 |----------------------------------------|
 |                  +----+                |
 |  +------------+  |    |  +----------+  |
 |  | Groovy DSL |  |Main|  |  Output  |  |
 |  +------------+  |    |  +----------+  |
 |                  |    |                |
 |  +---------------+----+-------------+  |
 |  |          Command-Query API       |  |
 |  +-----------------+----------------+  |
 |                    |                   |
 |  +-----------------v----------------+  |
 |  |            Coordinator           |  |
 |  +-----------------+----------------+  |
 +--------------------|-------------------+
                      |
    +-----------------v----------------+
    |              Runner(s)           |-+
    +----------------------------------+ |+
     +-----------------------------------+|
      +-----------------------------------+
```

## v3
Move out the `Coordinator` to its own application as it proved to work in v2.
```
 +----------------------------------------+
 |  Client                                |
 |----------------------------------------|
 |                  +----+                |
 |  +------------+  |    |  +----------+  |
 |  | Groovy DSL |  |Main|  |  Output  |  |
 |  +------------+  |    |  +----------+  |
 |                  |    |                |
 |  +---------------+----+-------------+  |
 |  |          Command-Query API       |  |
 |  +-----------------+----------------+  |
 |                    |                   |
 +--------------------|-------------------+
                      |
    +-----------------v----------------+
    |            Coordinator           |
    +-----------------+----------------+
                      |
    +-----------------v----------------+
    |              Runner(s)           |-+
    +----------------------------------+ |+
     +-----------------------------------+|
      +-----------------------------------+
```

## v..
...

(diagrams simply drawn with http://www.asciiflow.com)<commit_msg>Added micro-services overview idea to work towards<commit_after># Architecture Principles

- Keep it Simple, Stupid
- Don't Repeat Yourself
- Follow the Unix Philosophy:

  > Write programs that do one thing, and do it well. Write programs that work together.    
  > -- http://www.linfo.org/unix_philosophy.html

  Each program being a micro-service of the complete picture.
- Each micro-service:
    - Has its own git repo
    - Has its own versioning/release 
    - Exposes:
        - Metrics
        - Logging
        - API
    - Packaged as a single executable jar
    - Installable as you would with any other application, via an OS-dependant package, incl:
        - its configuration
        - its command script
        - its OS dependend "start as daemon" scripts (eg. unix rc.d scripts)

## Todo
- Determine universal dataformat and protocol for the micro-services to communicate, candidates are:
    - text streams
    - atom/json over http
    - ..?

# Design Roadmap

Namings in this roadmap are work-in-progress, just like the designs

## v1
Starting with a single application that does it all in a single process, calling external commands from the `Runner`.
```
 +---------------------------------------------------+
 |  Client                                           |
 |---------------------------------------------------|
 |                        +----+                     |
 |  +------------+  uses  |    | uses  +----------+  |
 |  | Groovy DSL |<-------+Main+------>|  Output  |  |
 |  +--------+---+        +-+--+       +----------+  |
 |           |              |                        |
 |           |uses          |uses                    |
 |           |              |                        |
 |  +--------v--------------v---------------------+  |
 |  |                Command-Query API            |  |
 |  +---------------------------------------------+  |
 |                          ^                        |
 |                          |impl.                   |
 |                          |                        |
 |  +-----------------------+---------------------+  |
 |  |                     Runner                  |  |
 |  +---------------------------------------------+  |
 +---------------------------------------------------+
```
(now called `pipe` or `pipe-runner`)

## v2
Start moving out the `Runner` to its own application, introducing a `Coordinator` to coordinate the runner processes.
```
 +----------------------------------------+
 |  Client                                |
 |----------------------------------------|
 |                  +----+                |
 |  +------------+  |    |  +----------+  |
 |  | Groovy DSL |  |Main|  |  Output  |  |
 |  +------------+  |    |  +----------+  |
 |                  |    |                |
 |  +---------------+----+-------------+  |
 |  |          Command-Query API       |  |
 |  +-----------------+----------------+  |
 |                    |                   |
 |  +-----------------v----------------+  |
 |  |            Coordinator           |  |
 |  +-----------------+----------------+  |
 +--------------------|-------------------+
                      |
    +-----------------v----------------+
    |              Runner(s)           |-+
    +----------------------------------+ |+
     +-----------------------------------+|
      +-----------------------------------+
```

## v3
Move out the `Coordinator` to its own application as it proved to work in v2.
```
 +----------------------------------------+
 |  Client                                |
 |----------------------------------------|
 |                  +----+                |
 |  +------------+  |    |  +----------+  |
 |  | Groovy DSL |  |Main|  |  Output  |  |
 |  +------------+  |    |  +----------+  |
 |                  |    |                |
 |  +---------------+----+-------------+  |
 |  |          Command-Query API       |  |
 |  +-----------------+----------------+  |
 |                    |                   |
 +--------------------|-------------------+
                      |
    +-----------------v----------------+
    |            Coordinator           |
    +-----------------+----------------+
                      |
    +-----------------v----------------+
    |              Runner(s)           |-+
    +----------------------------------+ |+
     +-----------------------------------+|
      +-----------------------------------+
```

## v..
```
                                         +-----------------------+
                                         |      pipe-poll        |
                                         |-----------------------|
                                         |              CSV      |
                                         |      SVN+-+    +      |
                                         |           |    |      |
                 Gitlab webhook          |           v    v      |
                         +               |       +--+Providers   |
                         |               |       |               |
        GitHub webhook   |               |  +-----------+        |
                   +     |  +--------------+| pipe-poll |        |
                   |     |  |            |  +-----------+        |
                   +--+  |  |            +-----------------------+
                      |  |  |
          +-----------|--|--|------------------------------------------+
          |           |  |  |    pipe-listen                           |
          |-----------|--|--|------------------------------------------|
          |           |  |  |                                          |
          |           v  v  v                                          |
          |          +-------------+                                   |
          |          | pipe-listen +-------------+Providers            |
          |          +--------+----+               +  +  +             |
          |                   |                +---+  |  +-----+       |
          |                   |                |      |        v       |
          |                   |                v      |    Pipe-poller |
          |                   |             GitHub    |                |
          |                   |                       v                |
          |                   |                      Gitlab            |
          |                   |                                        |
          +-------------------|----------------------------------------+
                              |
                              |                       +-----------------------------------------+
                              |                       |             pipe-monitor                |
                              v                       |-----------------------------------------|
          +----------------------------------+        | +---------------+                       |
          |    event-queue / message-bus     |<---------+ pipe-monitor  +-----+Publishers       |
          +----------------------------------+        | +---------------+         +    +        |
                  ^                    ^              |                           |    v        |
                  |                    |              |                           v    Graphite |
                  |                    |              |                       Ganglia           |
 +----------------|---------+          |              +-----------------------------------------+
 | pipe-messenger |         |          |
 |----------------|---------|    +-----|-----------------------------------+
 |        +-------+------+  |    |     |       pipe-worker                 |
 |        |pipe-messenger|  |    |-----|-----------------------------------|
 |        +-----+--------+  |    |     |                                   |
 |              |           |    |  +--+--------+                          |
 |              +           |    |  |pipe-worker+-------+scm-providers     |
 |          Publishers      |    |  +-+---------+            +  +  +       |
 |          + +   +  +      |    |    |                    +-+  |  +-+     |
 |       +--+ |   |  +-+    |    |    |                    v    |    v     |
 |       v    |   |    v    |    |    |                  Git    v    CSV   |
 |  Campfire  |   |   Email |    |    |                        SVN         |
 |            |   |         |    |    |                                    |
 |            v   v         |    +----|------------------------------------+
 |       Gitlab  GitHub     |         |
 +--------------------------+         |
                                   +--|---------------------+
                                   |  | pipe-runner         |
                                   |--|---------------------|
                                   |  |                     |
                                   |  +>                    |
                                   |                        |
                                   +------------------------+
```

(diagrams simply drawn with http://www.asciiflow.com)
{'fraction_non_alphanumeric': 0.3342696629213483, 'fraction_numerical': 0.0005913660555884093, 'mean_word_length': 0.37085824298307835, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 107, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '909049', 'n_tokens_mistral': 2946, 'n_tokens_neox': 2666, 'n_words': 483}