<commit_before># piper

`piper` is a manifest-based build system which is completely controlled from
configuration files local to a repository.

## Installation

As with any good Python project; `sudo pip install piper`.

## Rationale

*"Whatever you do, don't ever build a new build system. That would be the worst
possible thing to do."* - one of my managers, not realizing that telling
a hacker what they cannot do is the most inspiring thing that exists.

This is a project inspired by working with different build systems full-time
for a bit longer than a year. After trying to fit increasingly blob shaped pegs
into increasingly opinionated holes, a couple of thoughts started to form:

* Storing configuration outside of the repository is ridiculous.
* The system needs to be runnable locally in the same manner as it would happen
  on any build agent, at least in a dry-run mode.
* Inheritable templates are not the way to go, not at scale.
* [It can't be that hard][naivete] to build a pipeline - the difficulty of
  the problem is overstated.

## Design

The meat of the application lies in the [YAML][yaml] configuration file
`piper.yml`, found in the root of the repository. Its contents dictate what
`piper` will do in its execution steps and in which order.

### Piper()

The main executor of the `piper` pipeline is the `Piper()` class. It's
a singleton (shut up) that reads the configuration and uses that configuration
to set and host a couple of other classes:

* `Env()`: Sets and tears the execution env. The env can be a temporary
  directory, an AWS machine, a schroot or something of the sort. Can optionally
  provide a wrapper method for `Step()` that all execution will run through,
  e.g. `schroot -c 'squeeze-amd64' -- {0}` to run something in a schroot.
* `Step()`: Executes a build step. `Piper()` has a list of these and will
  execute them in order. Each step can specify upstream and downstream steps
  that will deterministically build a graph of steps.
* `Notifier()`: Handles notifications and logging.

These classes are all written to be subclassed for more specific cases, such as
a `PythonStep()` class that builds Python projects, a `IRCNotifier()` class
that does just that, etc.


[naivete]: http://thiderman.org/posts/naivete/
[yaml]: http://www.yaml.org/<commit_msg>Update README link for hardness :dash:<commit_after># piper

`piper` is a manifest-based build system which is completely controlled from
configuration files local to a repository.

## Installation

As with any good Python project; `sudo pip install piper`.

## Rationale

*"Whatever you do, don't ever build a new build system. That would be the worst
possible thing to do."* - one of my managers, not realizing that telling
a hacker what they cannot do is the most inspiring thing that exists.

This is a project inspired by working with different build systems full-time
for a bit longer than a year. After trying to fit increasingly blob shaped pegs
into increasingly opinionated holes, a couple of thoughts started to form:

* Storing configuration outside of the repository is ridiculous.
* The system needs to be runnable locally in the same manner as it would happen
  on any build agent, at least in a dry-run mode.
* Inheritable templates are not the way to go, not at scale.
* [It can't be that hard][hard] to build a pipeline - the difficulty of
  the problem is overstated.

## Design

The meat of the application lies in the [YAML][yaml] configuration file
`piper.yml`, found in the root of the repository. Its contents dictate what
`piper` will do in its execution steps and in which order.

### Piper()

The main executor of the `piper` pipeline is the `Piper()` class. It's
a singleton (shut up) that reads the configuration and uses that configuration
to set and host a couple of other classes:

* `Env()`: Creates and tears the execution env. The env can be a temporary
  directory, an AWS machine, a schroot or something of the sort. Can optionally
  provide a wrapper method for `Step()` that all execution will run through,
  e.g. `schroot -c 'squeeze-amd64' -- {0}` to run something in a schroot.
* `Step()`: Executes a build step. `Piper()` has a list of these and will
  execute them in order. Each step can specify upstream and downstream steps
  that will deterministically build a graph of steps.
* `Notifier()`: Handles notifications and logging.

These classes are all written to be subclassed for more specific cases, such as
a `PythonStep()` class that builds Python projects, a `IRCNotifier()` class
that does just that, etc.


[hard]: https://github.com/thiderman/talk-hard
[yaml]: http://www.yaml.org/
{'fraction_non_alphanumeric': 0.0680008607703895, 'fraction_numerical': 0.0012911555842479018, 'mean_word_length': 4.199105145413871, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1607007', 'n_tokens_mistral': 1265, 'n_tokens_neox': 1201, 'n_words': 722}