<commit_before># piston_meta
A DSL parsing library for human readable text documents

[![Travis](https://img.shields.io/travis/PistonDevelopers/meta.svg?style=flat-square)](https://travis-ci.org/PistonDevelopers/meta)
[![Crates.io](https://img.shields.io/crates/v/piston_meta.svg?style=flat-square)](https://crates.io/crates/piston_meta)

[Documentation](https://docs.rs/piston_meta/)

[Why Piston-Meta?](https://github.com/PistonDevelopers/meta/issues/1)

[self-syntax](https://raw.githubusercontent.com/PistonDevelopers/meta/master/assets/self-syntax.txt)

### Rules

The meta language is used to describe how to read other documents.
First you define some strings to reuse, then some node rules.
The last node is used to read the entire document.

`20 document = [.l(string:"string") .l(node:"node") .w?]`

Strings starts with underscore and can be reused among the rules:

`_opt: "optional"`

Nodes start with a number that gets multiplied with 1000 and used as debug id.<commit_msg>Added introduction section to README<commit_after># piston_meta
A DSL parsing library for human readable text documents

[![Travis](https://img.shields.io/travis/PistonDevelopers/meta.svg?style=flat-square)](https://travis-ci.org/PistonDevelopers/meta)
[![Crates.io](https://img.shields.io/crates/v/piston_meta.svg?style=flat-square)](https://crates.io/crates/piston_meta)

[Documentation](https://docs.rs/piston_meta/)

[Why Piston-Meta?](https://github.com/PistonDevelopers/meta/issues/1)

[self-syntax](https://raw.githubusercontent.com/PistonDevelopers/meta/master/assets/self-syntax.txt)

### Introduction

Piston-Meta makes it easy to write parsers for human readable text documents.
It can be used for language design, custom formats and data driven development.

Meta parsing is a development technique that goes back to the first modern computer.
The idea is to turn pieces of a computer program into a programmable pipeline,
and thereby accelerating development.
An important, but surprisingly reusable part across projects, is the concept of generating
structured data from text, since text is easy to modify and reason about.

Most programs that work with text use the following pipeline:

```ignore
f : text -> data
```

The problem with this approach is that `f` changes from project to project,
and the task of transforming text into a data structure can get very complex.
For example, to create a parser for the syntax of a programming language,
one might need several thousands lines of code.
This slows down development and increases the chance of making errors.

Meta parsing is a technique where `f` gets splitted into two steps:

```ignore
f <=> f2 . f1
f1 : text -> meta data
f2 : meta data -> data
```

The first step `f1` takes text and converts it into meta data.
A DSL (Domain Specific Language) is used to describe how this transformation happens.
The second step `f2` converts meta data into data, and this is often written as code.

### Rules

The meta language is used to describe how to read other documents.
First you define some strings to reuse, then some node rules.
The last node is used to read the entire document.

`20 document = [.l(string:"string") .l(node:"node") .w?]`

Strings starts with underscore and can be reused among the rules:

`_opt: "optional"`

Nodes start with a number that gets multiplied with 1000 and used as debug id.
{'fraction_non_alphanumeric': 0.09777117384843982, 'fraction_numerical': 0.005943536404160475, 'mean_word_length': 4.8336221837088384, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 7, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3328417', 'n_tokens_mistral': 1033, 'n_tokens_neox': 953, 'n_words': 393}