<commit_before># Lingo [![Build Status](https://travis-ci.org/rmosolgo/lingo.svg)](https://travis-ci.org/rmosolgo/lingo)

A parser generator for Crystal, inspired by [Parslet](https://github.com/kschiess/parslet).

Lingo provides text processing by:
- parsing the string into a tree of nodes
- providing a visitor to allow you to work from the tree

## Installation

Add this to your application's `shard.yml`:

```yaml
dependencies:
  lingo:
    github: rmosolgo/lingo
```

## Usage

Let's write a parser for highway names. The result will be a method for turning strings into useful objects:

```ruby
def parse_road(input_str)
  ast = RoadParser.parse(input_str)
  visitor = RoadVisitor.new
  visitor.visit(ast)
  visitor.road
end

road = parse_road("I-5N")
# <Road @interstate=true, @number=5, @direction="N">
```

(See more examples in [`/examples`](https://github.com/rmosolgo/lingo/tree/master/examples).)

In the USA, we write highway names like this:

```
50    # Route 50
I-64  # Interstate 64
I-95N # Interstate 95, Northbound
29B   # Business Route 29
```

### Parser

The general structure is `{interstate?}{number}{direction?}{business?}`. Let's express that with Lingo rules:

```ruby
class RoadParser < Lingo::Parser
  # Match a string:
  rule(:interstate) { str("I-") }
  rule(:business) { str("B") }

  # Match a regex:
  rule(:digit) { match(/\d/) }
  # Express repetition with `.repeat`
  rule(:number) { digit.repeat }

  rule(:north) { str("N") }
  rule(:south) { str("S") }
  rule(:east) { str("E") }
  rule(:west) { str("W") }
  # Compose rules by name
  # Express alternation with |
  rule(:direction) { north | south | east | west }

  # Express sequence with >>
  # Express optionality with `.maybe`
  # Name matched strings with `.named`
  rule(:road_name) {
    interstate.named(:interstate).maybe >>
      number.named(:number) >>
      direction.named(:direction).maybe >>
      business.named(:business).maybe
  }
  # You MUST name a starting rule:
  root(:road_name)
end
```

#### Applying the Parser

The resulting class has a `.parse` method which returns a tree of `Lingo::Node`s.

```ruby
RoadParser.parse("250B") # => <Lingo::Node ... >
```

It uses the rule named by `root`.

#### Making Rules
<commit_msg>Update readme with parse instance method<commit_after># Lingo [![Build Status](https://travis-ci.org/rmosolgo/lingo.svg)](https://travis-ci.org/rmosolgo/lingo)

A parser generator for Crystal, inspired by [Parslet](https://github.com/kschiess/parslet).

Lingo provides text processing by:
- parsing the string into a tree of nodes
- providing a visitor to allow you to work from the tree

## Installation

Add this to your application's `shard.yml`:

```yaml
dependencies:
  lingo:
    github: rmosolgo/lingo
```

## Usage

Let's write a parser for highway names. The result will be a method for turning strings into useful objects:

```ruby
def parse_road(input_str)
  ast = RoadParser.new.parse(input_str)
  visitor = RoadVisitor.new
  visitor.visit(ast)
  visitor.road
end

road = parse_road("I-5N")
# <Road @interstate=true, @number=5, @direction="N">
```

(See more examples in [`/examples`](https://github.com/rmosolgo/lingo/tree/master/examples).)

In the USA, we write highway names like this:

```
50    # Route 50
I-64  # Interstate 64
I-95N # Interstate 95, Northbound
29B   # Business Route 29
```

### Parser

The general structure is `{interstate?}{number}{direction?}{business?}`. Let's express that with Lingo rules:

```ruby
class RoadParser < Lingo::Parser
  # Match a string:
  rule(:interstate) { str("I-") }
  rule(:business) { str("B") }

  # Match a regex:
  rule(:digit) { match(/\d/) }
  # Express repetition with `.repeat`
  rule(:number) { digit.repeat }

  rule(:north) { str("N") }
  rule(:south) { str("S") }
  rule(:east) { str("E") }
  rule(:west) { str("W") }
  # Compose rules by name
  # Express alternation with |
  rule(:direction) { north | south | east | west }

  # Express sequence with >>
  # Express optionality with `.maybe`
  # Name matched strings with `.named`
  rule(:road_name) {
    interstate.named(:interstate).maybe >>
      number.named(:number) >>
      direction.named(:direction).maybe >>
      business.named(:business).maybe
  }
  # You MUST name a starting rule:
  root(:road_name)
end
```

#### Applying the Parser

An instance of a `Lingo::Parser` subclass has a `.parse` method which returns a tree of `Lingo::Node`s.

```ruby
RoadParser.new.parse("250B") # => <Lingo::Node ... >
```

It uses the rule named by `root`.

#### Making Rules

{'fraction_non_alphanumeric': 0.15093509350935094, 'fraction_numerical': 0.009240924092409241, 'mean_word_length': 3.9574700109051255, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 25, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6732457', 'n_tokens_mistral': 1584, 'n_tokens_neox': 1484, 'n_words': 446}