<commit_before>GoCYK
=====

GoCYK is an implementation of an online parsing using the CYK algorithm.
The given result recognition table is a merge between these two resources : 
* [Parsing techniques: A practical guide (*ch. 4*)](http://port70.net/~nsz/articles/book/grune_jacobs_parsing_techniques_2008.pdf)
* [Parallel on-line parsing in constant time per word](http://doc.utwente.nl/18047/1/Sikkel93parallel.pdf)

Installation
=====

After having a Go environment installed, just call `go get`:
```bash
$ go get github.com/ghigt/gocyk
```

Usage
=====

First, look at the grammar file `gram_cyk.go` and feel free to modify it. But be aware that it must respect the [CNF (Chomsky Normal Form)](http://en.wikipedia.org/wiki/Chomsky_normal_form).  
Then, launch the program with one parameter corresponding to the string to be parsed, for example `12.3e+1` for the default grammar (-v is the option to see the recognition tree in action):
```bash
$ gocyk -v "12.3e+1"

+--------------------------------------------------------------+
|Digit   |Number  |        |Number  |        |        |Number  |
|Number  |Integer |        |N1      |        |        |        |
|Integer |        |        |        |        |        |        |
+--------------------------------------------------------------|
         |Digit   |        |N1      |        |        |Number  |
         |Integer |        |Number  |        |        |        |
         |Number  |        |        |        |        |        |
         +-----------------------------------------------------|<commit_msg>add godoc reference<commit_after>GoCYK
=====

GoCYK is an implementation of an online parsing using the CYK algorithm.
The given result recognition table is a merge between these two resources : 
* [Parsing techniques: A practical guide (*ch. 4*)](http://port70.net/~nsz/articles/book/grune_jacobs_parsing_techniques_2008.pdf)
* [Parallel on-line parsing in constant time per word](http://doc.utwente.nl/18047/1/Sikkel93parallel.pdf)

[![GoDoc](https://godoc.org/github.com/ghigt/gocyk?status.svg)](https://godoc.org/github.com/ghigt/gocyk)

Installation
=====

After having a Go environment installed, just call `go get`:
```bash
$ go get github.com/ghigt/gocyk
```

Usage
=====

First, look at the grammar file `gram_cyk.go` and feel free to modify it. But be aware that it must respect the [CNF (Chomsky Normal Form)](http://en.wikipedia.org/wiki/Chomsky_normal_form).  
Then, launch the program with one parameter corresponding to the string to be parsed, for example `12.3e+1` for the default grammar (-v is the option to see the recognition tree in action):
```bash
$ gocyk -v "12.3e+1"

+--------------------------------------------------------------+
|Digit   |Number  |        |Number  |        |        |Number  |
|Number  |Integer |        |N1      |        |        |        |
|Integer |        |        |        |        |        |        |
+--------------------------------------------------------------|
         |Digit   |        |N1      |        |        |Number  |
         |Integer |        |Number  |        |        |        |
         |Number  |        |        |        |        |        |
         +-----------------------------------------------------|
{'fraction_non_alphanumeric': 0.21554116558741906, 'fraction_numerical': 0.015417823003391921, 'mean_word_length': 1.550314465408805, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1708110', 'n_tokens_mistral': 946, 'n_tokens_neox': 883, 'n_words': 265}