<commit_before>*****
Ruler
*****

.. image:: https://travis-ci.org/yanivmo/ruler.svg?branch=master
    :target: https://travis-ci.org/yanivmo/ruler
    :alt: Build status

.. image:: https://landscape.io/github/yanivmo/ruler/master/landscape.svg?style=flat
   :target: https://landscape.io/github/yanivmo/ruler/master
   :alt: Code Health


Ruler is a lightweight regular expressions wrapper. Its aim is to make regex definitions more
modular, intuitive, readable and the mismatch reporting more informative.


.. warning::
    This project is in active development and is still unstable. It still might contain serious
    defects and documentation inconsistencies.


Quick start
===========

Let's implement the following grammar, given in EBNF_::

    grammar = who, ' likes to drink ', what;
    who = 'John' | 'Peter' | 'Ann';
    what = tea | juice;
    juice = 'juice';
    tea = 'tea', [milk];
    milk = ' with milk';

Using ruler it looks almost identical to EBNF_::

    class Morning(Grammar):
        who = OneOf('John', 'Peter', 'Ann')
        juice = Rule('juice')
        milk = Optional(' with milk')
        tea = Rule('tea', milk)
        what = OneOf(juice, tea)

        _grammar_ = Rule(who, ' likes to drink ', what, '\.')

Now it is possible to match the grammar:

>>> morning = Morning()
>>> match, error = morning.match('Ann likes to drink beer.')

``match`` method returns a tuple of two elements. The second element contains the information about
the match error, if the match failed, and the first element contains information about the successful
match. Naturally, one and only one of the two will be ``None``. In this case the string actually
fails to match:

>>> error is None
False
>>> match is None
True
>>> print(error.long_description)
Mismatch at 19:
  Ann likes to drink beer.
                     ^
"beer." does not match "juice"
"beer." does not match "tea"

Beer isn't one of the options in the grammar and the error message clearly pinpoints the mismatch
location and reason. Now let's try something that matches:

>>> match, error = morning.match('Ann likes to drink tea with milk.')
>>> match is None
False
>>> error is None
True
>>> match
<Match('Ann likes to drink tea with milk.', ['who', 'what']) at 0x3ecaa20>

Rules that were defined as the grammar members act as capture groups:

>>> str(match.who)
'Ann'
>>> str(match.what)
'tea with milk'
>>> bool(match.what.juice)
False
>>> bool(match.what.tea)
True
>>> bool(match.what.tea.milk)
True

Let's try another match:

>>> match, error = morning.match('Peter likes to drink juice. And nothing else matters.')
>>> str(match)
'Peter likes to drink juice.'
>>> str(match.who)
'Peter'
>>> bool(match.what.juice)
True
>>> bool(match.what.tea)
False

Matches implement implicit conversions to string and bool, they can also be compared to strings:

>>> if match:
        if match.who == 'Ann':
            print('Girls like', match.what)
        else:
            print('Boys like', match.what)
Boys like juice

The text inside the rules can be any valid regular expression. So we could rewrite our
grammar like this::

    class Morning(Grammar):
        who = Rule('\w+')
        juice = Rule('juice')
        milk = Optional(' with milk')
        tea = Rule('tea', milk)
        what = OneOf(juice, tea)

        _grammar_ = Rule(who, ' likes to drink ', what, '\.')

>>> morning = Morning()
>>> match, error = morning.match('R2D2 likes to drink juice. And nothing else matters.')
>>> str(match.who)
'R2D2'


Performance
===========
The performance is measured by comparing the matching time with the standard re library.
Currently ruler measures 20 times slower than re.


Development
===========

Tox
---
Tox takes care of everything without installing anything manually. There are two groups of tox
environments: ``py*-test`` and ``py*-profile``. The test environments run the unit tests while the
profile environments run the performance profiling scripts. If tox is not enough then a development
environment can be generated by creating a new virtualenv and then running
``pip install -r requirements_develop.txt``.


Dependency management
---------------------
For the development needs, there are three requirements files in the project's root directory:

- ``requirements_test.txt`` contains all the dependencies needed to run the unit tests,
- ``requirements_profile.txt`` contains all the dependencies needed to run the performance profiling,
- ``requirements_develop.txt`` contains the testing dependencies, the profiling dependencies and some additional
  dependencies used in development.

The requirements files mentioned above are not intended for manual editing. Instead they are managed
using `pip-tools`_. The process of updating the requirements is as follows:

#. Add, remove or update a dependency in one of the ``reqs_*.dep`` files:

  - Update ``reqs_install.dep`` if the dependency is needed for the regular installation by the end user,
  - Update ``reqs_test.dep`` if the dependency is needed to run the unit tests but is not necessary for the
    regular installation,
  - Update ``reqs_profile.dep`` if the dependency is needed to run the performance profiling but is not necessary
    for the regular installation,
  - Update ``reqs_develop.dep`` if the dependency is not in one of the previous categories.

#. Generate the requirements file running ``pip-compile``. The exact command is documented in the beginning of each
   requirements file.
#. Consider running ``pip-sync requirements_develop.txt``.

Notice that there is no need to edit ``setup.py`` - it will pull the dependencies by itself from ``reqs_install.dep``.


TODO
====
::
<commit_msg>Minor markup fix in readme<commit_after>*****
Ruler
*****

.. image:: https://travis-ci.org/yanivmo/ruler.svg?branch=master
    :target: https://travis-ci.org/yanivmo/ruler
    :alt: Build status

.. image:: https://landscape.io/github/yanivmo/ruler/master/landscape.svg?style=flat
   :target: https://landscape.io/github/yanivmo/ruler/master
   :alt: Code Health


Ruler is a lightweight regular expressions wrapper. Its aim is to make regex definitions more
modular, intuitive, readable and the mismatch reporting more informative.


.. warning::
    This project is in active development and is still unstable. It still might contain serious
    defects and documentation inconsistencies.


Quick start
===========

Let's implement the following grammar, given in EBNF_::

    grammar = who, ' likes to drink ', what;
    who = 'John' | 'Peter' | 'Ann';
    what = tea | juice;
    juice = 'juice';
    tea = 'tea', [milk];
    milk = ' with milk';

Using ruler it looks almost identical to EBNF_::

    class Morning(Grammar):
        who = OneOf('John', 'Peter', 'Ann')
        juice = Rule('juice')
        milk = Optional(' with milk')
        tea = Rule('tea', milk)
        what = OneOf(juice, tea)

        _grammar_ = Rule(who, ' likes to drink ', what, '\.')

Now it is possible to match the grammar:

>>> morning = Morning()
>>> match, error = morning.match('Ann likes to drink beer.')

``match`` method returns a tuple of two elements. The second element contains the information about
the match error, if the match failed, and the first element contains information about the successful
match. Naturally, one and only one of the two will be ``None``. In this case the string actually
fails to match:

>>> error is None
False
>>> match is None
True
>>> print(error.long_description)
Mismatch at 19:
  Ann likes to drink beer.
                     ^
"beer." does not match "juice"
"beer." does not match "tea"

Beer isn't one of the options in the grammar and the error message clearly pinpoints the mismatch
location and reason. Now let's try something that matches:

>>> match, error = morning.match('Ann likes to drink tea with milk.')
>>> match is None
False
>>> error is None
True
>>> match
<Match('Ann likes to drink tea with milk.', ['who', 'what']) at 0x3ecaa20>

Rules that were defined as the grammar members act as capture groups:

>>> str(match.who)
'Ann'
>>> str(match.what)
'tea with milk'
>>> bool(match.what.juice)
False
>>> bool(match.what.tea)
True
>>> bool(match.what.tea.milk)
True

Let's try another match:

>>> match, error = morning.match('Peter likes to drink juice. And nothing else matters.')
>>> str(match)
'Peter likes to drink juice.'
>>> str(match.who)
'Peter'
>>> bool(match.what.juice)
True
>>> bool(match.what.tea)
False

Matches implement implicit conversions to string and bool, they can also be compared to strings:

>>> if match:
        if match.who == 'Ann':
            print('Girls like', match.what)
        else:
            print('Boys like', match.what)
Boys like juice

The text inside the rules can be any valid regular expression. So we could rewrite our
grammar like this::

    class Morning(Grammar):
        who = Rule('\w+')
        juice = Rule('juice')
        milk = Optional(' with milk')
        tea = Rule('tea', milk)
        what = OneOf(juice, tea)

        _grammar_ = Rule(who, ' likes to drink ', what, '\.')

>>> morning = Morning()
>>> match, error = morning.match('R2D2 likes to drink juice. And nothing else matters.')
>>> str(match.who)
'R2D2'


Performance
===========
The performance is measured by comparing the matching time with the standard re library.
Currently ruler measures 20 times slower than re.


Development
===========

Tox
---
Tox takes care of everything without installing anything manually. There are two groups of tox
environments: ``py*-test`` and ``py*-profile``. The test environments run the unit tests while the
profile environments run the performance profiling scripts. If tox is not enough then a development
environment can be generated by creating a new virtualenv and then running
``pip install -r requirements_develop.txt``.


Dependency management
---------------------
For the development needs, there are three requirements files in the project's root directory:

- ``requirements_test.txt`` contains all the dependencies needed to run the unit tests,
- ``requirements_profile.txt`` contains all the dependencies needed to run the performance profiling,
- ``requirements_develop.txt`` contains the testing dependencies, the profiling dependencies and some additional
  dependencies used in development.

The requirements files mentioned above are not intended for manual editing. Instead they are managed
using `pip-tools`_. The process of updating the requirements is as follows:

#. Add, remove or update a dependency in one of the ``reqs_*.dep`` files:

   - Update ``reqs_install.dep`` if the dependency is needed for the regular installation by the end user,
   - Update ``reqs_test.dep`` if the dependency is needed to run the unit tests but is not necessary for the
     regular installation,
   - Update ``reqs_profile.dep`` if the dependency is needed to run the performance profiling but is not necessary
     for the regular installation,
   - Update ``reqs_develop.dep`` if the dependency is not in one of the previous categories.

#. Generate the requirements file running ``pip-compile``. The exact command is documented in the beginning of each
   requirements file.
#. Consider running ``pip-sync requirements_develop.txt``.

Notice that there is no need to edit ``setup.py`` - it will pull the dependencies by itself from ``reqs_install.dep``.


TODO
====
::

{'fraction_non_alphanumeric': 0.10992876615952862, 'fraction_numerical': 0.0021106323102629498, 'mean_word_length': 3.6933553446141145, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 143, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3538630', 'n_tokens_mistral': 3198, 'n_tokens_neox': 3070, 'n_words': 1394}