<commit_before>cnx-query-grammar
=================

Install
-------

Use setup.py to install cnx-query-grammar:

::

    $ python setup.py install

This creates a script called ``query_parser``.

Usage
-----

::

    >>> from cnxquerygrammar.query_parser import grammar, DictFormater

    >>> node_tree = grammar.parse('Some text')
    >>> DictFormater().visit(node_tree)
    [('text', 'Some'), ('text', 'text')]

    >>> node_tree = grammar.parse('"A phrase"')
    >>> DictFormater().visit(node_tree)
    [('text', 'A phrase')]

    >>> node_tree = grammar.parse('author:"<NAME>" type:book')
    >>> DictFormater().visit(node_tree)
    [('author', '<NAME>'), ('type', 'book')]

Test
----

To run the tests:

::

    $ python -m unittest discover<commit_msg>Add license info and fix formating in README<commit_after>cnx-query-grammar
=================

The Connections project query grammar parsing library.

Install
-------

Use setup.py to install cnx-query-grammar::

    $ python setup.py install

This creates a script called ``query_parser``.

Usage
-----

::

    >>> from cnxquerygrammar.query_parser import grammar, DictFormater

    >>> node_tree = grammar.parse('Some text')
    >>> DictFormater().visit(node_tree)
    [('text', 'Some'), ('text', 'text')]

    >>> node_tree = grammar.parse('"A phrase"')
    >>> DictFormater().visit(node_tree)
    [('text', 'A phrase')]

    >>> node_tree = grammar.parse('author:"<NAME>" type:book')
    >>> DictFormater().visit(node_tree)
    [('author', '<NAME>'), ('type', 'book')]

Test
----

To run the tests:

::

    $ python -m unittest discover

License
-------

This software is subject to the provisions of the GNU Affero General
Public License Version 3.0 (AGPL). See license.txt for details.
Copyright (c) 2013 Rice University
{'fraction_non_alphanumeric': 0.1954087346024636, 'fraction_numerical': 0.0033594624860022394, 'mean_word_length': 3.6056701030927836, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6105881', 'n_tokens_mistral': 593, 'n_tokens_neox': 563, 'n_words': 148}