<commit_before>.. http://www.sphinx-doc.org/en/stable/rest.html
.. http://rst.ninjs.org/

********************
User guide
********************

:Last Reviewed: 2017-04-17

.. contents:: Contents
    :local:


Running pyrata (in console)
============================

First run python in console:

::

  python3

Then import the main pyrata regular expression module:

.. doctest ::

  >>> import pyrata.re as pyrata_re



Language expressivity
=====================

Basic concepts
--------------

pyrata data structure
  Pyrata is intented to process *data* made of *sequence of elements*, each element being a *features set* i.e. a set of name-value attributes. In other words the pyrata data structure is litteraly a ``list`` of ``dict``. The expected type of values is the primitive type ``String``.


.. doctest ::

  >>> data = [{'pos': 'PRP', 'raw': 'It'}, {'pos': 'VBZ', 'raw': 'is'}, {'pos': 'JJ', 'raw': 'fast'}, {'pos': 'JJ', 'raw': 'easy'}, {'pos': 'CC', 'raw': 'and'}, {'pos': 'JJ', 'raw': 'funny'}, {'pos': 'TO', 'raw': 'to'}, {'pos': 'VB', 'raw': 'write'}, {'pos': 'JJ', 'raw': 'regular'}, {'pos': 'NNS', 'raw': 'expressions'}, {'pos': 'IN', 'raw': 'with'},{'pos': 'NNP', 'raw': 'Pyrata'}]

There is *no requirement on the names of the features*.
In the previous code, you see that the names ``raw`` and ``pos`` have been arbitrary choosen to means respectively the surface form of a word and its part-of-speech.

pyrata pattern
  Pyrata allows to define *regular expressions* over the pyrata data structure.

pattern step
  The elementary component of a pyrata pattern is the **step**. It corresponds to the set of constraints a data element should match.

Let's say you want to search all the adjectives in the sentence. By chance there is a property which specifies the part of speech of tokens, *pos*, the value of *pos* which stands for adjectives is *JJ*. Your pattern will be:

.. doctest ::

  >>> pattern = 'pos="JJ"'


Step constraint operators (*equal, match, in, chunk*)
------------------

Equal operator
^^^^^^^^^^^^^^^

Classically, the value of the refered feature name should be equal to the specified value.


Regular expression match operator
^^^^^^^^^^^^^^^

At the atomic level, there is not only the equal operator to set a constraint. You can also **set a regular expression as a value**. 
In that case, the operator will not be ``=`` but ``~`` 

.. doctest ::

    >>> pyrata_re.findall('pos~"NN."', data)
    [[{'raw': 'expressions', 'pos': 'NNS'}], [{'raw': 'Pyrata', 'pos': 'NNP'}]]

Consequently ``[pos="NNS" | pos="NNP"]``, ``pos~"NN[SP]"`` and 'pos~"(NNS|NNP)"' are equivalent forms. They may not have the same processing time.


In 'list' operator
^^^^^^^^^^^^^^^

You can also **set a list of possible values (lexicon)**. In that case, the operator will be ``@`` in your pattern and the value will be the name of the lexicon. The lexicon is specified as a parameter of the pyrata_re methods (``lexicons`` parameter). Indeed, multiple lexicons can be specified. The data structure for storing lexicons is a dict/map of lists. Each key of the dict is the name of a lexicon, and each corresponding value a list of elements making of the lexicon.

.. doctest ::

    >>> pyrata_re.findall('lem@"positiveLexicon"', data, lexicons = {'positiveLexicon':['easy', 'funny']})
    [[ {'pos': 'JJ', 'raw': 'easy'}], [{'pos': 'JJ', 'raw': 'funny'}]]

IOB Chunk operator
^^^^^^^^^^^^^^^

.. epigraph::

   The most widespread representation of chunks uses IOB tags. In this scheme, each token is tagged with one of three special chunk tags, I (inside), O (outside), or B (begin). A token is tagged as B if it marks the beginning of a chunk. Subsequent tokens within the chunk are tagged I. All other tokens are tagged O. The B and I tags are suffixed with the chunk type, e.g. B-NP, I-NP. Of course, it is not necessary to specify a chunk type for tokens that appear outside a chunk, so these are just labeled O.

   -- [nltk book](http://www.nltk.org/book/ch07.html)

An example of pyrata data structure with chunks annotated in IOB tagged format is shown below. See the values of the ``chunk`` feature.  

.. doctest ::

    >>> data = [{'pos': 'NNP', 'chunk': 'B-PERSON', 'raw': 'Mark'}, {'pos': 'NNP', 'chunk': 'I-PERSON', 'raw': 'Zuckerberg'}, {'pos': 'VBZ', 'chunk': 'O', 'raw': 'is'}, {'pos': 'VBG', 'chunk': 'O', 'raw': 'working'}, {'pos': 'IN', 'chunk': 'O', 'raw': 'at'}, {'pos': 'NNP', 'chunk': 'B-ORGANIZATION', 'raw': 'Facebook'}, {'pos': 'NNP', 'chunk': 'I-ORGANIZATION', 'raw': 'Corp'}, {'pos': '.', 'chunk': 'O', 'raw': '.'}] 


.. warning:: 

  The following subsubsection is incomplete.

.. todo:: 

  finish the subsubsection

.. seealso:: 

  nothing to see here
  

TODO

    chunk-"PERSON" [pos~"VB"]* FIXME
    pos="IN" chunk."ORGANIZATION" FIXME

    Before introducing the chunk operator: introduce the annotate methods

What can do the annotate method:
- each feature set of the matched sequences are updated with a given feature set
- each feature set of the matched sequences are updated with a given feature set ; some of them should follow the iob scheme.
- by default group 0 is updated or the given groups of the matched squences

    annotation = {'chunk':'PERSON'}
    new_data = annotate (pattern, data, annotation, iob=['chunk'], groups = ['1'])


Sequence of steps
------------------

You can search a **sequence of steps**, for example an adjective (tagged *JJ*) followed by a noun in plural form  (tagged *NNS*):

.. doctest ::

    >>> pattern = 'pos="JJ" pos="NNS"'
    >>> pyrata_re.search(pattern, data).group()
    [{'pos': 'JJ', 'raw': 'regular'}, {'pos': 'NNS', 'raw': 'expressions'}]

Class of step
------------------

You can specify a **class of steps** by combining single constraints on the properties of the required step with logical operators like:

.. doctest ::

    >>> pyrata_re.findall('[(pos="NNS" | pos="NNP") & !raw="pattern"]', data)<commit_msg>update doc/user-guide.rst<commit_after>.. http://www.sphinx-doc.org/en/stable/rest.html
.. http://rst.ninjs.org/

********************
User guide
********************

:Last Reviewed: 2017-04-17

.. contents:: Contents
    :local:


Running pyrata (in console)
============================

First run python in console:

::

  python3

Then import the main pyrata regular expression module:

.. doctest ::

  >>> import pyrata.re as pyrata_re



Language expressivity
=====================

Basic concepts
--------------

pyrata data structure
  Pyrata is intented to process *data* made of *sequence of elements*, each element being a *features set* i.e. a set of name-value attributes. In other words the pyrata data structure is litteraly a ``list`` of ``dict``. The expected type of values is the primitive type ``String``.


.. doctest ::

  >>> data = [{'pos': 'PRP', 'raw': 'It'}, {'pos': 'VBZ', 'raw': 'is'}, {'pos': 'JJ', 'raw': 'fast'}, {'pos': 'JJ', 'raw': 'easy'}, {'pos': 'CC', 'raw': 'and'}, {'pos': 'JJ', 'raw': 'funny'}, {'pos': 'TO', 'raw': 'to'}, {'pos': 'VB', 'raw': 'write'}, {'pos': 'JJ', 'raw': 'regular'}, {'pos': 'NNS', 'raw': 'expressions'}, {'pos': 'IN', 'raw': 'with'},{'pos': 'NNP', 'raw': 'Pyrata'}]

There is *no requirement on the names of the features*.
In the previous code, you see that the names ``raw`` and ``pos`` have been arbitrary choosen to means respectively the surface form of a word and its part-of-speech.

pyrata pattern
  Pyrata allows to define *regular expressions* over the pyrata data structure.

pattern step
  The elementary component of a pyrata pattern is the **step**. It corresponds to the set of constraints a data element should match.

Let's say you want to search all the adjectives in the sentence. By chance there is a property which specifies the part of speech of tokens, *pos*, the value of *pos* which stands for adjectives is *JJ*. Your pattern will be:

.. doctest ::

  >>> pattern = 'pos="JJ"'


Step constraint operators (*equal, match, in, chunk*)
------------------

Equal operator
^^^^^^^^^^^^^^^

Classically, the value of the refered feature name should be equal to the specified value.


Regular expression match operator
^^^^^^^^^^^^^^^

At the atomic level, there is not only the equal operator to set a constraint. You can also **set a regular expression as a value**. 
In that case, the operator will not be ``=`` but ``~`` 

.. doctest ::

    >>> pyrata_re.findall('pos~"NN."', data)
    [[{'raw': 'expressions', 'pos': 'NNS'}], [{'raw': 'Pyrata', 'pos': 'NNP'}]]

Consequently ``[pos="NNS" | pos="NNP"]``, ``pos~"NN[SP]"`` and 'pos~"(NNS|NNP)"' are equivalent forms. They may not have the same processing time.


In 'list' operator
^^^^^^^^^^^^^^^

You can also **set a list of possible values (lexicon)**. In that case, the operator will be ``@`` in your pattern and the value will be the name of the lexicon. The lexicon is specified as a parameter of the pyrata_re methods (``lexicons`` parameter). Indeed, multiple lexicons can be specified. The data structure for storing lexicons is a dict/map of lists. Each key of the dict is the name of a lexicon, and each corresponding value a list of elements making of the lexicon.

.. doctest ::

    >>> pyrata_re.findall('lem@"positiveLexicon"', data, lexicons = {'positiveLexicon':['easy', 'funny']})
    [[ {'pos': 'JJ', 'raw': 'easy'}], [{'pos': 'JJ', 'raw': 'funny'}]]

IOB Chunk operator
^^^^^^^^^^^^^^^

.. epigraph::

   The most widespread representation of chunks uses IOB tags. In this scheme, each token is tagged with one of three special chunk tags, I (inside), O (outside), or B (begin). A token is tagged as B if it marks the beginning of a chunk. Subsequent tokens within the chunk are tagged I. All other tokens are tagged O. The B and I tags are suffixed with the chunk type, e.g. B-NP, I-NP. Of course, it is not necessary to specify a chunk type for tokens that appear outside a chunk, so these are just labeled O.

   -- [nltk book](http://www.nltk.org/book/ch07.html)

An example of pyrata data structure with chunks annotated in IOB tagged format is shown below. See the values of the ``chunk`` feature.  

.. doctest ::

    >>> data = [{'pos': 'NNP', 'chunk': 'B-PERSON', 'raw': 'Mark'}, {'pos': 'NNP', 'chunk': 'I-PERSON', 'raw': 'Zuckerberg'}, {'pos': 'VBZ', 'chunk': 'O', 'raw': 'is'}, {'pos': 'VBG', 'chunk': 'O', 'raw': 'working'}, {'pos': 'IN', 'chunk': 'O', 'raw': 'at'}, {'pos': 'NNP', 'chunk': 'B-ORGANIZATION', 'raw': 'Facebook'}, {'pos': 'NNP', 'chunk': 'I-ORGANIZATION', 'raw': 'Corp'}, {'pos': '.', 'chunk': 'O', 'raw': '.'}] 


.. warning:: 

  The following subsubsection is incomplete.

..  
    chunk-"PERSON" [pos~"VB"]* FIXME
    pos="IN" chunk."ORGANIZATION" FIXME

    Before introducing the chunk operator: introduce the annotate methods



Sequence of steps
------------------

You can search a **sequence of steps**, for example an adjective (tagged *JJ*) followed by a noun in plural form  (tagged *NNS*):

.. doctest ::

    >>> pattern = 'pos="JJ" pos="NNS"'
    >>> pyrata_re.search(pattern, data).group()
    [{'pos': 'JJ', 'raw': 'regular'}, {'pos': 'NNS', 'raw': 'expressions'}]

Class of step
------------------

You can specify a **class of steps** by combining single constraints on the properties of the required step with logical operators like:

.. doctest ::

    >>> pyrata_re.findall('[(pos="NNS" | pos="NNP") & !raw="pattern"]', data)
{'fraction_non_alphanumeric': 0.19110407876230662, 'fraction_numerical': 0.002109704641350211, 'mean_word_length': 3.968122270742358, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '464333', 'n_tokens_mistral': 3624, 'n_tokens_neox': 3384, 'n_words': 1492}