<commit_before>BLLIP Reranking Parser
----------------------
.. image:: https://travis-ci.org/BLLIP/bllip-parser.png?branch=master
   :target: https://travis-ci.org/BLLIP/bllip-parser

.. image:: https://badge.fury.io/py/bllipparser.png
   :target: https://badge.fury.io/py/bllipparser

Copyright <NAME>, <NAME>, 24th November 2005 --- August 2006

We request acknowledgement in any publications that make use of this
software and any code derived from this software. Please report the
release date of the software that you are using, as this will enable
others to compare their results to yours.

References:

* <NAME>, and <NAME>. "Coarse-to-fine n-best parsing and
  MaxEnt discriminative reranking." Proceedings of the 43rd Annual Meeting
  on Association for Computational Linguistics. Association for
  Computational Linguistics, 2005.
  http://aclweb.org/anthology/P/P05/P05-1022.pdf

* Charniak, Eugene. "A maximum-entropy-inspired parser." Proceedings of
  the 1st North American chapter of the Association for Computational
  Linguistics conference. Association for Computational Linguistics, 2000.
  http://aclweb.org/anthology//A/A00/A00-2018.pdf

Overview
~~~~~~~~
BLLIP Parser is a statistical natural language parser including a
generative constituent parser (``first-stage``) and discriminative
maximum entropy reranker (``second-stage``). The latest version can be
found on `GitHub <https://github.com/BLLIP/bllip-parser>`_. This
document describes basic usage of the command-line interface and
describes how to build and run the reranking parser. There are now
`Python <http://pypi.python.org/pypi/bllipparser/>`_ and Java interfaces
as well. The Python interface described in ``README-python.rst``.

Compiling the parser
~~~~~~~~~~~~~~~~~~~~
1. *(optional)* For optimal speed, you may want to define ``$GCCFLAGS``
   specifically for your machine. This step can be safely skipped as the
   defaults should be okay. With ``csh`` or ``tcsh``, try something
   like::

     shell> setenv GCCFLAGS "-march=pentium4 -mfpmath=sse -msse2 -mmmx"``

   or::

     shell> setenv GCCFLAGS "-march=opteron -m64"``

2. Build the parser with::

    shell> make

   -  Sidenote on compiling on OS X

      OS X uses the ``clang`` compiler by default which cannot currently
      compile the parser. Try setting this environment variable before
      building to change the default C++ compiler::

         shell> setenv CXX g++

      See http://github.com/BLLIP/bllip-parser/issues/13 for more
      information. Recent versions of OS X may have additional issues.

Running the parser
~~~~~~~~~~~~~~~~~~
After it has built, the parser can be run with::

    shell> parse.sh <sourcefile.txt>

For example::

    shell> parse.sh sample-text/sample-data.txt

The input text must be pre-sentence segmented with each sentence in an
``<s>`` tag::
<commit_msg>README.rst: correct MD->RST conversion error<commit_after>BLLIP Reranking Parser
----------------------
.. image:: https://travis-ci.org/BLLIP/bllip-parser.png?branch=master
   :target: https://travis-ci.org/BLLIP/bllip-parser

.. image:: https://badge.fury.io/py/bllipparser.png
   :target: https://badge.fury.io/py/bllipparser

Copyright <NAME>, <NAME>, 24th November 2005 --- August 2006

We request acknowledgement in any publications that make use of this
software and any code derived from this software. Please report the
release date of the software that you are using, as this will enable
others to compare their results to yours.

References:

* Charniak, Eugene, and <NAME>. "Coarse-to-fine n-best parsing and
  MaxEnt discriminative reranking." Proceedings of the 43rd Annual Meeting
  on Association for Computational Linguistics. Association for
  Computational Linguistics, 2005.
  http://aclweb.org/anthology/P/P05/P05-1022.pdf

* Charniak, Eugene. "A maximum-entropy-inspired parser." Proceedings of
  the 1st North American chapter of the Association for Computational
  Linguistics conference. Association for Computational Linguistics, 2000.
  http://aclweb.org/anthology//A/A00/A00-2018.pdf

Overview
~~~~~~~~
BLLIP Parser is a statistical natural language parser including a
generative constituent parser (``first-stage``) and discriminative
maximum entropy reranker (``second-stage``). The latest version can be
found on `GitHub <https://github.com/BLLIP/bllip-parser>`_. This
document describes basic usage of the command-line interface and
describes how to build and run the reranking parser. There are now
`Python <http://pypi.python.org/pypi/bllipparser/>`_ and Java interfaces
as well. The Python interface described in ``README-python.rst``.

Compiling the parser
~~~~~~~~~~~~~~~~~~~~
1. *(optional)* For optimal speed, you may want to define ``$GCCFLAGS``
   specifically for your machine. This step can be safely skipped as the
   defaults should be okay. With ``csh`` or ``tcsh``, try something
   like::

     shell> setenv GCCFLAGS "-march=pentium4 -mfpmath=sse -msse2 -mmmx"

   or::

     shell> setenv GCCFLAGS "-march=opteron -m64"

2. Build the parser with::

    shell> make

   -  Sidenote on compiling on OS X

      OS X uses the ``clang`` compiler by default which cannot currently
      compile the parser. Try setting this environment variable before
      building to change the default C++ compiler::

         shell> setenv CXX g++

      See http://github.com/BLLIP/bllip-parser/issues/13 for more
      information. Recent versions of OS X may have additional issues.

Running the parser
~~~~~~~~~~~~~~~~~~
After it has built, the parser can be run with::

    shell> parse.sh <sourcefile.txt>

For example::

    shell> parse.sh sample-text/sample-data.txt

The input text must be pre-sentence segmented with each sentence in an
``<s>`` tag::

{'fraction_non_alphanumeric': 0.11281070745697896, 'fraction_numerical': 0.01564401181992004, 'mean_word_length': 3.8030050083472453, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 31, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3538534', 'n_tokens_mistral': 1793, 'n_tokens_neox': 1633, 'n_words': 651}