<commit_before>Python Linguistic Annotation Libary
===================================
Poio API is a Python Library to access and manipulate linguistically
annotated corpus files. Supported file format is currently only Elan XML,
with Kura XML and Toolbox files support planned for future releases. A
Corpus Reader API is provided to support statistical analysis within the
Natural Language Toolkit.
The software is licensed under the GNU General Public License. 


REQUIREMENTS
============
You need to install the following packages:

- Python: http://python.org/download
- If you want to process data with NLTK: http://www.nltk.org/download

INSTALLATION
============
To install Poio API on Windows just start the .exe file you downloaded and
follow the instructions in the setup process.
To install Poio API on Linux, Unix and other platforms you need to unpack
the file and start "setup.py" on the command line. Change to the directory
into which you downloaded the package and unpack it::

  $ tar xzf cidles-poio-api-x.y.z.tar.gz
  $ cd cidles-poio-api

Then, to install the package locally into your python repository (you may need
to have root privileges)::

  $ python setup.py install

The installation process will give you feedback and should finish without
errors.


BASIC USAGE
===========
Here are a few examples what you can do with Poio API. All the examples
process Elan files which are stored in one directory, the directory here is
"example_data" which is part of the package you downloaded. The package also
contains a sample script "example1.py" that runs all the commands presented
here, so you might just call "python example1.py" and see all the results on
your own computer at once. First, start a python interpreter and import
pyanntation for Elan::

  $ python
  Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) 
  [GCC 4.3.3] on linux2
  Type "help", "copyright", "credits" or "license" for more information.

First, import the corpus reader module:

>>> import poioapi.corpusreader

Then load create a corpus reader and load a file into your corpus. The
second argument to the addFile method is the file type (.eaf here):

>>> cr = poioapi.corpusreader.GlossCorpusReader()
>>> cr.addFile("example_data/turkish.eaf", poioapi.data.EAF)

To get all sentences with their tags that have a gloss "ANOM" (here: tags
are morphemes and their glosses stored in a kind of tree):

>>> result = [s for s in cr.tagged_sents() for (word, tag) in s
...             for (morphem, gloss) in tag
...             if 'ANOM' in gloss and s not in locals()['_[1]']]
>>> print result
[[('eve', [('ev', ['home']), ('e', ['DIR'])]), ('geldi\xc4\x9fimde', ...

Only the sentences of the result:

>>>sents = [[w for (w, t) in s] for s in result]
>>> print sents<commit_msg>Readme updated.<commit_after>Python Linguistic Annotation Libary
===================================
Poio API is a Python Library to access and manipulate linguistically
annotated corpus files. Supported file format is currently only Elan XML,
with Kura XML and Toolbox files support planned for future releases. A
Corpus Reader API is provided to support statistical analysis within the
Natural Language Toolkit.
The software is licensed under the GNU General Public License. 


REQUIREMENTS
============
You need to install the following packages:

- Python: http://python.org/download
- If you want to process data with NLTK: http://www.nltk.org/download

INSTALLATION
============
To install Poio API on Windows just start the .exe file you downloaded and
follow the instructions in the setup process.
To install Poio API on Linux, Unix and other platforms you need to unpack
the file and start "setup.py" on the command line. Change to the directory
into which you downloaded the package and unpack it::

  $ tar xzf cidles-poio-api-x.y.z.tar.gz
  $ cd cidles-poio-api

Then, to install the package locally into your python repository (you may need
to have root privileges)::

  $ python setup.py install

The installation process will give you feedback and should finish without
errors.


BASIC USAGE
===========
Here are a few examples what you can do with Poio API. All the examples
process Elan files which are stored in one directory, the directory here is
"example_data" which is part of the package you downloaded. The package also
contains a sample script "poioapiexample.py" that runs all the commands presented
here, so you might just call "python poioapisearch.py" and see all the results on
your own computer at once. First, start a python interpreter and import
pyanntation for Elan::

  $ python
  Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) 
  [GCC 4.3.3] on linux2
  Type "help", "copyright", "credits" or "license" for more information.

First, import the corpus reader module:

>>> import poioapi.corpusreader

Then load create a corpus reader and load a file into your corpus. The
second argument to the addFile method is the file type (.eaf here):

>>> cr = poioapi.corpusreader.GlossCorpusReader()
>>> cr.addFile("example_data/turkish.eaf", poioapi.data.EAF)

To get all sentences with their tags that have a gloss "ANOM" (here: tags
are morphemes and their glosses stored in a kind of tree):

>>> result = [s for s in cr.tagged_sents() for (word, tag) in s
...             for (morphem, gloss) in tag
...             if 'ANOM' in gloss and s not in locals()['_[1]']]
>>> print result
[[('eve', [('ev', ['home']), ('e', ['DIR'])]), ('geldi\xc4\x9fimde', ...

Only the sentences of the result:

>>>sents = [[w for (w, t) in s] for s in result]
>>> print sents
{'fraction_non_alphanumeric': 0.10349775784753364, 'fraction_numerical': 0.008968609865470852, 'mean_word_length': 4.177344475394615, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5055751', 'n_tokens_mistral': 1657, 'n_tokens_neox': 1550, 'n_words': 788}