<commit_before>This file requires editing
==========================

Note to the author: Please add something informative to this README *before*
releasing your software, as `a little documentation goes a long way`_.  Both
README.rst (this file) and NEWS.txt (release notes) will be included in your
package metadata which gets displayed in the PyPI page for your project.

You can take a look at the README.txt of other projects, such as repoze.bfg
(http://bfg.repoze.org/trac/browser/trunk/README.txt) for some ideas.

.. _`a little documentation goes a long way`: http://www.martinaspeli.net/articles/a-little-documentation-goes-a-long-way

Credits
-------

- `Distribute`_
- `Buildout`_
- `modern-package-template`_

.. _Buildout: http://www.buildout.org/
.. _Distribute: http://pypi.python.org/pypi/distribute
.. _`modern-package-template`: http://pypi.python.org/pypi/modern-package-template<commit_msg>new documentation<commit_after>Engerek
========
Turkish natural language processing library.
--------------------------------------------

Engerek is a Python library that brings together commonly used natural language processing techniques, with a special focus on Turkish text.

Our aim is to provide a clean and consistent interface for existing open source Turkish processing libraries.

Currently you can

- tokenize text (Engerek's own tokenizer that recognizes hashtags, user mentions, URLs)
- stem words (`via Zemberek <https://code.google.com/p/zemberek/>`_)
- deasciify Turkish text that is written only by ASCII characters (`via Ahmet Balkan's deasciifier <https://code.google.com/p/turkish-deasciifier/>`_)

Usage
-----

```python
>>> from engerek import *
>>> deasciifier = BalkansDeasciifier()
>>> print deasciifier.deasciify(u'Hadi bir masal uyduralim, icinde mutlu, doygun, telassiz durdugumuz.')
Hadi bir masal uyduralım, içinde mutlu, doygun, telaşsız durduğumuz.
>>> tokenizer.tokenize(u"Hadi bir masal uyduralım, içinde mutlu, doygun, telaşsız durduğumuz.")
[u'Hadi', u'bir', u'masal', u'uydural\u0131m', u'i\xe7inde', u'mutlu', u'doygun', u'tela\u015fs\u0131z', u'durdu\u011fumuz']
>>> stemmer = ZemberekStemmer(prefer_infinitive_form=True)
>>> print stemmer.stem(u'atandı')
[u'atmak', u'atamak', u'ata']
```

Installation and Setup
----------------------
Install Python dependencies and Engerek.
```sh
$ pip install cython
$ pip install git+git://github.com/cilekagaci/engerek.git#egg=engerek
```

In the last above step, if you get a failure about missing Cython.Distutils just re-run the same command.

Download Balkan's deasciifier and Zemberek libraries as jar files and put them under a directory in your Java classpath.
```sh
$ mkdir -p $HOME/lib && cd $HOME/lib
$ wget https://zemberek.googlecode.com/files/zemberek-2.1.1.zip
$ unzip zemberek-2.1.1.zip && rm zemberek-2.1.1.zip
$ mv jar/*.jar .
$ rm -r belgeler
$ rm -r jar
$ rm surumler.txt
$ wget https://turkish-deasciifier.googlecode.com/files/turkish-deasciifier-1.0.jar
$ export CLASSPATH=$CLASSPATH:$HOME/lib/*
```

Check if the installation works:
```python
>>> from engerek import *
```
{'fraction_non_alphanumeric': 0.14948119325551232, 'fraction_numerical': 0.008430609597924773, 'mean_word_length': 4.237691001697793, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 26, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5744655', 'n_tokens_mistral': 1067, 'n_tokens_neox': 1024, 'n_words': 304}