<commit_before>Classical Latin
************************

POS Tagging
===========

.. warning::

   POS tagging is a work in progress. A new tagging dictionary has been created, though a tagger has not yet been written.

First, `obtain the Latin POS tagging files <http://cltk.readthedocs.org/en/latest/import_corpora.html#pos-tagging>`_. The important file here is ``cltk_latin_pos_dict.txt``, which is saved at ``~/cltk_data/compiled/pos_latin``. This file is a Python ``dict`` type which aims to give all possible parts-of-speech for any given form, though this is based off the incomplete Perseus ``latin-analyses.txt``. Thus, there may be gaps either in (i) the inflected forms defined and (ii) the comprehensiveness of the analyses of any given form. ``cltk_latin_pos_dict.txt`` looks like:

.. code-block:: python

   {'-nam': {'perseus_pos': [{'pos0': {'case': 'indeclform',
                                       'gloss': '',
                                       'type': 'conj'}}]},
    '-namque': {'perseus_pos': [{'pos0': {'case': 'indeclform',
                                          'gloss': '',
                                          'type': 'conj'}}]},
    '-sed': {'perseus_pos': [{'pos0': {'case': 'indeclform',
                                       'gloss': '',
                                       'type': 'conj'}}]},
    'Aaron': {'perseus_pos': [{'pos0': {'case': 'nom',
                                        'gender': 'masc',
                                        'gloss': 'Aaron',
                                        'number': 'sg',
                                        'type': 'substantive'}}]},
   }

If you wish to edit the POS dictionary creator, ``cltk_latin_pos_dict.txt`` may be recreated with:

.. code-block:: python

   In [1]: from cltk.tag.classical_latin.pos_latin import MakePOSTagger

   In [2]: m = MakePOSTagger()

   In [3]: m.make_file()

Sentence Tokenization
=====================

.. warning::

   This sentence tokenizer appears to work well, though it was trained on a small training set of 8439 words and 529 sentences  (Cicero, *Pro Caelio*).

.. note::

   A batch processor should be added to this, so as to process more texts. Once the training set has been expanded and a good tokenizer has been created, this code should be moved to a `cltk_data` repository.

The file `latin.txt` was generated with the following commands, executed in IPython, from within `~/cltk/cltk/tokenizers/`.

.. code-block:: python

   In [1]: from sentence_tokenizer import train_from_file

   In [2]: train_from_file('training_set/sents_cael.txt')
     Abbreviation: [50.5402] m
     Abbreviation: [0.3420] sex
     Abbreviation: [17.6891] l
     Abbreviation: [5.0540] c
     Abbreviation: [0.9296] cn
     Abbreviation: [15.1621] q
     Abbreviation: [10.1080] p
     Rare Abbrev: putaverunt.
     Sent Starter: [42.0530] 'sed'
     Sent Starter: [48.3078] 'nam'
   <nltk.tokenize.punkt.PunktTrainer object at 0x103b06278>

To tokenize a text, pass a text to `tokenize_sentences()` as follows:

.. code-block:: python

   In [1]: from sentence_tokenizer import tokenize_sentences
   
   In [2]: tokenize_sentences('phil1.txt') 
   ['Cicero: Philippic I\n\t\t \n\n\t\t \n\t\t \n\t \n\t\n \n\n M. TVLLI CICERONIS IN M. ANTONIVM ORATIO PHILIPPICA PRIMA\n \n\n \n\n \n 1 \t 2 \t 3 \t 4 \t 5 \t 6 \t 7 \t 8 \t 9 \t 10 \t 11 \t 12 \t 13 \t 14 \t 15 \t 16 \t 17 \t 18 \t 19 \t 20 \t 21 \t 22 \t 23 \t 24 \t 25 \t 26 \t 27 \t 28 \t 29 \t 30 \t 31 \t 32 \t 33 \t 34 \t 35 \t 36 \t 37 \t 38 \n \n\n \n\n \n[ 1 ] Antequam de republica, patres conscripti, dicam ea, quae dicenda hoc tempore arbitror, exponam vobis breviter consilium et profectionis et reversionis meae.', 'Ego cum sperarem aliquando ad vestrum consilium auctoritatemque rem publicam esse revocatam, manendum mihi statuebam, quasi in vigilia quadam consulari ac senatoria.', 'Nec vero usquam discedebam nec a re publica deiciebam oculos ex eo die, quo in aedem Telluris convocati sumus.', 'In quo templo, quantum in me fuit, ieci fundamenta pacis Atheniensiumque renovavi vetus exemplum; Graecum etiam verbum usurpavi, quo tum in sedandis discordiis usa erat civitas illa, atque omnem memoriam discordiarum oblivione sempiterna delendam censui.', ... ]


Text Processing
===============

Filter Stopwords
----------------

.. code-block:: python

   In [1]: import nltk.tokenize

   In [2]: from cltk.stop.classical_latin.stops import STOPS_LIST

   In [3]: SENTENCE = 'Quo usque tandem abutere, Catilina, patientia nostra?'

   In [4]: lowered = SENTENCE.lower()

   In [5]: tokens = nltk.word_tokenize(lowered)

   In [6]: [w for w in tokens if not w in STOPS_LIST]
   Out[6]: 
   ['usque',
    'tandem',
    'abutere',
    ',',
    'catilina',
    ',',
    'patientia',
    'nostra',
    '?']

   
Convert J to I, V to U
----------------------

.. code-block:: python

   In [1]: from cltk.stem.classical_latin.j_and_v_converter import JVReplacer

   In [2]: j = JVReplacer()

   In [3]: j.replace('vem jam')
   Out[3]: 'uem iam'
<commit_msg>philippics example imporoved<commit_after>Classical Latin
************************

POS Tagging
===========

.. warning::

   POS tagging is a work in progress. A new tagging dictionary has been created, though a tagger has not yet been written.

First, `obtain the Latin POS tagging files <http://cltk.readthedocs.org/en/latest/import_corpora.html#pos-tagging>`_. The important file here is ``cltk_latin_pos_dict.txt``, which is saved at ``~/cltk_data/compiled/pos_latin``. This file is a Python ``dict`` type which aims to give all possible parts-of-speech for any given form, though this is based off the incomplete Perseus ``latin-analyses.txt``. Thus, there may be gaps either in (i) the inflected forms defined and (ii) the comprehensiveness of the analyses of any given form. ``cltk_latin_pos_dict.txt`` looks like:

.. code-block:: python

   {'-nam': {'perseus_pos': [{'pos0': {'case': 'indeclform',
                                       'gloss': '',
                                       'type': 'conj'}}]},
    '-namque': {'perseus_pos': [{'pos0': {'case': 'indeclform',
                                          'gloss': '',
                                          'type': 'conj'}}]},
    '-sed': {'perseus_pos': [{'pos0': {'case': 'indeclform',
                                       'gloss': '',
                                       'type': 'conj'}}]},
    'Aaron': {'perseus_pos': [{'pos0': {'case': 'nom',
                                        'gender': 'masc',
                                        'gloss': 'Aaron',
                                        'number': 'sg',
                                        'type': 'substantive'}}]},
   }

If you wish to edit the POS dictionary creator, ``cltk_latin_pos_dict.txt`` may be recreated with:

.. code-block:: python

   In [1]: from cltk.tag.classical_latin.pos_latin import MakePOSTagger

   In [2]: m = MakePOSTagger()

   In [3]: m.make_file()

Sentence Tokenization
=====================

.. warning::

   This sentence tokenizer appears to work well, though it was trained on a small training set of 8439 words and 529 sentences  (Cicero, *Pro Caelio*).

.. note::

   A batch processor should be added to this, so as to process more texts. Once the training set has been expanded and a good tokenizer has been created, this code should be moved to a `cltk_data` repository.

The file `latin.txt` was generated with the following commands, executed in IPython, from within `~/cltk/cltk/tokenizers/`.

.. code-block:: python

   In [1]: from sentence_tokenizer import train_from_file

   In [2]: train_from_file('training_set/sents_cael.txt')
     Abbreviation: [50.5402] m
     Abbreviation: [0.3420] sex
     Abbreviation: [17.6891] l
     Abbreviation: [5.0540] c
     Abbreviation: [0.9296] cn
     Abbreviation: [15.1621] q
     Abbreviation: [10.1080] p
     Rare Abbrev: putaverunt.
     Sent Starter: [42.0530] 'sed'
     Sent Starter: [48.3078] 'nam'
   <nltk.tokenize.punkt.PunktTrainer object at 0x103b06278>

   To tokenize a text, such as Cicero's *Philippic* 1, pass it to `tokenize_sentences()` as follows:

.. code-block:: python

   In [1]: from sentence_tokenizer import tokenize_sentences
   
   In [2]: tokenize_sentences('phil1.txt')

   ['Antequam de republica, patres conscripti, dicam ea, quae dicenda hoc tempore arbitror, exponam vobis breviter consilium et profectionis et reversionis meae.', 'Ego cum sperarem aliquando ad vestrum consilium auctoritatemque rem publicam esse revocatam, manendum mihi statuebam, quasi in vigilia quadam consulari ac senatoria.', 'Nec vero usquam discedebam nec a re publica deiciebam oculos ex eo die, quo in aedem Telluris convocati sumus.', 'In quo templo, quantum in me fuit, ieci fundamenta pacis Atheniensiumque renovavi vetus exemplum; Graecum etiam verbum usurpavi, quo tum in sedandis discordiis usa erat civitas illa, atque omnem memoriam discordiarum oblivione sempiterna delendam censui.', ... ]


Text Processing
===============

Filter Stopwords
----------------

.. code-block:: python

   In [1]: import nltk.tokenize

   In [2]: from cltk.stop.classical_latin.stops import STOPS_LIST

   In [3]: SENTENCE = 'Quo usque tandem abutere, Catilina, patientia nostra?'

   In [4]: lowered = SENTENCE.lower()

   In [5]: tokens = nltk.word_tokenize(lowered)

   In [6]: [w for w in tokens if not w in STOPS_LIST]
   Out[6]: 
   ['usque',
    'tandem',
    'abutere',
    ',',
    'catilina',
    ',',
    'patientia',
    'nostra',
    '?']

   
Convert J to I, V to U
----------------------

.. code-block:: python

   In [1]: from cltk.stem.classical_latin.j_and_v_converter import JVReplacer

   In [2]: j = JVReplacer()

   In [3]: j.replace('vem jam')
   Out[3]: 'uem iam'

{'fraction_non_alphanumeric': 0.13643331630045988, 'fraction_numerical': 0.025447112927950946, 'mean_word_length': 2.575447570332481, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 25, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6240105', 'n_tokens_mistral': 3456, 'n_tokens_neox': 3133, 'n_words': 1092}