<jupyter_start><jupyter_text>Automatic Prediction of Lezgi Morpheme BreaksThis program does supervised morphological segmentation of all morphemes and glossing of either just affixes or affixes and stems. This classifier can be used on any language but it expects cleanly segmented, glossed, and POS-tagged (lexical categories) as training data. This program is intended to quickly increase the amount of accessible data from low resource, and often endangered, languages.  It is considered successful if it reaches 80% accuracy. This goal comes from the Pareto Principle - the idea that 20% of one's effort produces 80% of one's results, and vice versa. This program should accurately complete 80% of the annotations, leaving the most interesting and informative 20% for the human linguist to complete.This project was inspired by an ongoing fieldwork project. A native Lezgi speaker who has no background in linguistics has been annotating the collection of texts. She has quickly learned basic morphology and gained FLEx skills. However, simultaneously learning and doing basic linguistic analysis produces inaccurate and inconsistent annotations. It is also time-consuming. Many of the mistakes are due to the repetitive nature of the work. Not every part of speech has inflectional morphology. The annotator is most likely to skip over essential words with simple morphology, such as ergative case-marked arguments, and concentrate on morphologicaly complex words. Once the training is complete, the program should predict morpheme breaks and affix glosses for any text that has been labeled with parts of speech. Identifying parts of speech is required because this seems a reasonable task for a non-linguist native speaker. The data used in this example does include two distinctions in Lezgi that might be difficult without linguistic training. Participles are distinguished from verbs, but Lezgi participles end in a unique letter. Demonstrative pronouns are distinguished from pronouns. This distinction was used primarily because it was already consistently annotated in the data. This example uses Lezgi [lez], a Nakh-Daghestanian language spoken in Russia and Azerbaijan. Lezgi is an agglutinating language that is overwhelmingly suffixing. The training and test data came from a collection of 21 transcribed oral narratives spoken in the Qusar dialect of northwest Azerbaijan. Nine texts with about 2,500 words were used for training data after having been cleanly annotated with morpheme breaks and part of speech. All but three of affixes were glossed. Many of the stems are not glossed. The FlexText XML export labels each morpheme as stem, suffix, or prefix.   Preprocessing Data This process assumes that 1) the data has been analyzed in FLEx and exported as a FlexText, then saved with an .xml file extension, 2) words have been annotated in FLEx for part of speech, (for the example - verb, participle, adjective, adverb, noun/proper noun, particle, (personal) pronoun, demonstrative, and postposition), 3) morpheme breaks are consistent, and 4) all affixes, but not necessarily stems, are glossed.<jupyter_code>#API for parsing XML docs
import xml.etree.ElementTree as ET
from itertools import chain
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelBinarizer
import sklearn
import pycrfsuite
from collections import Counter<jupyter_output><empty_output><jupyter_text>The function below takes the data from the FLExText XML export. The data is read from the XML file and broken down by morphemes. The POS and a "label" is associated with each word. The label for stems is "stem" if stems are not being glossed, otherwise it is their gloss as it always is for affixes.<jupyter_code>def XMLtoWords(filename,stems=False):
    '''Takes FLExText text as .xml. Returns data as list: [[[[[[morpheme, gloss], pos],...],lexemes],sents]].
    Ignores punctuation. Morph_types can be: stem, suffix, prefix, or phrase when lexical item is made up of two words.'''
    
    datalists = []

    #open XML doc using xml parser
    root = ET.parse(filename).getroot()

    for text in root:
        for paragraphs in text:
            #Only get paragraphs, ignore metadata.
            if paragraphs.tag == 'paragraphs':
                for paragraph in paragraphs:
                    #jump straight into items under phrases
                    for phrase in paragraph[0]:
                        sent = []
                        #ignore first item which is the sentence number
                        for word in phrase[1]:
                            #ignore punctuation tags which have no attributes
                            if word.attrib:
                                lexeme = []
                                for node in word:
                                    if node.tag == 'morphemes':
                                        for morph in node:
                                            morpheme = []
                                            #note morph type 
                                            morph_type = morph.get('type')
                                            #Treat MWEs or unlabeled morphemes as stems.
                                            if morph_type == None or morph_type == 'phrase':
                                                morph_type = 'stem'                                            
                                            for item in morph:
                                                #get morpheme token
                                                if item.get('type') == 'txt':
                                                    form = item.text
                                                    #without hyphens demarcating affixes
                                                    if morph_type == 'suffix':
                                                        form = form[1:]
                                                    if morph_type == 'prefix':
                                                        form = form[:-1]
                                                    morpheme.append(form)
                                                #get affix glosses
                                                if item.get('type') == 'gls' and morph_type != 'stem':
                                                    morpheme.append(item.text)
                                                #get stem "gloss":'stem' or actual gloss
                                                    if item.get('type') == 'gls' and morph_type == 'stem':
                                                        if not stems:
                                                            morpheme.append(morph_type)
                                                        else:
                                                            morpheme.append(item.text)
                                            lexeme.append(morpheme)
                                    #get word's POS
                                    if node.get('type') == 'pos':
                                        lexeme.append(node.text)
                                sent.append(lexeme)
                        datalists.append(sent)
    return datalists<jupyter_output><empty_output><jupyter_text>Here the morphemes are broken down by letter. Each letter is associated with the word's POS tag and a BIO tag is added to its label/gloss. "B" denotes the initial letter of a morpheme. "I" marks non-initial letters.<jupyter_code>def WordsToLetter(wordlists):
    '''Takes data from XMLtoWords: [[[[[[morpheme, gloss], pos],...],words],sents]]. 
    Returns [[[[[letter, POS, BIO-label],...],words],sents]]'''

    letterlists = []
    
    for phrase in wordlists:
        sent = []
        for lexeme in phrase:
            word = []
            #Skip POS label
            for morpheme in lexeme[:-1]:
                #use gloss as BIO label
                label = morpheme[1]
                #Break morphemes into letters
                for i in range(len(morpheme[0])):
                    letter = [morpheme[0][i]]
                    #add POS label to each letter
                    letter.append(lexeme[-1])
                    #add BIO label
                    if i == 0:
                        letter.append('B-' + label)
                    else:
                        letter.append('I-' + label)
                        #letter.append('I')
                    word.append(letter)
            sent.append(word)
        letterlists.append(sent)
    
    return letterlists<jupyter_output><empty_output><jupyter_text>Below two functions above are called and the data is split into train/test sets.With a corpus of a little less than 2,500 words, I originally tried a 90/10 split. The accuracy results ranged from 92% to 97% but the test data was seeing a dozen or less labels. An 80/20 random split results has less range in accuracy (within 2%) but still averages 94%. The number of labels the test data encounters, however, is nearly doubled.<jupyter_code>#Randomize and split the data
traindata,testdata = train_test_split(WordsToLetter(XMLtoWords("FLExTxtExport2.xml")),test_size=0.2)<jupyter_output><empty_output><jupyter_text>CRFSuite  Define FeaturesIt is assumed that a "phrase" in FLEx is equivalent to a complete sentence. In reality, some "phrases" contain more than one sentence, some contain only a sentence fragment. This means that the word position in the sentence is often inaccurate, but it was retained to take into account Lezgi's strong tendency for verb-final word order. Affixes are rarely more than 3 letters long, so features include the previous and next 1-4 letters. This ensures that the program is viewing at least one letter in the previous/next morpheme. More often it is viewing the whole previous/next 1-2 morphemes. Since Lezgi is primarily suffixing, the position of a letter in a word is counted from the end of the word.<jupyter_code>def extractFeatures(sent):
    '''Takes data as [[[[[letter, POS, BIO-label],...],words],sents]].
    Returns list of words with characters as features list: [[[[[letterfeatures],POS,BIO-label],letters],words]]'''
    
    featurelist = []
    senlen = len(sent)
    
    #each word in a sentence
    for i in range(senlen):
        word = sent[i]
        wordlen = len(word)
        lettersequence = ''
        #each letter in a word
        for j in range(wordlen):
            letter = word[j][0]
            #gathering previous letters
            lettersequence += letter
            #ignore digits             
            if not letter.isdigit():
                features = [
                    'bias',
                    'letterLowercase=' + letter.lower(),
                    'postag=' + word[j][1],
                ] 
                #position of word in sentence and pos tags sequence
                if i > 0:
                    features.append('prevpostag=' + sent[i-1][0][1])
                    if i != senlen-1:
                        features.append('nxtpostag=' + sent[i+1][0][1])
                    else:
                        features.append('EOS')
                else:
                    features.append('BOS')
                    #Don't get pos tag if sentence is 1 word long
                    if i != senlen-1:
                        features.append('nxtpostag=' + sent[i+1][0][1])
                #position of letter in word
                if j == 0:
                    features.append('BOW')
                elif j == wordlen-1:
                    features.append('EOW')
                else:
                    features.append('letterposition=-%s' % str(wordlen-1-j))
                #letter sequences before letter
                if j >= 4:
                    features.append('prev4letters=' + lettersequence[j-4:j].lower() + '>')
                if j >= 3:
                    features.append('prev3letters=' + lettersequence[j-3:j].lower() + '>')
                if j >= 2:
                    features.append('prev2letters=' + lettersequence[j-2:j].lower() + '>')
                if j >= 1:
                    features.append('prevletter=' + lettersequence[j-1:j].lower() + '>')
                #letter sequences after letter
                if j <= wordlen-2:
                    nxtlets = word[j+1][0]
                    features.append('nxtletter=<' + nxtlets.lower())
                    #print('\nnextletter:', nxtlet)
                if j <= wordlen-3:
                    nxtlets += word[j+2][0]
                    features.append('nxt2letters=<' + nxtlets.lower())
                    #print('next2let:', nxt2let)
                if j <= wordlen-4:
                    nxtlets += word[j+3][0]
                    features.append('nxt3letters=<' + nxtlets.lower())
                if j <= wordlen-5:
                    nxtlets += word[j+4][0]
                    features.append('nxt4letters=<' + nxtlets.lower())
                
            featurelist.append(features)
    
    return featurelist

def extractLabels(sent):
    labels = []
    for word in sent:
        for letter in word:
            labels.append(letter[2])
    return labels

def extractTokens(sent):
    tokens = []
    for word in sent:
        for letter in word:
            tokens.append(letter[0])
    return tokens

def sent2features(data):
    return [extractFeatures(sent) for sent in data]

def sent2labels(data):
    return [extractLabels(sent) for sent in data]

def sent2tokens(data):
    return [extractTokens(sent) for sent in data]
X_train = sent2features(traindata)
Y_train = sent2labels(traindata)

X_test = sent2features(testdata)
Y_test = sent2labels(testdata)<jupyter_output><empty_output><jupyter_text>Train the model<jupyter_code>trainer = pycrfsuite.Trainer(verbose=False)

for xseq, yseq in zip(X_train, Y_train):
    trainer.append(xseq, yseq)<jupyter_output><empty_output><jupyter_text>Set training parameters. L-BFGS (what is this) is default. Using Elastic Net (L1 + L2) regularization [ditto?].<jupyter_code>trainer.set_params({
        'c1': 1.0, #coefficient for L1 penalty
        'c2': 1e-3, #coefficient for L2 penalty
        'max_iterations': 50 #early stopping
    })<jupyter_output><empty_output><jupyter_text>The program saves the trained model to a file:<jupyter_code>model_filename = 'LING5800_lezgi.crfsuite'
trainer.train(model_filename)<jupyter_output><empty_output><jupyter_text>Make Predictions<jupyter_code>tagger = pycrfsuite.Tagger()
tagger.open(model_filename)<jupyter_output><empty_output><jupyter_text>First, let's use the trained model to make predications for just one example sentence from the test data. The predicted labels are printed out for comparison above the correct labels. Most examples have 100% accuracy.<jupyter_code>example_sent = testdata[0]
print('Letters:', '  '.join(extractTokens(example_sent)), end='\n')

print('Predicted:', ' '.join(tagger.tag(extractFeatures(example_sent))))
print('Correct:', ' '.join(extractLabels(example_sent)))<jupyter_output>Letters: х  ъ  с  а  н  т  о  с  т  а  р  л  а  г  ь  а  н  а  р  а  з  и  в  и  л  е  р  х  ь  а  н  а  ч  у  н  м  е  р  е  к  а  д  и  л  а  й  р  а  з  и  а  м  у  к  ь  н  а
Predicted: B-stem I-stem I-stem I-stem I-stem B-stem I-stem I-stem I-stem B-PL I-PL B-stem I-stem I-stem I-stem I-stem B-AOR I-AOR B-stem I-stem I-stem I-stem I-stem I-stem I-stem B-PL I-PL B-stem I-stem I-stem B-AOR I-AOR B-stem I-stem I-stem B-stem I-stem I-stem I-stem I-stem I-stem B-OBL I-OBL B-SUPER B-ELAT I-ELAT B-stem I-stem I-stem I-stem B-stem I-stem I-stem I-stem I-stem B-AOR I-AOR
Correct: B-stem I-stem I-stem I-stem I-stem B-stem I-stem I-stem I-stem B-PL I-PL B-stem I-stem I-stem I-stem I-stem B-AOR I-AOR B-stem I-stem I-stem I-stem I-stem I-stem I-stem B-PL I-PL B-stem I-stem I-stem B-AOR I-AOR B-stem I-stem I-stem B-stem I-stem I-stem I-stem I-stem I-stem B-OBL I-OBL B-SUPER B-ELAT I-ELAT B-stem I-stem I-stem I-stem B-stem I-stem I-stem I-stem I-stem B-AOR I-AOR<jupyter_text>Evaluate the ModelThe following function will evaluate how well the model performs. Unlike CRF example found at https://github.com/scrapinghub/python-crfsuite/blob/master/examples/CoNLL%202002.ipynb, this model is not designed to disregard "O" labels, since all characters that are not part of a word (e.g. digits and punctuation) are already eliminated during pre-processing.<jupyter_code>def bio_classification_report(y_correct, y_pred):
    '''Takes list of correct and predicted labels from tagger.tag. 
    Prints a classification report for a list of BIO-encoded sequences.
    It computes letter-level metrics.'''

    labeler = LabelBinarizer()
    y_correct_combined = labeler.fit_transform(list(chain.from_iterable(y_correct)))
    y_pred_combined = labeler.transform(list(chain.from_iterable(y_pred)))
    
    tagset = set(labeler.classes_)
    tagset = sorted(tagset, key=lambda tag: tag.split('-', 1)[::-1])
    class_indices = {cls: idx for idx, cls in enumerate(labeler.classes_)}
    
    return classification_report(
        y_correct_combined,
        y_pred_combined,
        labels = [class_indices[cls] for cls in tagset],
        target_names = tagset)<jupyter_output><empty_output><jupyter_text>First, we will predict BIO labels in the test data:<jupyter_code>Y_pred = [tagger.tag(xseq) for xseq in X_test]<jupyter_output><empty_output><jupyter_text>Get results for labeled position evaluation. This evaluates how well the classifier performed on each morpheme as a whole and their tags, rather than evaluating character-level.<jupyter_code>def concatenateLabels(y_list):
    '''Return list of morpheme labels [[B-label, I-label,...]morph,[B-label,...]]'''
    
    morphs_list = []
    labels_list = []
    morph = []
    for sent in y_list:
        for label in sent:
            labels_list.append(label)
            if label[0] == 'I':
                #build morpheme shape, adding to first letter
                morph.append(label)
            else:
                # Once processed first morph, add new morphemes & gloss labels to output
                if morph:
                    morphs_list.append(morph)
                #Extract morpheme features
                morph = [label]
    
    return morphs_list, labels_list

def countMorphemes(morphlist):
    counts = {}
    for morpheme in morphlist:
        counts[morpheme[0][2:]] = counts.get(morpheme[0][2:], 0) + 1
    return counts

def eval_labeled_positions(y_correct, y_pred):
    
    #group the labels by morpheme and get list of morphemes
    correctmorphs,_ = concatenateLabels(y_correct)
    predmorphs,predLabels = concatenateLabels(y_pred)
    #Count instances of each morpheme
    test_morphcts = countMorphemes(correctmorphs)
    pred_morphcts = countMorphemes(predmorphs)
    
    correctMorphemects = {}
    idx = 0
    num_correct = 0
    for morpheme in correctmorphs:
        correct = True
        for label in morpheme:
            if label != predLabels[idx]:
                correct = False
            idx += 1
        if correct == True:
            num_correct += 1
            correctMorphemects[morpheme[0][2:]] = correctMorphemects.get(morpheme[0][2:], 0) + 1
    #calculate P, R F1 for each morpheme
    results = ''
    for firstlabel in correctMorphemects.keys():
        lprec = correctMorphemects[firstlabel]/pred_morphcts[firstlabel]
        lrecall = correctMorphemects[firstlabel]/test_morphcts[firstlabel]
        results += firstlabel + '\t\t{0:.2f}'.format(lprec) + '\t\t' + '{0:.2f}'.format(lrecall) + '\t' + '{0:.2f}'.format((2*lprec*lrecall)/(lprec+lrecall)) +'\t\t' + str(test_morphcts[firstlabel]) + '\n'
    #overall results
    precision = num_correct/len(predmorphs)
    recall = num_correct/len(correctmorphs)
    
    print('\t\tPrecision\tRecall\tf1-score\tInstances\n\n' + results + '\ntotal/avg\t{0:.2f}'.format(precision) + '\t\t' + '{0:.2f}'.format(recall) + '\t' + '{0:.2f}'.format((2*precision*recall)/(precision+recall)))<jupyter_output><empty_output><jupyter_text>Then, we check the results and print a report of the results. These results are for character level.<jupyter_code>eval_labeled_positions(Y_test, Y_pred)
print(bio_classification_report(Y_test, Y_pred))<jupyter_output>precision    recall  f1-score   support

     B-ADSS       1.00      1.00      1.00         1
      B-AOR       0.92      1.00      0.96        12
      B-DAT       0.86      0.75      0.80         8
     B-ELAT       1.00      1.00      1.00         2
      B-ERG       0.00      0.00      0.00         2
      B-FOC       0.86      1.00      0.92         6
      B-FUT       0.00      0.00      0.00         2
      B-GEN       1.00      0.75      0.86         4
          I       0.95      0.99      0.97       417
      B-IMC       0.00      0.00      0.00         1
     B-IMPF       1.00      0.75      0.86         4
    B-INESS       0.00      0.00      0.00         1
      B-INF       1.00      1.00      1.00         1
      B-NEG       1.00      1.00      1.00         2
  B-NEG.PST       0.00      0.00      0.00         1
      B-OBL       0.80      0.67      0.73        12
     B-PERF       1.00      0.75      0.86         4
       B-PL       0.89      0.67      0.76   [...]<jupyter_text>The model, with a 80/20 split, produces an average accuracy of 94% with a less than 2% range over randomized test data. This is significantly above the targeted accuracy of 80%. Table 1 shows the results of one run. |__label__|__precision__|__recall__|__f1-score__|__instances__||---------|-------------|----------|------------|-------------||B-AOR|1.00|0.88|0.94|17||B-DAT|0.92|1.00|0.96|11||B-ELAT|0.67|1.00|0.80|2||B-ENT|0.33|0.50|0.40|2||B-ERG|0.00|0.00|0.00|3||B-FOC|0.86|1.00|0.92|6||B-FUT|0.00|0.00|0.00|2||B-GEN|0.50|0.33|0.40|6||B-HORT|0.00|0.00|0.00|1||I|0.95|0.99|0.97|480||B-INESS|1.00|0.33|0.50|3||B-MSDR|0.00|0.00|0.00|2||B-NEG|1.00|1.00|1.00|1||B-OBL|0.80|0.60|0.69|20||B-PL|0.50|0.50|0.50|2||B-POESS|1.00|1.00|1.00|3||B-PTP|1.00|0.67|0.80|3||B-SBST|1.00|0.50|0.67|2||B-SUPER|0.67|1.00|0.80|2||B-TEMP|0.00|0.00|0.00|1||B-UNK|0.00|0.00|0.00|1||B-stem|1.00|0.99|0.99|138||__avg / total__|__0.94__|__0.94__|__0.94__|__708__|Table 1: Results of morpheme predictionsAs might be expected, the classifier has less success predicting less frequent labels. This makes the results of the I labels (non-initial letters in a morpheme) surprising, until one considers that transitions between morphemes may not always be clear. Other results become more interesting with some knowledge of Lezgi morphology. The inessive (INESS) and ergative (ERG) case and the oblique stem morpheme (OBL) are identical. The only difference between the first two is the tendency of sentence position, even with Lezgi's free word order. The difference between the latter two is that the ergative morpheme is word final and the the oblique stem is follow by another case morpheme. |__precision__|__recall__|__f1-score__||-----|------|------||0.54|0.49|0.49|Table 2: Average score of affix labels only.The classifier has most success identifying stem morphemes (STEM) and non-initial letters (I), the majority of which belong to stem morphemes. It has less success with identifying affixes. The classifier is clearly adept at splitting affixes from stems and this is already helpful to human annotators but it would be less helpful splitting strings of affixes and correcly glossing them. Table 2 shows average precision, recall, and f1-score of affix labels is much less accurate than the overall accuracy. This is most likely due in part to homonymic affixes and in part to the fewer instances of affixes compared to stems. As the more texts are correctly annotated with the help of the model, more data can be fed into the training, hopefully increasing the accuracy and incrementally speeding the annotation process.The data was also run on a bidirectional sequence-to-sequence deep neural network with attention. The hidden layer size was set at 128, the batch size as 32, the teacher forcing ratio at 0.5. The results in Table 3 indicate that with a small amount of data a supervised classifier can produce equal or better results than a neural network.|epochs|accuracy||------|--------||50|0.57||100|0.75||200|0.90||300|0.92||__500__|__0.93__||600|0.89||1000|0.91|Table 3: Results of deep neural network  What the Classifier LearnedBy using methods of the crfsuite, we can look insider classifier and see what it learned. From the example printout in Table 3, we can see, for example, that the stem, elative (ELAT), imperfective (IMPF), aorist (AOR), perfective (PF), and plural (PL) morphemes most often consist of more than one letter  but superessive (SUPER), oblique (OBL), and subessive (SUB) morphemes usually consist of just one letter. We can also see that temporal converb (TEMP) morpheme often follows the participle (PTP) morpheme, and another case morpheme tends to follow the oblique, superessive, and subessive case morphemes. These patterns correspond to the facts of Lezgi morphology. On the other hand, both Table 4 and Table 5 indciate that is highly likelythat a  genitive case (GEN) morpheme will be a prefix, which is impossible. This indicates that the affix type (prefix or suffix) might be a useful feature to include.|-|-|-|weights||---|---|----|-----||B-SUPER| ->| B-ELAT|  4.820010||B-OBL|  ->| B-SPSS|  3.806645||B-SUB|  ->| B-ELAT|  3.444584||B-OBL|  ->| B-DAT|   2.946830||B-stem| ->| I|       2.258064||B-OBL|  ->| B-GEN|   2.247354||I|      ->| B-OBL|   1.913825||B-stem| ->| B-OBL|   1.862016||B-ELAT| ->| I|       1.711584||B-PTP|  ->| B-TEMP|  1.620690||B-IMPF| ->| I|       1.300227||B-AOR|  ->| I|       1.252594||B-PERF| ->| I|       1.135483||B-PL|   ->| I|       1.043438||B-GEN|  ->| B-stem|  0.956780|Table 4: Top most likely transitionsOn the other hand, Table 5, for example, indicates that the negative affix rarely follows a non-initial letter of another morpheme. This is accurate because the negative affix is the only prefix in the language. It is not surprising that the transition still has a greater than zero probability since it is often only one letter long and this letter may be found at the beginning of any word.|-|-|-|weights||---|---|---|----||B-ERG|  ->| B-stem|  0.295926||B-TEMP| ->| I|       0.254567||B-SBST| ->| I|       0.249661||I|      ->| B-NEG|   0.221662||B-INF|  ->| B-stem|  0.196340||I|      ->| B-DAT|   0.057729||B-NEG|  ->| B-stem  |0.013683||I|      ->| B-stem|  0.009557||I|      ->| B-ERG|   0.000074||I|      ->| B-SUPER| -0.000692||I|      ->| B-FOC|   -0.003919||I|      ->| B-SBST|  -0.023268||B-OBL|  ->| I|       -0.034257||B-INESS| ->| I|       -0.157967||I|      ->| B-GEN|   -1.180139|Table 5: Top most unlikely transitions<jupyter_code>info = tagger.info()

def print_transitions(trans_features):
    '''Print info from the crfsuite.'''
    
    for (label_from, label_to), weight in trans_features:
        print("%-6s -> %-7s %0.6f" % (label_from, label_to, weight))

print("Top likely transitions:")
print_transitions(Counter(info.transitions).most_common(15))

print("\nTop unlikely transitions:")
print_transitions(Counter(info.transitions).most_common()[-15:])<jupyter_output>Top likely transitions:
B-SUPER -> B-ELAT  4.491556
B-OBL  -> B-SPSS  3.999741
B-SUB  -> B-ELAT  2.783580
B-OBL  -> B-GEN   2.581786
B-stem -> I       2.441328
B-OBL  -> B-DAT   2.354065
I      -> B-OBL   2.319193
B-ELAT -> I       2.027671
B-AOR  -> I       1.810235
B-OBL  -> B-SUPER 1.537023
B-IMPF -> I       1.262218
B-GEN  -> B-stem  1.204093
B-PL   -> I       1.181239
B-PERF -> I       1.172315
B-ERG  -> B-stem  1.059358

Top unlikely transitions:
B-stem -> B-OBL   0.263636
B-stem -> B-PL    0.178466
B-POESS -> I       0.174021
B-FOC  -> I       0.163086
I      -> B-stem  0.081534
B-NEG  -> B-stem  0.062830
B-INESS -> B-stem  0.062172
I      -> B-AOR   0.033769
B-ENT  -> I       0.009679
B-PTP  -> B-stem  0.005626
B-GEN  -> B-FOC   0.000337
B-OBL  -> I       -0.004404
I      -> B-SBST  -0.031958
B-INESS -> I       -0.095406
I      -> B-GEN   -0.620703<jupyter_text>We can make some observations about the state features. For example, Table 6 indicates that the model rightly recognized that the stem is nearly always at the beginning of the word and there are no consistent feature to identify the non-initial letters of various morphemes. |weight|label|feature||---|---|----||13.385742| B-stem| BOW||6.80475| I|      bias||5.169367| B-PL|   nxt2letters=<ур||5.142534| B-DAT|  letterLowercase=з||4.858094| B-NEG|  letterLowercase=ш||4.568794| B-PTP|  letterLowercase=й||4.513613| B-PST|  letterLowercase=й||4.361416| B-ADSS| letterLowercase=в||4.269127| B-PL|   nxtletter=<р||4.216564| B-FOC|  nxtletter=<и||4.203677| B-GEN|  letterLowercase=н||4.023482| B-INF|  letterLowercase=з||3.977504| B-IMPF| letterLowercase=з||3.868088| B-NEG|  letterLowercase=ч||3.636859| B-FOC|  letterLowercase=н|Table 6: Top positive featuresTable 7 indicates that certain letter sequences might be less likely to begin a morpheme. One interesting observation that could be easily confirmed by a corpus study is that the focus particle is least likely to occur on a verb than on any other lexical category. |weight|label|feature||---|----|---||-0.606766| I|      prev2letters=ча>||-0.679616| I|      letterLowercase=ч||-0.704380| I|      prevletter=ш>||-0.741532| I|      prev2letters=ич>||-0.833423| B-FOC|  postag=v||-0.937032| B-FOC|  bias||-1.029693| I|      prev3letters=вал>||-1.071785| I|      nxtletter=<й||-1.073034| I|      prev3letters=гьу>||-1.126576| I|      prev2letters=ди>||-1.150632| B-AOR|  bias||-1.201650| I|      letterLowercase=н||-1.240373| I|      letterLowercase=з||-1.250568| I|      prevletter=р>|Table 7: Top negative<jupyter_code>def print_state_features(state_features):
    for (attr, label), weight in state_features:
        print("%0.6f %-6s %s" % (weight, label, attr))    

print("Top positive:")
print_state_features(Counter(info.state_features).most_common(15))

print("\nTop negative:")
print_state_features(Counter(info.state_features).most_common()[-15:])<jupyter_output>Top positive:
13.322432 B-stem BOW
5.645198 I      bias
5.374702 B-DAT  letterLowercase=з
5.080712 B-PL   nxt2letters=<ур
5.016308 B-NEG  letterLowercase=ш
4.591081 B-IMPF letterLowercase=з
4.378502 B-AOR  letterLowercase=н
4.364921 B-GEN  letterLowercase=н
4.319225 B-PL   nxtletter=<р
4.172178 B-FOC  letterLowercase=н
4.108242 B-FOC  nxtletter=<и
4.009077 B-INF  letterLowercase=з
3.860550 B-ADSS letterLowercase=в
3.858400 B-MSDR letterLowercase=н
3.552755 B-INELAT nxtletter=<й

Top negative:
-0.819617 I      letterLowercase=з
-0.855908 B-FOC  bias
-1.046632 I      prev2letters=ча>
-1.071244 I      prev2letters=уш>
-1.171039 I      nxt2letters=<ур
-1.234349 I      prev2letters=ил>
-1.320892 I      prev2letters=ун>
-1.366267 I      prev2letters=ди>
-1.404496 I      prev3letters=вал>
-1.575353 I      prev3letters=ава>
-1.649316 I      prevletter=р>
-1.717653 B-AOR  bias
-1.778882 I      prev2letters=да>
-1.805477 I      letterLowercase=д
-2.029026 I      prev2letters=уд>
{'fraction_non_alphanumeric': 0.10549429414027439, 'fraction_numerical': 0.049621746377740736, 'mean_word_length': 2.306518282988871, 'pattern_counts': {'":': 1, '<': 78, '<?xml version=': 0, '>': 148, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 63, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '323599', 'license': 'MIT', 'n_tokens_mistral': 10688, 'n_tokens_neox': 9531, 'n_words': 3022, 'path': 'MorphSegmenter_Example-Lezgi.ipynb', 'repo_name': 'theSarahRu/FLExMorphSegmenter', 'size': '31196'}