<jupyter_start><jupyter_text>Load the corpus<jupyter_code>P = PostOcr(4)<jupyter_output>Loading TF data ...
Loading tf data from ~/github/annotation/clariah-dr/tf/daghregister/004/0.1
done
24788 words in 229023 occurrences
All words frequencies written to ~/github/annotation/clariah-dr/postocr/daghregister/004/wordfreqs.tsv
word frequency│word occurrence
──────────────┼─────────────────────────────────────────────────────────────────────────────────────
          7714│■1
          5392│■1
          3770│■■■3
          3152│■1
          2635│■1
          2203│■■2
          1540│■■■■■■6
          1288│■■■3
          1077│■■■■■■6
           900│■1
           753│■■■■■■■■■9
           629│■■■■■■■7
           526│■■■■■■■■■■■■13
           440│■■■■■■■■■■■■■■17
           368│■■■■■■■■■9
           308│■■■■■■■■■■■■■■■■■■26
           257│■■■■■■■■■■■■■■■■21
           215│■■■■■■■■■■■■■■■18
           180│■■■■■■■■■■■■■■■■■25
           150│■■■■■■■■■■■■■■■■■■28
           126│■■■■■■■■■■■■■■■■■■■■■40
           105│■■■■■■■■■■■■■■■■■■■■■■■48
            88│■■■■■■■■■■■■■■■■■■■■■■■47
  [...]<jupyter_text>Morpheme splitting by information reductionIdea: pick a word and try to split off morpheme-like bits from the start and from the end of a word.We do it in rounds, in each round at most one morf per side of the word is split off.What we do is: ```word = morf + main```and/or```word = main + morf```How do we determine whether a possible split is a good split?By using several indicators:1. the number of other *mains* that can combined with the *morf*2. the number of other *morfs* that can be combined with the *main*3. the quality of those other *mains*4. the quality of those other *morfs*.We then weigh the values found for these indicators, and take a decision.When we look for possible *morfs* that combine with a *main*, we also want to knowif the *main* with the empty *morf* is a valid word.Likewise, when we look for possible *mains* that combine with a *morf*,we take the empty *main* into account.We use an upper bound for morf lengths.In short words, if we detect overlapping morfs at opposing ends, we pick the one with the best indicators  PreparationWe compute data in advance that we will need frequently: the compositions of all word forms.<jupyter_code>P.getCompositions()<jupyter_output>82115 distinct suffixes
85590 distinct prefixes<jupyter_text>Quality of a word splitWe define the quality of word splits in an almost cyclical manner.First we define a very coarse notion of quality.Based on that we refine the notion.And again, and again.We split the word into two parts one called morf and one called lex.The morf is no longer than a limited amount of characters, e.g. 5.The morf is either at the end or at the start of a word.The lex is the rest of the word.A morf may be empty.A lex may not be empty, but it may be short. Round 1We start with the quality of a lex:We look at the number of morfs that combine with a lex at each side.If for a side that number is 1, and it is not the empty morf,it means that the lex is not a word itself, and that it can be extended to a wordin exactly one wayHence the lex is not a real lex, but just a fragment of a lex.We give it quality 0.If the lex can be extended in very many ways, it is also a sure sign that the lex is nota real lex, but just a fragment of a lex.We give it quality 0.With a positive, moderate amount of possible completions, we give it a positive quality.We count only the longest extensions, i.e. if one suffix is a prefix of another suffix, we only count the latter suffix.And if one prefix is a suffix of another prefix, we only count the former prefix.We assign a quality as follows:* if there are no morfs that complete the lex: 0* else if the empty morf completes the lex: 1* else if n is the number of morfs that completes the lex:  * if n = 1: 0  * else if n = 2: 1  * else if n < 5: 2  * else if n < 8: 1  * else 0  We do this separately for both sides and take the maximum of them.Then the quality of a morf.Morfs at the end are separately calculated as morfs at the beginning.We look at each lex-candidate that can complete it. For each lex-candidate we compute the round-1-quality.Then we take the average of all those qualities.That is the quality of the morf. Round n + 1We start with the quality of a lex. If the lex has round-n quality 0 it also has round n+1 quality zero.Otherwise: it is the average of the round-n quality of the morphs that combine with it,The the quality of a morf.We take the average of all the round-n+1 qualities of the lex that combine with it.  Quality of a word splitThe computation of the quality measures is potentially costly.We do not compute them in advance for the whole corpus, but on a just-in-time basis.Whenever we have computed a quality measure, we memoize it, so that it can be retrieved rather thancomputed when it is needed again.This pays off, because the quality of an item is typically computed by applying a formula tothe qualities of a (big) number of other items.We experiment with notions of quality.Suppose we have `word = morf + main`.A simple notion of quality is the amount of possible *mains* that are possible after the *morf*.But this will give an enormous quality to one-letter morfs. In general, there is a big number of words starting with thesame letter.A bit more sophisticated is to weigh all *mains*, by counting how many other *morfs* can precede it.If `mainOther` is another main that can follow `morf`, but `mainOther` cannot be preceded by any other *morf* than `morf`,it does not contribute to the quality of `morf` as morpheme.But if it can be preceded by multiple *morfs*, it contributes to the fact that `morf` is a morpheme.So the quality of a *morf* is the weighted sum of all its (non-empty) *mains* with which it forms a word,where each main is weighted by the number of other *morfs* with which it can form a word, *including* the empty *morf*.The quality of a *main* is the non-weighted sum of all its other (possibly empty) *morfs* with which it forms a word.**Note the asymmetry:**In the quality of a *morph*, we weigh its *mains* in such a way that a *main* that is a word on its own does contribute.In the quality of a *main*, we weigh its *morfs* in such a way that a *morf* that is a word on its own does *not* contribute.In the quality of a *morf*, we do weigh the possible mains.In the quality of a *main*, we just take the number of other morfs, not the sum of their qualities.If a *main* has exactly one morf, then the result is 0; such mains are not good mains.How how do we weigh an empty morf exactly? If we do not say anything, it will be the sum of all words in the corpus.Instead, the empty morf has weight 1.**Recursive closure?**Note that we could also have defined the notion of quality in a recursive way:The quality of a *morf* is the sum of the qualities of its *mains*.The quality of a *main* is the sum of the qualities of its *morfs*.The question is then: how does this recursion stop?We can approach this definition by the one we have given, by computing the qualities in rounds, where in each round wecompute new qualities based on the old qualities.And then we stop when the qualities do not change anymore.But at present I have no idea whether this will ever stop.So we stick with just one iteration. Building intuitionWe show first these quality measures in some examples, in order to get intuition.When we have seen enough examples, we can proceed to define a decision procedurebased on the quality measures. Some examples.<jupyter_code>EXAMPLES = """
aanneemt
neemt
aengeschreeven
H
hunner
monterende
gelegentheyt
ongelegentheyt
heeft
gesanten
ootmoedige
""".strip().split()
P.COMPOSE[True]["eemt"]
P.clearQuality()
len(P.getFills("aann", "eemt", False, False))
P.getQuality("vr", "eemt", False, False)<jupyter_output><empty_output><jupyter_text>Examine the examples:<jupyter_code>len(P.COMPOSE[False]["n"])
len(P.getFills("n", None, True, False))
P.clearQuality()
P.showExamples(EXAMPLES)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09746079163554892, 'fraction_numerical': 0.020786656708986807, 'mean_word_length': 3.9598765432098766, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '241585', 'license': 'MIT', 'n_tokens_mistral': 2565, 'n_tokens_neox': 2381, 'n_words': 1222, 'path': 'programs/legacy/morph-try2.ipynb', 'repo_name': 'CLARIAH/wp6-daghregisters', 'size': '8034'}