<jupyter_start><jupyter_text>What kind of language is Middle High German?<jupyter_code>import sigurd.nib_marburg.mhg_lemmatizer as mhgl
import sigurd.nib_marburg.mhg_pos_tagger as mhgpt
import numpy<jupyter_output><empty_output><jupyter_text>Data retrieval<jupyter_code>forms_to_lemmata = mhgl.read_lemmatizer()
lemmata_to_forms = mhgl.read_lemma_to_forms()
list(lemmata_to_forms.keys())[10000]
lemmata_to_forms['bluotig']
list(forms_to_lemmata.keys())[10000]
forms_to_lemmata['bluotiger']
lemmata_to_pos = mhgpt.read_lemmata_to_pos()
lemmata_to_pos["bluotig"]
norm_to_pos_tagger = mhgpt.read_norm_to_pos_tagger()
norm_to_pos_tagger["bluotic"]
pos_tagger = mhgpt.read_pos_tagger()
print(pos_tagger["ADJD"][:20])<jupyter_output>['stèinëht', 'durh-vèrtig', 'ober', 'în-ge-tân', 'jung', 'ver-gihtig', 'èin-jærig', 'ge-hülfig', 'ge-stalt', 'hol', 'un-ge-sprochen', 'tèil-haft', 'wètte', 'bürtig', 'un-zal(e)-haft', 'wankel-müète', 'wunsch-lich', 'un-vèrtig', 'un-be-rèitet', 'sælig-lich']<jupyter_text>How ambiguous are MHG words?One way to define is to calculate the mean of the number of POS tags given a word form.<jupyter_code>numpy.mean([len(norm_to_pos_tagger[norm]) for norm in norm_to_pos_tagger])
numpy.mean([len(lemmata_to_pos[norm]) for norm in lemmata_to_pos])<jupyter_output><empty_output><jupyter_text>One way to define is to calculate the mean of the number of lemmata given a word form.<jupyter_code>numpy.mean([len(forms_to_lemmata[norm]) for norm in forms_to_lemmata])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13690476190476192, 'fraction_numerical': 0.007936507936507936, 'mean_word_length': 9.506944444444445, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '402390', 'license': 'MIT', 'n_tokens_mistral': 673, 'n_tokens_neox': 653, 'n_words': 105, 'path': 'sigurd/nib_marburg/rem-analysis.ipynb', 'repo_name': 'clemsciences/comparison_sigurdr_siegfried', 'size': '1512'}