<jupyter_start><jupyter_text>Loading the model takes a while.<jupyter_code>asptagger = ASPTagger()
asptagger.load(model)<jupyter_output><empty_output><jupyter_text>POS-taggingSoMeWeTa is a POS-tagger.<jupyter_code>for sentence in sentences:
    tagged_sentence = asptagger.tag_sentence(sentence)
    print("\n".join(["\t".join(t) for t in tagged_sentence]), "\n", sep="")<jupyter_output><empty_output><jupyter_text>Use together with somajoSoMeWeTa requires that the input texts are tokenized and split into sentences. Tokenization and sentence splitting have to be consistent with the corpora the tagger model has been trained on.<jupyter_code>model = (
    "/home/inga/projects/corpus-workbench/someweta/english_newspaper_2017-09-15.model"
)
name = "../data/Original/iued_test_original.txt"
with open(name, "r") as myfile:
    data = myfile.read().replace("\n", "")

asptagger = someweta.ASPTagger()
asptagger.load(model)
# See https://github.com/tsproisl/SoMaJo#using-the-module
tokenizer = somajo.SoMaJo("de_CMC", split_camel_case=False)
sentences = tokenizer.tokenize_text_file(name, paragraph_separator="empty_lines")
for sentence in sentences:
    tokens = [token.text for token in sentence]
    tagged_sentence = asptagger.tag_sentence(tokens)
    print("\n".join("\t".join(t) for t in tagged_sentence), "\n", sep="")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12850812407680945, 'fraction_numerical': 0.005908419497784343, 'mean_word_length': 6.207446808510638, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '634180', 'license': 'MIT', 'n_tokens_mistral': 482, 'n_tokens_neox': 456, 'n_words': 95, 'path': 'notebooks/sowemeta_test.ipynb', 'repo_name': 'ssciwr/argumentation-management', 'size': '1354'}