<jupyter_start><jupyter_text>Jupyter Playground*Jupyter notebook for testing purposes.*<jupyter_code>import sys

sys.path.insert(0, '../scripts')
from tokenizer import Tokenizer
from lemmatizer import Lemmatizer
from topicizer import Topicizer
from factorizer import Factorizer<jupyter_output><empty_output><jupyter_text>1. Tokenizer<jupyter_code>s = '"Siemens is building a factory in India for approximately 7.3 Million Euros."'
t = Tokenizer(s)
t._print_noun_chunks()<jupyter_output>Siemens
a factory
India
approximately 7.3 Million Euros<jupyter_text>2. Lemmatizer<jupyter_code>seq = 'I am a student studying various Natural Language Processing algorothms to obtain a deeper understanding.'
l = Lemmatizer()
l._print_lemmas(seq)<jupyter_output>I            PRON   561228191312463089   -PRON-
am           AUX    10382539506755952630 be
a            DET    11901859001352538922 a
student      NOUN   8980609300697527695  student
studying     VERB   4251533498015236010  study
various      ADJ    15505458595570984623 various
Natural      PROPN  16016219431787577165 Natural
Language     PROPN  2953755589855585080  Language
Processing   PROPN  5477194979658911992  Processing
algorothms   NOUN   3969165751382591598  algorothms
to           PART   3791531372978436496  to
obtain       VERB   8991116519625795538  obtain
a            DET    11901859001352538922 a
deeper       ADJ    12691978708603459222 deep
understanding NOUN   1392792401359955612  understanding
.            PUNCT  12646065887601541794 .<jupyter_text>3. Topic Modeling- LDA- Matrix Factorization<jupyter_code># LDA
t = Topicizer(os.getcwd().replace('/notebooks', '/data/articles.csv'), classes=10)
lda_cluster_data = t.get_topk(top_k=15)
# Articles with assigned topic
print(lda_cluster_data)<jupyter_output>Article  Topic
0      In the Washington of 2016, even when the polic...      0
1        <NAME> has used Twitter  —   his prefe...      0
2        <NAME> is unabashedly praising Russian...      0
3      Updated at 2:50 p. m. ET, Russian President Vl...      0
4      From photography, illustration and video, to d...      5
...                                                  ...    ...
11987  The number of law enforcement officers shot an...      2
11988    Trump is busy these days with victory tours,...      0
11989  It’s always interesting for the Goats and Soda...      3
11990  The election of D<NAME> was a surprise to...      5
11991  Voters in the English city of Sunderland did s...      1

[11992 rows x 2 columns]<jupyter_text>Non-negative Matrix FactorizationPerform SVD to obtain topics aims to minimize the *Frobenius Norm*:$\frac{1}{2} || A - WH||^2_F = \sum \sum \left( A_{ij} - (WH)_{ij }\right)^2$<jupyter_code>f = Factorizer(os.getcwd().replace('/notebooks', '/data/articles.csv'), classes=10)
svd_cluster_data = f.get_topk(top_k=15)
print(svd_cluster_data)<jupyter_output>Article  Topic
0      In the Washington of 2016, even when the polic...      1
1        <NAME> has used Twitter  —   his prefe...      1
2        <NAME> is unabashedly praising Russian...      1
3      Updated at 2:50 p. m. ET, Russian President Vl...      9
4      From photography, illustration and video, to d...      6
...                                                  ...    ...
11987  The number of law enforcement officers shot an...      8
11988    Trump is busy these days with victory tours,...      1
11989  It’s always interesting for the Goats and Soda...      7
11990  The election of <NAME> was a surprise to...      4
11991  Voters in the English city of Sunderland did s...      0

[11992 rows x 2 columns]
{'fraction_non_alphanumeric': 0.07828983897834536, 'fraction_numerical': 0.12104386451971128, 'mean_word_length': 2.7609603340292277, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '482947', 'license': 'MIT', 'n_tokens_mistral': 1409, 'n_tokens_neox': 1143, 'n_words': 380, 'path': 'code/nlp/notebooks/playground.ipynb', 'repo_name': 'lukaschoebel/LUMOS', 'size': '3637'}