<jupyter_start><jupyter_text>Tutorial 2: Mixture Models and Expectation Maximization  Exercise 1: Categorical Mixture Model (CMM)<jupyter_code># Import libraries
import numpy as np
import pandas as pd
from ast import literal_eval
import matplotlib.pyplot as plt

import gensim
from wordcloud import WordCloud, STOPWORDS

from categorical_em import CategoricalEM<jupyter_output><empty_output><jupyter_text>1.4) Play around with the dataset  Load and pre-process the dataLoad the data from the tweets_cleaned csv file as `pandas` dataframe. It contains the documents already pre-processed and cleaned after applying the following steps:1. Tokenization2. Homogeneization, which includes:    1. Removing capitalization.    2. Removing non alphanumeric tokens (e.g. punktuation signs)    3. Stemming/Lemmatisation.3. Cleaning4. Vectorization<jupyter_code>df = pd.read_csv('tweets_cleaned.csv')
df.drop_duplicates(subset="tweet", inplace=True)  # drop duplicates tweets
df['tokens'] = df['tokens'].apply(literal_eval)  # transform the string into a list of tokens
X_tokens = list(df['tokens'].values)
print('Columns: {}\n'.format(' | '.join(df.columns.values)))

print('Tweet:\n{}'.format(df.loc[1, 'tweet']))
print('Tweet cleaned:\n{}'.format(df.loc[1, 'tweets_clean']))
print('Tweet tokens:\n{}'.format(X_tokens[1]))<jupyter_output>Columns: tweet_id | timestamp | user_id | tweet | tweets_clean | tokens

Tweet:
OSINT people - please retweet, if possible. My friend is looking for women involved in OSINT. https://twitter.com/manisha_bot/status/1181594280336531457 …
Tweet cleaned:
osint people   please retweet  if possible  my friend is looking for women involved in osint
Tweet tokens:
['osint', 'peopl', 'retweet', 'possibl', 'friend', 'look', 'woman', 'involv', 'osint']<jupyter_text>Create the dictionaryWe have transformed the raw text collection in a list of documents stored in `X_tokens`, where each document is a collection of words which are the most relevant according to the semantic analysis. We now convert these data (a list of token lists) into a numerical representation (a list of vectors, or a matrix). For this purpose we use the `gensim` library.<jupyter_code>I = 120  # hyperparameter: number of different words to keep
dictionary = gensim.corpora.Dictionary(X_tokens)
print(dictionary)
dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=I)
print(dictionary)<jupyter_output>Dictionary(12243 unique tokens: ['collin', 'cum', 'domin', 'phil', 'room']...)
Dictionary(120 unique tokens: ['look', 'peopl', 'woman', 'love', 'work']...)<jupyter_text>Create Bag of Words (BoW)Let's create the numerical version of our corpus using the `doc2bow` method. In general, `D.doc2bow(token_list)` transforms any list of tokens into a list of tuples `(token_id, n)`, one per each token in `token_list`, where `token_id` is the token identifier (according to dictionary `D`) and `n` is the number of occurrences of such token in `token_list`.<jupyter_code>X_bow = list()
keep_tweet = list()
for tweet in X_tokens:
    tweet_bow = dictionary.doc2bow(tweet)
    if len(tweet_bow) > 1:
        X_bow.append(tweet_bow)
        keep_tweet.append(True)
    else:
        keep_tweet.append(False)

df_data = df[keep_tweet]
N = len(df_data)<jupyter_output><empty_output><jupyter_text>Create the matrixFinally, we transform the BoW representation `X_bow` into a matrix, namely `X_matrix`, in which the n-th row and j-th column represents the number of occurrences of the j-th word of the dictionary in the n-th document. This will be the matrix used in the algorithm.<jupyter_code>X_matrix = np.zeros([N, I])
for i, doc_bow in enumerate(X_bow):
    word_list = list()
    for word in doc_bow:
        X_matrix[i, word[0]] = word[1]
X_matrix.shape<jupyter_output><empty_output><jupyter_text>1.5) Implement the EM algorithm<jupyter_code>K = 6  # hyperparameter: number of topics
i_theta = 1
i_pi = 1
model = CategoricalEM(K, I, N, delta=0.01, epochs=200, init_params={'theta': i_theta, 'pi': i_pi})
model.fit(X_matrix)<jupyter_output>ITER: 0 Q= -115633.6841 diff= 200
ITER: 5 Q= -104752.3399 diff= 644.3929
ITER: 10 Q= -103523.2635 diff= 121.6231
ITER: 15 Q= -103178.5907 diff= 47.806
ITER: 20 Q= -102993.0518 diff= 24.4851
ITER: 25 Q= -102921.4326 diff= 9.2681
ITER: 30 Q= -102893.9936 diff= 3.2741
ITER: 35 Q= -102888.0141 diff= 0.1591
ITER: 36 Q= -102888.1505 diff= -0.1364<jupyter_text>1.6) Show the ten most representative words for each topic using a wordcloud, and the ten most relevant documents for each topic Words per topic<jupyter_code>argsort = np.argsort(model.theta_matrix, axis=1)
argsort = [x[::-1] for x in argsort]
fig, axs = plt.subplots(2, 3, figsize=(30, 10))
for k in range(K):
    words = [(dictionary[m], model.theta_matrix[k][m]) for m in argsort[k][:10]]
    axs[k//3,k%3].imshow(WordCloud().fit_words(dict(words)))
    axs[k//3,k%3].set_title(f'Topic {k+1}', fontsize=25)
    axs[k//3,k%3].axis('off')<jupyter_output><empty_output><jupyter_text>Documents per topic<jupyter_code>rnk = model.r_matrix.T
rnk.shape
argsort = np.argsort(rnk, axis=1)
argsort = [x[::-1][:10] for x in argsort] 
# topic 1
df_data.iloc[argsort[0]]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12082609534838834, 'fraction_numerical': 0.04748118123914302, 'mean_word_length': 5.405438813349814, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 31, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '11149', 'license': 'MIT', 'n_tokens_mistral': 1959, 'n_tokens_neox': 1786, 'n_words': 555, 'path': 'L2/L2_solution.ipynb', 'repo_name': 'WinckelData/APMLA-WS_21-22_material', 'size': '5181'}