<jupyter_start><jupyter_text>4. Topic modelling This notebook will introduce you to the basics of topic modelling. You'll learn how identify topics in documents using the tf-idf matrix and ML classifier. Legend of symbols:- 🤓: Tips- 🤖📝: Your turn- ❓: Question- 💫: Extra exercise   3.1. Tf-idf matrix + ML classifier In this section, we'll learn how to do topic modelling based on bag of words and ML classification.  3.1.1. Build tf-idf matrix First, let's import the news dataset:<jupyter_code>import pandas as pd

df = pd.read_csv('../data/news.csv')
len(df)
df.head()<jupyter_output><empty_output><jupyter_text>To begin with, we need to build a **tf-idf matrix** with the corpus. Let's start with cleaning the text:<jupyter_code># We are going to start with applying a first round of data cleaning (square brackets, lower case)  
import re
import string

def clean_text(text):
    """
    This function
    makes text lowercase, 
    remove text in square brackets, 
    remove punctuation and 
    remove words containing numbers.
    """
    text = str(text).lower() # make text lower case
    text = re.sub('\[.*?\]', '', str(text)) # remove text in square brackets
    text = re.sub('[%s]' % re.escape(string.punctuation), '', str(text)) # remove punctuation
    text = re.sub('\w*\d\w*', '', str(text)) # remove words containing numbers.
    return text

df['corpus_clean'] = df['corpus'].apply(clean_text)
df['corpus_clean'].head()<jupyter_output><empty_output><jupyter_text>Now the text is cleaned, we're ready to build the tf-idf matrix:<jupyter_code># We are going to create a document-term matrix using TfidfVectorizer, and exclude common English stop words
from sklearn.feature_extraction.text import TfidfVectorizer

vec = TfidfVectorizer(max_features = 200, stop_words='english')
data_cv = vec.fit_transform(df.corpus_clean)
data_dtm = pd.DataFrame(data_cv.toarray(), columns=vec.get_feature_names())
data_dtm.index = df.index
data_dtm.head()<jupyter_output><empty_output><jupyter_text>🤓 As you might realised, there exists words (columns) that express the same idea: (woman, women), (work, workers, working). So, we need to use lemmatization or stemming.<jupyter_code>import spacy
nlp = spacy.load('en_core_web_sm')
from nltk import SnowballStemmer
stemmer = SnowballStemmer('english')

def stemming(new):
    """
    This function reduces 
    words to stems.
    """
    tokens = [[token.text for token in sentence] for sentence in nlp(new).sents]
    stems = [[stemmer.stem(token) for token in sentence] for sentence in tokens]
    stems = " ".join(str(token) for token in stems)
    return stems

df['corpus_stem'] = df['corpus_clean'].apply(stemming)
df['corpus_stem'].head()<jupyter_output><empty_output><jupyter_text>Now, let's compute the tf-idf matrix again:<jupyter_code>data_cv = vec.fit_transform(df.corpus_stem)
data_dtm_stem = pd.DataFrame(data_cv.toarray(), columns=vec.get_feature_names())
data_dtm_stem.index = df.index
data_dtm_stem.head()<jupyter_output><empty_output><jupyter_text>Now, we will add the topic of each document by adding the topic column to ** data_dtm_stem **.<jupyter_code>df_ml = pd.concat([data_dtm_stem.reset_index(drop=True), df.topic.reset_index(drop=True)], axis=1)
df_ml.head()<jupyter_output><empty_output><jupyter_text>🤖📝 **Your turn** Build a ** data_dtm_lem ** using lemmatization. Is there any difference?  3.1.2. Build machine learning classifier Now, we are ready to classify a ML algorithm to **detect** which new corresponds to climatic change on the dataset. To do so, we're going to apply a *decision tree* classifier. 🌍 https://scikit-learn.org/stable/modules/tree.html🌍 https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.htmlsklearn.tree.DecisionTreeClassifier First, we'll create a dummy variable that shows which new correponds to climatic topic:<jupyter_code>df_ml['topic_climatic_dummy'] = 0

df_ml.loc[df['topic'] == 'climatic', 'topic_climatic_dummy'] = 1
df_ml.head()<jupyter_output><empty_output><jupyter_text>We will split the dataset into training and testing.  ** Sklearn ** has already a function thas does it:<jupyter_code>from sklearn.model_selection import train_test_split

df_train, df_test, topic_train, topic_test = train_test_split(df_ml.iloc[:,:-2], df_ml.iloc[:,-1], test_size=0.33, random_state=42)<jupyter_output><empty_output><jupyter_text>And now, let's train our classifier with our training data:<jupyter_code>from sklearn.tree import DecisionTreeClassifier

dt = DecisionTreeClassifier(max_depth=4)
dt = dt.fit(df_train, topic_train)<jupyter_output><empty_output><jupyter_text>Finally, we test the performance of the classifier with our testing data:<jupyter_code>from sklearn.metrics import accuracy_score

topic_pred = dt.predict(df_test)
score = accuracy_score(topic_test, topic_pred)

score<jupyter_output><empty_output><jupyter_text>Oh wow! Our decision tree correclty classifies 96.85% of news Let's inspect the tree: 🌍 https://scikit-learn.org/stable/auto_examples/tree/plot_unveil_tree_structure.htmlsphx-glr-auto-examples-tree-plot-unveil-tree-structure-py<jupyter_code>from sklearn import tree
import graphviz 

dot_data = tree.export_graphviz(dt, out_file=None) 
graph = graphviz.Source(dot_data) 

dot_data = tree.export_graphviz(dt, out_file=None, 
                      feature_names=df_train.columns,    
                      filled=True, rounded=True,  
                      special_characters=True)  
graph = graphviz.Source(dot_data)  
graph<jupyter_output><empty_output><jupyter_text>🤖📝 **Your turn** Build a classifier to classify *child _soldiers*.<jupyter_code>df_ml['topic_child_soldiers_dummy'] = 0

df_ml.loc[df['topic'] == 'child soldiers', 'topic_child_soldiers_dummy'] = 1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1124978332466632, 'fraction_numerical': 0.005026867741376321, 'mean_word_length': 5.490438695163105, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '248886', 'license': 'MIT', 'n_tokens_mistral': 2010, 'n_tokens_neox': 1918, 'n_words': 505, 'path': 'notebooks/04_topic-modelling.ipynb', 'repo_name': 'anavaldi/digital-methods-text-mining', 'size': '5769'}