<jupyter_start><jupyter_text>Introduction Natural language processing (NLP) refers to the branch of computer science—and more specifically, the branch of artificial intelligence or AI—concerned with giving computers the ability to understand text and spoken words in much the same way human beings can. Its end applications are many — chatbots, recommender systems, search, virtual assistants, etc.For this project we chose to focus on the Topic Modelling aspect of Natural Language Processing.Topic modelling is recognizing the words from the topics present in the document or the corpus of data. This is useful because extracting the words from a document takes more time and is much more complex than extracting them from topics present in the document.<jupyter_code># install if not available
%%capture
!pip install pyLDAvis
# import libraries
%%capture
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn
import spacy
import re
import nltk
from nltk.tokenize import RegexpTokenizer
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import LatentDirichletAllocation
import pickle
from sklearn.model_selection import GridSearchCV
import pyLDAvis
import pyLDAvis.sklearn
pyLDAvis.enable_notebook()
import pickle
from wordcloud import WordCloud, STOPWORDS
import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>DatasetWe used Machine Learning to explore which topics our followers on twitter are engaging with the most. Next step is to actually get the data from a Twitter api using Tweepy and other tools to pull data and refine data to get to the data we need. As a proof of concept, the region was isolated to tweets in South Africa.<jupyter_code># download the tweet dataset 
%%capture
!wget https://dsiwork.s3.amazonaws.com/dataset.csv
data = pd.read_csv("dataset.csv", parse_dates=["date_created"],   encoding="ISO-8859-1")
data.head()
# Size of dataset
print("Dataset size:", len(data))
print(data.info())
# Checking for nulls

data.isnull().sum()
#Checking for duplicates

data.duplicated().sum()<jupyter_output><empty_output><jupyter_text>Data PreprocessingAfter pulling and refining the data from the Twitter api and importing the required packages, we then converted it into a data frame and cleaned the data using a regex function to remove emojis, hashtags, extra spaces, punctuations, usernames, urls, and other unecessary signs and symbols so we can tokenize it for the next steps.<jupyter_code>def clean(text): 
    # cleans the text using regex library
    # input: text => Strings 
    # returns: text => cleaned Strings
      
    text = str(text).lower()
    text = re.sub(r'@\w+', ' ', text)
    text = re.sub('https?://\S+|www\.\S+', '', text)
    text = re.sub(r'[^a-z A-Z]', ' ',text)
    text = re.sub(r'\b\w{1,2}\b', '', text)
    text = re.sub(r'[^\w\s]','',text)
    text = re.sub(r'^RT[\s]+', '', text)
    text = re.sub('\[.*?\]', '', text)
    text = re.sub('\n', '', text)
    text = re.sub('\w*\d\w*', '', text)
    text = re.sub(r'#', '', text)
    text = re.sub(r'[^\w\s]','',text)
    text = re.sub(r'@[A-Za-z0–9]+', '', text) 
    text = re.sub(r' +', ' ', text)
    return text<jupyter_output><empty_output><jupyter_text>Clean the tweet column by passing it through the regex function above and create a new column called clean_tweet for the cleaned tweets.<jupyter_code>data['clean_tweet'] = data.tweet.apply(clean)
data.head()<jupyter_output><empty_output><jupyter_text>Word CloudWord cloud of our cleaned dataset to visualize the most important words.Tags are usually single words, and the importance of each tag is shown with font size or color.<jupyter_code>customStopwords=list(STOPWORDS)


wordcloudimage=WordCloud(max_words=2400,
                         max_font_size=500,
                         font_step=2,
                         stopwords=customStopwords,
                         background_color='black',
                         width=1000,
                         height=720).generate(str(data['clean_tweet'].values))                          
 
plt.figure(figsize=(15,7))
plt.axis("off")
plt.imshow(wordcloudimage)
wordcloudimage
plt.show()<jupyter_output><empty_output><jupyter_text>From our word cloud above, we see our most dominant and important words aside South Africa relate to violence and that exactly is what we are tracking.  Removing Stop Words Stop words are generally filtered out before processing in natural language. words. These are actually the most common words in any language (like articles, prepositions, pronouns, conjunctions, etc) and does not add much information to the text. They are available in abundance in any human language. By removing these words, we remove the low-level information from our text in order to give more focus to the important information. We use the NLTK library and import the English stop words list.<jupyter_code># download stopwords
nltk.download('stopwords')
# Remove stopwords
stop_words = set(stopwords.words("english"))
data["clean_tweet"] = data["clean_tweet"].apply(lambda x : " ".join([w.lower() for w in x.split() if w not in stop_words and len(w) > 3]))
data["clean_tweet"].head()<jupyter_output><empty_output><jupyter_text>Tokenization Tokenization is the process of breaking apart raw text into individual pieces (tokens) for further analysis. This process is important because the meaning of the text can be interpreted through analysis of the words present in the text.<jupyter_code># Tokenize tweet
tweets = data["clean_tweet"].apply(lambda x : x.split())
tweets<jupyter_output><empty_output><jupyter_text>Lemmatization Lemmatization is a process where we convert words to its root word. The advantage of this is, we get to reduce the total number of unique words in the dictionary. As a result, the number of columns in the document-word matrix (created by TfidfVectorizer in the next step) will be denser with lesser columns.<jupyter_code>def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV']):
    """
    function to lemmatize text imput, keeping noun, adj, verb, adverb
    paramters
    ---------
    texts: list of text strings
    allowed_postags: list of tags to keep

    returns
    ------
    texts: lemmatized string tokens
    """
    texts_out = []
    for sent in texts:
        doc = nlp(" ".join(sent)) 
        texts_out.append(" ".join([token.lemma_ if token.lemma_ not in ['-PRON-'] else '' for token in doc if token.pos_ in allowed_postags]))
    return texts_out

# Initialize spacy 'en' model, keeping only tagger component (for efficiency)
nlp = spacy.load('en', disable=['parser', 'ner'])

# lemmatize keeping only Noun, Adj, Verb, Adverb
data_lemmatized = lemmatization(tweets, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV'])
data_lemmatized[:10]<jupyter_output><empty_output><jupyter_text>VectorizationThe LDA topic model algorithm requires a document word matrix as the main input. We used the scikit learn TfidfVectorizer together with n_grams. TfidfVectorizer helps us in dealing with most frequent words and weights the word counts by a measure of how often they appear in the documents.So, to create the doc-word matrix, you need to first initialise the TfidfVectorizer with the n_gram range and instead of a single word count, we can count the word pairs. In our case bigrams and trigrams so that we are no longer treating a single word, but two words and three words frequently occurring together in the document. Then we apply fit_transform to actually create the matrix. Since most cells contain zeros, the result will be in the form of a sparse matrix to save memory.<jupyter_code>vectorizer = TfidfVectorizer(ngram_range=(2,3))
data_vectorized = vectorizer.fit_transform(data_lemmatized)
data_vectorized<jupyter_output><empty_output><jupyter_text>LDA Model In natural language processing, the latent Dirichlet allocation (LDA) discovers topics in a collection of documents, and then automatically classify any individual document within the collection in terms of how "relevant" it is to each of the discovered topics. A topic is considered to be a set of terms (i.e. individual words or phrases) that, taken together, suggest a shared theme.[link text](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation)The most important tuning parameter for LDA models is n_components (number of topics). We set ours to 10.<jupyter_code># Build LDA Model
lda_model = LatentDirichletAllocation(n_components=10,          
                                      max_iter=10,               
                                      learning_method='online',   
                                      random_state=100,          
                                      batch_size=128,            
                                      evaluate_every = -1,      
                                      n_jobs = -1,              
                                     )
lda_output = lda_model.fit_transform(data_vectorized)<jupyter_output><empty_output><jupyter_text>Grid Search to find the Optimal model Grid search is a tuning technique that attempts to compute the optimum values of hyperparameters. We search for the most important tuning parameter n_components (number of topics) and the learning_decay (which controls the learning rate) as well.<jupyter_code># Define Search Param
search_params = {'n_components': [5, 10, 15, 20, 25, 30], 'learning_decay': [.5, .7, .9]}

# Init the Model
lda = LatentDirichletAllocation()

# Init Grid Search Class
model = GridSearchCV(lda, param_grid=search_params)

# Do the Grid Search
model.fit(data_vectorized)
# Best Model
best_lda_model = model.best_estimator_

# Model Parameters
print("Best Model's Params: ", model.best_params_)<jupyter_output>Best Model's Params:  {'learning_decay': 0.9, 'n_components': 5}<jupyter_text>Our Grid search returned 5 as the optimal number of topics with a learning decay of 0.5. Next, we check which topic has the highest contribution to that document and assign the topic to it.In the table below, we greened out all major topics in a document and assigned the most dominant topic in its own column.<jupyter_code># Create Document - Topic Matrix
lda_output = best_lda_model.transform(data_vectorized)

# column names
topicnames = ["Topic" + str(i) for i in range(best_lda_model.n_components)]

# index names
docnames = ["Tweet" + str(i) for i in range(len(data["clean_tweet"]))]

# Make the pandas dataframe
df_document_topic = pd.DataFrame(np.round(lda_output, 2), columns=topicnames, index=docnames)

# Get dominant topic for each document
dominant_topic = np.argmax(df_document_topic.values, axis=1)
df_document_topic['Dominant_topic'] = dominant_topic

# Styling
def color_green(val):
    color = 'green' if val > .1 else 'black'
    return 'color: {col}'.format(col=color)

def make_bold(val):
    weight = 700 if val > .1 else 400
    return 'font-weight: {weight}'.format(weight=weight)

# Apply Style
df_document_topics = df_document_topic.head(10).style.applymap(color_green).applymap(make_bold)
df_document_topics<jupyter_output><empty_output><jupyter_text>Topics distribution across Tweets This is to view the total number of tweets for each topic.<jupyter_code>df_topic_distribution = df_document_topic['Dominant_topic'].value_counts().reset_index(name="Num Tweets")
df_topic_distribution.columns = ['Topic Num', 'Num Tweets']
df_topic_distribution<jupyter_output><empty_output><jupyter_text>Visualize the LDA model with pyLDAvisThe pyLDAvis offers the best visualization to view the topics-keywords distribution.A good topic model will have non-overlapping, fairly big sized blobs for each topic.<jupyter_code>pyLDAvis.enable_notebook()
panel = pyLDAvis.sklearn.prepare(best_lda_model, data_vectorized, vectorizer, mds='tsne')
panel<jupyter_output><empty_output><jupyter_text>To view the visualization above in a seperate webpage<jupyter_code>p = pyLDAvis.sklearn.prepare(best_lda_model, data_vectorized, vectorizer, mds='tsne')
pyLDAvis.save_html(p, 'lda.html')<jupyter_output><empty_output><jupyter_text>Top 15 keywords for each topic<jupyter_code># Show top n keywords for each topic
def show_topics(vectorizer=vectorizer, lda_model=lda_model, n_words=20):
    """
    function to show table of top words in the topic model
    paramaters
    ----------
    vectorizer: vector obtained used vectorizer
    lda_model: topic model instance
    n_words: number of words to show in table

    returns:
    topic_kewords: list of words for each topic
    """
    keywords = np.array(vectorizer.get_feature_names())
    topic_keywords = []
    for topic_weights in lda_model.components_:
        top_keyword_locs = (-topic_weights).argsort()[:n_words]
        topic_keywords.append(keywords.take(top_keyword_locs))
    return topic_keywords

topic_keywords = show_topics(vectorizer=vectorizer, lda_model=best_lda_model, n_words=15)        

# Topic - Keywords Dataframe
df_topic_keywords = pd.DataFrame(topic_keywords)
df_topic_keywords.columns = ['Word '+str(i) for i in range(df_topic_keywords.shape[1])]
df_topic_keywords.index = ['Topic '+str(i) for i in range(df_topic_keywords.shape[0])]
df_topic_keywords<jupyter_output><empty_output><jupyter_text>Inference and Model Deployment For inference, both the model and the vectorizer are saved in pickle file format for deployment on an AWS EC2 using the functions below. The detailed pipeline and associated functions using Airflow for scheduling can be found [here](https://github.com/SitwalaM/nlp-topic-modelling/blob/main/scripts/nlp_dag.py)<jupyter_code>def get_inference(model, vectorizer, topics, text, threshold):
    """
    runs inference on text input
    paramaters
    ----------
    model: loaded model to use to transform the input
    vectorizer: instance of the vectorizer e.g TfidfVectorizer(ngram_range=(2, 3))
    topics: the list of topics in the model
    text: input string to be classified
    threshold: float of threshold to use to output a topic based on the score
    returns
    -------
    tuple => top score
    
    """
    v_text = vectorizer.transform([text])
    score = model.transform(v_text)

    labels = set()
    for i in range(len(score[0])):
        if score[0][i] > threshold:
            labels.add(topics[i])
    if not labels:
        return 'None', -1, set()

    return topics[np.argmax(score)]

def modelling():
    # function to run model in the pipeline
    # returns dataframe with predictions in column topic

    with open("lda_model.pk","rb") as f:
        lda_model = pickle.load(f)
    
    # vectorizer  
    vectorizer = pickle.load(open("vectorizer.pickle", 'rb'))

    # topics
    topics = list(np.arange(0,10))
    

    #cleaned_df = extract_clean()    
    cleaned_df = pd.read_csv("clean_"+str(dt.date.today()) + ".csv")
    cleaned_df["topic"] = cleaned_df.clean_tweet.apply(lambda x: get_inference(lda_model,vectorizer,topics,np.str_(x),0))
    cleaned_df.to_csv("predict_"+"clean_"+str(dt.date.today()) + ".csv")

    return cleaned_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09028188448649714, 'fraction_numerical': 0.006373611932452855, 'mean_word_length': 4.1768707482993195, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 75, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '328346', 'license': 'MIT', 'n_tokens_mistral': 4591, 'n_tokens_neox': 4430, 'n_words': 1551, 'path': 'Topic_Modelling_Final_TeamB.ipynb', 'repo_name': 'SitwalaM/nlp-topic-modelling', 'size': '15219'}