<jupyter_start><jupyter_text>This note book will clean data and create bag of word and tf-idf sparse matrix and save them with help of Pickle. They will be used again when we build LDA and LSA model.  Google Colab Setup<jupyter_code>from google.colab import drive
# This will prompt for authorization.
# authorization code: 4/OwErfUj6QceGXhIGx_RWv0MKclb9rilw8UsJnZqFbSez-QS8zQ399JU
drive.mount('/content/drive')

!pip install PyDrive

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)


#import accessory_functions.py from Colab
#https://drive.google.com/open?id=1S7URZIBq4zMh5QWv0qXPHv4ixhgHWN_y

my_module = drive.CreateFile({'id':'1S7URZIBq4zMh5QWv0qXPHv4ixhgHWN_y'})
my_module.GetContentFile('accessory_functions.py')<jupyter_output><empty_output><jupyter_text>Import libraries<jupyter_code>import numpy as np
import string
import pandas as pd

import nltk
from nltk.stem import WordNetLemmatizer, SnowballStemmer
from nltk.stem.porter import *
nltk.download('wordnet')
nltk.download('stopwords')
from nltk.corpus import stopwords, wordnet
from nltk.tokenize import word_tokenize
nltk.download('averaged_perceptron_tagger')

import gensim
from gensim.utils import simple_preprocess
import gensim.corpora as corpora


import pickle
from wordcloud import WordCloud
# Plotting tools

import matplotlib.pyplot as plt
%matplotlib inline
# Make all my plots 538 Style
plt.style.use('fivethirtyeight')
import warnings
warnings.simplefilter('ignore')

from accessory_functions import preprocess_series_text
from accessory_functions import write_pickle_file<jupyter_output>[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!<jupyter_text>Data Path<jupyter_code>datapath = '/content/drive/My Drive/data/'

data = pd.read_csv(datapath + 'breakingnews.csv')

data.head(2)
data_text = data[['headline_text']]
data_text['index'] = data_text.index
documents = data_text<jupyter_output><empty_output><jupyter_text>cleaning data and save it<jupyter_code>processed_docs = preprocess_series_text(documents['headline_text'])<jupyter_output><empty_output><jupyter_text><jupyter_code>plt.figure(figsize=(15, 15))
wordcloud = WordCloud(
        background_color='white'
#        max_words=200,
        ,max_font_size=20
        , scale=2) # chosen at random by flipping a coin; it was heads)

wordcloud.generate(str(processed_docs))

plt.imshow(wordcloud, interpolation="bilinear")
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>From the visualization, we can guess the topics should be about <NAME>, Uber, Bank, Tesla, minister. We will check it out in another notebook. Now we save the processed_docs oject for latter use<jupyter_code>processed_docs = processed_docs.values.tolist()

write_pickle_file(processed_docs, datapath + 'processed_docs.pkl')<jupyter_output><empty_output><jupyter_text>Create gensim dictionary  from processed_docs<jupyter_code># filter out the less common words
# Keep tokens which are contained in at least 15 documents
# Keep tokens which are contained in no more than 50% documents
# Keep only the first 10000 most frequent tokens
#dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=10000)
dictionary = corpora.Dictionary(processed_docs)

print("1st word in dictionary: {}".format(dictionary[0]))

print("1OOth word in dictionary: {}".format(dictionary[100]))<jupyter_output><empty_output><jupyter_text>Convert processed_docs into bad of word (bow) using dictionary  It is a list of word id and theirs frequencies<jupyter_code>bow = [dictionary.doc2bow(doc) for doc in processed_docs]<jupyter_output><empty_output><jupyter_text>View the first document in bow and using dictionary to see which words and their frequency<jupyter_code>doc = bow[1]

[(dictionary[id], freq) for id, freq in doc]<jupyter_output><empty_output><jupyter_text>Save bag-of_word<jupyter_code>write_pickle_file(bow, datapath+'bow.pkl')

"""
for id, freq in doc:
  print("{} : {}".format(dictionary[id], freq))
"""<jupyter_output><empty_output><jupyter_text>We build Tf-Idf, a simple transformation which takes documents represented as bag-of-words counts and applies a weighting which discounts common terms or equivalently, promotes rare terms  Then, save tfidf for using later<jupyter_code>tfidf_model = gensim.models.TfidfModel(bow)
tfidf = tfidf_model[bow]

write_pickle_file(tfidf, datapath + "tfidf.pkl")
doc = tfidf[1]
[(dictionary[id], freq) for id, freq in doc]<jupyter_output><empty_output><jupyter_text>Finally, save gensim dictionary object<jupyter_code>write_pickle_file(dictionary, datapath + 'dictionary.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09204940177537631, 'fraction_numerical': 0.012157468159011964, 'mean_word_length': 7.0984375, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '488526', 'license': 'MIT', 'n_tokens_mistral': 1847, 'n_tokens_neox': 1750, 'n_words': 434, 'path': 'model_preparation.ipynb', 'repo_name': 'diem-ai/topic-modeling', 'size': '5188'}