<jupyter_start><jupyter_text>Importing Libraries<jupyter_code>import pandas as pd
import numpy as np

# Convert text to vectors of numbers => document-term-matrix
from sklearn.feature_extraction.text import CountVectorizer

# for tokenizing
import nltk
nltk.download('punkt')
from nltk.tokenize import word_tokenize

# plots
import matplotlib.pyplot as plt

# for wordlcloud graphs
from wordcloud import WordCloud
from google.colab import drive
drive.mount('/content/gdrive')
path = 'gdrive/My Drive/GH_NLP_Course/Data/'<jupyter_output><empty_output><jupyter_text>Reading Dataset<jupyter_code>reviews = pd.read_csv(path + 'Input/2.input_data_prepped.csv')
reviews.head()
reviews.dtypes
# Convert text_prep to string to counteract issues later
reviews['text_prep'] = reviews['text_prep'].astype(str)<jupyter_output><empty_output><jupyter_text>Lexical Complexity- Word Frequency: number of words per corpus. Top Words- Word Length: average length of individual words in a text- Lexical Diversity: number of unique words used in a text- Lexical Density: the number of unique words divided by the total number of words (word repetition)  Bag of Words - Top 6000<jupyter_code>len(reviews)<jupyter_output><empty_output><jupyter_text>Matrix VS Array- matrices are strictly 2-dimensional - numpy arrays (ndarrays) are N-dimensional.DF VS Array- DF = 2-dimensional array with labeled axes (matrix with column names for columns, and index labels for rows)- A single column or row in a Pandas DataFrame is a Pandas series — a one-dimensional array with axis labelsSparse Matrix= Matrices that contain mostly zero values are called sparse, distinct from matrices where most of the values are non-zero, called dense.Compressed Sparse Row Matrix: a compressed matrix, to save space (contains one vector for index pointers, one vector for indexes, one vector with data, does not save 0s)Converting compressed sparse matrix to- dense matrix: counts_per_review.todense()- array: counts_per_review.toarray()  Create a document-term-matrix<jupyter_code># max_features = If not None, build a vocabulary that only consider the top 
# max_features ordered by term frequency across the corpus.

# construct the vocab on most frequent 6000 words
vec = CountVectorizer(max_features = 6000)

# creates a matrix with rows = documents, cols = words
counts_per_review = vec.fit_transform(reviews.text_prep)

# the 6000 words => will become the name of the cols
review_words = vec.get_feature_names()

dtm = pd.DataFrame(counts_per_review.toarray(), 
                   columns=review_words)
dtm.head()
dtm['pron']
# Delete pron column
del dtm['pron']
dtm.shape
reviews.text_prep.iloc[250]
dtm['third'].iloc[250]
# Count number of words in text prep
reviews['text_prep_tokens'] = reviews['text_prep'].apply(word_tokenize)
reviews['word_len_prep'] = reviews['text_prep_tokens'].apply(len)
reviews.head()<jupyter_output><empty_output><jupyter_text>Create a new column, text_prep_lim, that contains only the words in BoW Lambda Function: anonymous function (a function that is defined without a name => we don't use def)<jupyter_code>reviews['text_prep_lim_tokens'] = reviews['text_prep_tokens'].apply(lambda x: 
                                                   [word for word in x if word in review_words])
reviews.head()
reviews['word_len_prep_lim'] = reviews['text_prep_lim_tokens'].apply(len)
reviews.head()
# # Change output from list of strings to sentence
reviews['text_prep_lim'] = reviews['text_prep_lim_tokens'].apply(lambda x: ' '.join(x))
reviews.head()
reviews['text_prep_lim'].iloc[250]
print(reviews.text.iloc[0])
print(reviews.text_prep.iloc[0])
print(reviews.text_prep_lim.iloc[0])

# the word 'croed' & pronoun '-PRON-' are missing from text_prep_lim
reviews.columns
reviews = reviews[['rest_id', 'text', 'rating', 'char_count', 'positive', 
                   'text_prep', 'text_prep_tokens', 'word_len_prep', 
                   'text_prep_lim', 'text_prep_lim_tokens','word_len_prep_lim']]
reviews.head(2)<jupyter_output><empty_output><jupyter_text>Save BOW DTM to csv<jupyter_code>dtm.to_csv(path + 'Output/1.dtm_1_bow.csv', index=False)
reviews.to_csv(path + 'Input/3.input_data_prepped_bow.csv', 
               index = False)<jupyter_output><empty_output><jupyter_text>Word Cloud<jupyter_code>dtm.head()<jupyter_output><empty_output><jupyter_text>- Axis 0 will act on all the ROWS in each COLUMN- Axis 1 will act on all the COLUMNS in each ROW<jupyter_code>word_cloud_freq = dtm.sum(axis=0)
word_cloud_freq
wordcloud = WordCloud(max_words=100, 
                      background_color="white",
                      font_path = path + 'Input/SignPainter.ttc',
                      colormap="Blues")
wordcloud.generate_from_frequencies(word_cloud_freq)
plt.imshow(wordcloud)
# Turn off axis lines and labels
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>Word Cloud Positive - Top 20<jupyter_code>dtm_pos = dtm[reviews['positive']==1].copy()
dtm_pos.shape
word_cloud_freq_pos = dtm_pos.sum(axis=0)
wordcloud = WordCloud(max_words=20,
                      background_color="white",
                      font_path = path + 'Input/SignPainter.ttc',
                      colormap="Greens")

wordcloud.generate_from_frequencies(word_cloud_freq_pos)
plt.imshow(wordcloud)
plt.axis("off")
plt.show()
wordcloud.words_
# 8 of the 20 words are positive words
# always, amazing, delicious, friendly, good, great, like, love<jupyter_output><empty_output><jupyter_text>Word Cloud Negative - Top 20<jupyter_code>dtm_neg = dtm[reviews['positive']==0].copy()
dtm_neg.shape
word_cloud_freq_neg = dtm_neg.sum(axis=0)
word_cloud_freq_neg
wordcloud = WordCloud(max_words=20,
                      background_color="white",
                      font_path = path + 'Input/SignPainter.ttc',
                      colormap="Reds")
wordcloud.generate_from_frequencies(word_cloud_freq_neg)

plt.imshow(wordcloud)
plt.axis("off")
plt.show()
wordcloud.words_
# 1 of the 20 words is negative, 2 are positive
# wait
# good, like<jupyter_output><empty_output><jupyter_text>Word Frequency<jupyter_code># Most Frequent Words
word_freq = pd.DataFrame(dtm.sum(axis=0)).reset_index()
word_freq.head()
word_freq.columns = ['Word', 'Count']
word_freq.head()
word_freq = word_freq.sort_values(by = 'Count', 
                                  ascending = False)
word_freq.head()
# Plot Top 20 Words
top20 = word_freq[0:20].copy().reset_index(drop=True)
top20.head()
plt.barh(y = top20['Word'], 
         width = top20['Count'], 
         height = 0.5,
         align='center')

plt.title('Top 20 Tokens')
plt.xlabel('Word')
plt.ylabel('Frequency')

plt.show();<jupyter_output><empty_output><jupyter_text>Word Length<jupyter_code># Word Length
word_freq['word_len'] = word_freq['Word'].apply(len)

word_freq.head()
word_freq.describe()
# Average word length = 6 chars<jupyter_output><empty_output><jupyter_text>Lexical Diversity: How many unique words in each review?<jupyter_code>dtm.head()
# For each doc, count number of unique words
# <=> count how many columns have non-zero values
lexical = pd.DataFrame(data=np.count_nonzero(dtm, axis=1),
                       columns = ['Unique Words'])
lexical.head()
lexical['Total Words'] = dtm.sum(axis=1)
lexical.head()
reviews['text_prep_lim'].iloc[0]
# dinner & pm appear twice;
lexical.describe()
# On avg, 34 unique words per review, 40 total words per review
lexical['positive']=reviews['positive'].copy()
lexical.head()
lexical.groupby('positive').mean()
# On avg, 42 unique words per negative review, 30 unique words per positive review
# On avg, 53 total words per negative review, 35 unique words per positive review<jupyter_output><empty_output><jupyter_text>Lexical Density: Unique Words / Total Words<jupyter_code># As lexical density increases, repetition decreases. 
lexical.head()
lexical['Density'] = lexical['Unique Words']/lexical['Total Words']
lexical.head()
lexical['Density'].hist()
lexical.groupby('positive').mean()
# Positive Reviews have larger density, which suggests lower repetition.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10955508737142149, 'fraction_numerical': 0.011773453959598463, 'mean_word_length': 4.977777777777778, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '45650', 'license': 'MIT', 'n_tokens_mistral': 2775, 'n_tokens_neox': 2579, 'n_words': 667, 'path': 'SA2_Lexical_Complexity.ipynb', 'repo_name': 'DanielaManate/SentimentAnalysis-TopicModeling', 'size': '8069'}