<jupyter_start><jupyter_text>Text Analysis for Institutional ResearchBCIRP 2020 Virtual WorkshopThank you for your interest in our NLP workshop.Whether it's student satisfaction, campus climate, or alumni surveys, analyzing open-ended comments from the results can be a daunting task. We hope these workshop resources will help make the process easier and unlock new insights for you. *   The workbook accompanies the presentation on [Google Slides](https://bit.ly/bcirp_nlp_prez)*   The recorded presentation can be accessed via [YouTube](https://www.youtube.com/watch?v=2fzFQWX5DzE)If  you're seeking more customized solutions for your open-ended responses feel free to reach out at [<EMAIL>](<EMAIL>)Sincerely yours,<NAME>Founder and CEO, Kai Analytics and Survey Research Inc.Copyright (c) 2020 Kai Analytics and Survey Research Inc.  1.Getting StartedThe document you are reading is not a static web page, but an interactive environment called **Jupyter Notebook** (*powered by GoLabs*) that lets you write and execute code.For example, below is **code cell** with a short Python script that prints a statement. To run the code you can:*   mouse over and click the ![Play Button](https://img.icons8.com/ios-glyphs/30/000000/play-button-circled.png); or*   press **shift** + **enter**<jupyter_code>print("Hello Everyone!")<jupyter_output><empty_output><jupyter_text>Woohoo! You ran a Python code! Congratulations. ![Thumbs up](https://img.icons8.com/dusk/64/000000/thumb-up.png)  2.Loading our DatasetToday we will work with a subset of Coursera course feedback dataset from [kaggle](https://www.kaggle.com/septa97/100k-courseras-course-reviews-dataset/)Kaggle is the world's largest data science community with powerful tools and resources to help you achieve your data science goals.This dataset contains only feedback related to courses that teach English writing. In the next **code cell**, we will:*   Load the first Python library in today's workshop, PANDAS. It will help us easily read, and manipulate our dateset*   Read our dataset and look at some of its attributes<jupyter_code># Let's import PANDAS (Python Data Analysis Library) library.
# This one is built into CoLabs, so we don't have to install it.
 
import pandas as pd
 
# Set the url to where the dataset is located.
# In this case, it's one of my public gitHub repositories.
url = 'https://raw.githubusercontent.com/kevin-kaianalytics/bcirpNlpWorkshop2020/master/writing_course_review.csv'
 
# Read the CSV file.
df = pd.read_csv(url) # df is shorthand for dataframe
 
# If you want to upload from your desktop
#from google.colab import files
#df = files.upload()
 
# Let's check and see we've loaded our data properly.
# We do this by checking its first 10 rows.
pd.options.display.max_columns = None
display(df.head(10))<jupyter_output><empty_output><jupyter_text>We can also see how many rows is in our data with the **count()** command. How many rows should we expect to see?<jupyter_code>print("Number of rows in the 'Review' column: ", df['Review'].count())<jupyter_output><empty_output><jupyter_text>2.1 Our First Word CloudOkay let's see what a word cloud of the responses look like. *(Don't worry about the complexity of the code here, we're just generating a word cloud)*<jupyter_code>from wordcloud import WordCloud
import matplotlib.pyplot as plt
 
# quickly make all lower case, replace non alpha-numeric characters
df_wordcloud = df['Review'].astype(str).str.lower().str.replace('[^\w\s]','')
 
text = df_wordcloud.values 
wordcloud = WordCloud().generate(str(text))
plt.figure(figsize=(15,10))
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>3.Install and Import the NLTK Package* "The Natural Language Toolkit, or more commonly NLTK, is a suite of libraries and programs for symbolic and statistical natural language processing for English written in the Python programming language."* It is great for for understanding the key concepts we are discussing today.* In the next **code cell**, you we will be installing and importing the NLTK package.*Note: This may take a moment*<jupyter_code>! pip install NLTK==3.4 #Install the Natural Language Toolkit
import nltk<jupyter_output><empty_output><jupyter_text>4.Text Pre-ProcessingIn this section, we use some of NLTK text pre-processing functions to help us improve the results of our analysis. As introduced in our presentation, these steps will include:*   Tokenizing text into words*   Removing stop words*   Lemmatization*   Parts of Speech Tagging  4.1 Tokenize the comments From the NLTK Tokenize library, we will import the word_tokenize module.We will also need to download the pre-trained [Punkt tokenizer](https://www.nltk.org/api/nltk.tokenize.html) for English. This tokenizer divides a text into a list of words, by using an unsupervised algorithm to build a model for abbreviation words, collocations, and words that start sentences.<jupyter_code># Import word_tokenize function for the nltk.tokenize library
from nltk.tokenize import word_tokenize
 
# Download the Punkt Sentence Tokenizer from the NLTK Tokenize library
nltk.download('punkt')
 
# Some standard text pre-processing here
df['Review'] = df['Review'].astype(str).str.lower().str.replace('[^a-zA-Z#]',' ')
 
# Apply Tokenizer and create a new column named "ReviewTokenized"
df['tokenizeReview'] = df['Review'].apply(word_tokenize)
 
# Output the first 10 results
print()
print(df['tokenizeReview'].head().to_string())<jupyter_output><empty_output><jupyter_text>4.2 Remove StopwordsRemove frequent words that are redundant to our analysis. Let's see what stop words come out of the box.<jupyter_code># Load the stopwords module from the NLTK Tokenize library
from nltk.corpus import stopwords

# Download the default list of stopwords
nltk.download('stopwords')

# We will load the English stop words into the stop variable and see it's output
stop = stopwords.words('english')

# Output all the default stopwords
print(stop)<jupyter_output><empty_output><jupyter_text>Now let's remove the stop words from our dataset.* **Head's up:** *This next **code cell** contains some advanced concepts: Lamdas and List Comprehensions. Basically, we are applying a one-time function (lambda) that loops through every token in a row and every row, keeping only words not in the stop word list*<jupyter_code>df['removeStop'] = df['tokenizeReview'].apply(lambda x: [item for item in x if item not in stop])

# Output the result and compare it with the last steps
row_number = 6
print(df[['Review']].iloc[row_number].to_string())
print('')
print(df[['tokenizeReview']].iloc[row_number].to_string())
print('')
print(df[['removeStop']].iloc[row_number].to_string())<jupyter_output><empty_output><jupyter_text>4.3 LemmatizationWe can lemmatize using WordNet's built-in morphy function. It returns the input word unchanged if it cannot be found in WordNet. [WordNet -> Stem](https://www.nltk.org/_modules/nltk/stem/wordnet.html)<jupyter_code># Import the WordNewLemmatizer module from NLTK.stem
from nltk.stem import WordNetLemmatizer

# Download the wordnet model
nltk.download('wordnet')

# Name the WordNetLemmatizer function simply as lemmitizer
lemmatizer = nltk.WordNetLemmatizer()<jupyter_output><empty_output><jupyter_text>Now we test it with some English words.<jupyter_code>print('dogs: ', lemmatizer.lemmatize('dogs'))
print('kittens: ', lemmatizer.lemmatize('kittens'))
print('students: ', lemmatizer.lemmatize('students'))
print('profs: ', lemmatizer.lemmatize('profs'))
print('professors: ', lemmatizer.lemmatize('professors'))
print('')
print('**teaching: ', lemmatizer.lemmatize('teaching'))<jupyter_output><empty_output><jupyter_text>By default, WordNetLemmatizer() treats all words as nouns. So it needs a **part of speech** (POS) tag to help it out.* *Note that the 'v' in the next **code cell** stands for verb*.<jupyter_code>print('Now with a verb tag')
print('---------')
print('teaching: ', lemmatizer.lemmatize('teaching', 'v'))
print('teaches: ', lemmatizer.lemmatize('teaches', 'v'))
print('taught: ', lemmatizer.lemmatize('taught', 'v'))<jupyter_output><empty_output><jupyter_text>As you can see, with the verb tag, the WordNetLemmatizer() was correcntly lemmatized the words to "teach". So how can we assign POS tags more efficiently? We can use the [NLTK Taggers](https://www.nltk.org/_modules/nltk/tag.html) library! This libray contains functions for part-of-speech tagging, or simply "tagging".  4.4 Parts of Speech TaggingIn the next section we will: 1.   Import the pos_tag module2.   Take a list of tokens3.   Assign a tag to each one4.   Return the resulting list of tagged tokens. The taggers are assigned automatically based on a pre-trained corpus that checks what is the frequent tag for a word.<jupyter_code># Import the pos_tag module from NLTK
from nltk import pos_tag

# Download the pre-trained tagging model
nltk.download('averaged_perceptron_tagger')

# Apply the pos_tag function to every row
df['tokenizePOS'] = df['removeStop'].apply(pos_tag)

print()
print(df[['tokenizePOS']].iloc[6].to_string())<jupyter_output><empty_output><jupyter_text>Now let's try Lemmitizing again with the additional POS tags. To begin, we need to define a custom function that converts the tagsets to nouns (n), verbs (v) and adjectives (a). These latter three POS are what is supported in NLTK. Actually, if you think about it, prepositions don't have much left in them to truncate! *Sorry, but you won't find any 'But' jokes in this workshop!*<jupyter_code>def lemmatize_all(sentence):
    for word, tag in pos_tag(word_tokenize(sentence)):
        if tag.startswith("NN"):
            return lemmatizer.lemmatize(word, pos='n')
        elif tag.startswith('VB'):
            return lemmatizer.lemmatize(word, pos='v')
        elif tag.startswith('JJ'):
            return lemmatizer.lemmatize(word, pos='a')
        else:
            return word<jupyter_output><empty_output><jupyter_text>With the function defined, we can now apply the tagged lemmatization to every row in our dataset. *This might just take a second or two*<jupyter_code>df['lemmatizeToken'] = df['tokenizeReview'].apply(lambda x: [lemmatize_all(y) for y in x]).apply(lambda x: [item for item in x if item not in stop])<jupyter_output><empty_output><jupyter_text>Great! Now we're ready to look at the results. We'll run it just for the first row.<jupyter_code>row_number = 6

print(df[['Review']].iloc[row_number].to_string())
print('')
print(df[['tokenizeReview']].iloc[row_number].to_string())
print('')
print(df[['removeStop']].iloc[row_number].to_string())
print('')
print(df[['lemmatizeToken']].iloc[row_number].to_string())<jupyter_output><empty_output><jupyter_text>4.5 Reflection* Notice how we've simplified our sentences yet retained much of its semantic?* By the way, the scripts ran in seconds. It's that fast!---  *A note on the "greedy averaged perceptron tagger." Greedy algorithm is an approach to solving an optimization problem. In the case for this tagger, it picks the most obvious POS representation of a word based on a pre-trained corpus ([Penn Treebank tagset](https://catalog.ldc.upenn.edu/docs/LDC95T7/cl93.html)). There are numerous benefits to this approach but it can have trouble correctly identifying English words with multiple POS representations (AKA **Hynonyms**). For example: ![Via https://www.bizarro.com/blog/2020/5/24/sit-on-it](https://images.squarespace-cdn.com/content/v1/5c08178c1137a64a2bef43f2/1590357331811-PSRDUR31QXSJOFT59OX5/ke17ZwdGBToddI8pDm48kEbXm70uG42CVd5Zg5r_V8ZZw-zPPgdn4jUwVcJE1ZvWEtT5uBSRWt4vQZAgTJucoTqqXjS3CfNDSuuf31e0tVHr3VZ8lJijBynlOcIcX9aZd0hHYtrh_wB3Kimrt6XWSgoxDhVsbCA7GIk3IE-1TtU/Bz+C+200522+P.jpg?format=500w)Average means that it that it took an average weight over several iterations during training. *Said plainly, it's quite fast, but it's not perfect due to words with more rare POS representations. So use with caution!*---  **[Optional]** Export your results to a CSV fileIf you want, you can download a copy of the results up to this point by running the next **code cell**.<jupyter_code>from google.colab import files

df.to_csv('nlpResults.csv')
files.download('nlpResults.csv')<jupyter_output><empty_output><jupyter_text>5.Text AnalysisNow that we've applied some common text pre-processing methods (*By the way, congratulations on getting this far*), we can analyze some text! In the next two sections we will take a look at the N-Grams methods nad see what that looks like in terms of visualization.---  5.1 N-Grams"*An n-gram is a contiguous sequence of n items from a given sample of text or speech. The items can be phonemes, syllables, letters, words or base pairs according to the application. The n-grams typically are collected from a text or speech.*" -- [Wikipedia](https://en.wikipedia.org/wiki/N-gram).* We're usually dealing with words, but sometimes we might get to the character level. For example "A," "T," "G," "C," the building blocks of DNA.* N-grams can be forward or in reverse. For example, the user may have searched "blue sky", but was he/she, in fact, interested in the colour, "sky blue"?* Sometimes you want to skip-gram, especially when it comes to predicting the next word (e.g. autofill). For example, fill in the blank to this sentence,"*the cat ___ on the mat*"... Is it "sat", "played", or "rolled"?--- Let's start by importing some libraries to help us with our analysis.<jupyter_code>from collections import Counter
# Counter will help us keep track of elements (our n-grams) and their count.

from nltk.util import ngrams
# This is for a generalized n-gram. See also bigrams.
# Specify number of grams: bigrams(2), trigrams(3), etc.
number_of_grams = 2

ngrams_results = []
for x in (df['lemmatizeToken']):
    ngrams_results.append(list(ngrams(x,number_of_grams)))

ngrams_results = [y for x in ngrams_results for y in x]

# Count the number of matching ngrams
ngram_count = Counter(ngrams_results)

# Turn our ngram_count into a dataframe
dataset_ngram = pd.DataFrame.from_dict(ngram_count, orient='index').reset_index()
dataset_ngram.columns = ["ngrams", "count"]

# Sort and keep the largest 40 ngrams
dataset_ngram = dataset_ngram.sort_values(by=['count'], ascending=False).head(40)

# Print the results
print(dataset_ngram.head().to_string())
print()<jupyter_output><empty_output><jupyter_text>5.2 Vizualizing a Network Graph"In graph theory, a graph is a structure amounting to a set of objects in which some pairs of the objects are in some sense "related". The objects correspond to mathematical abstractions called vertices and each of the related pairs of vertices is called an edge." -- [Wikipedia](https://en.wikipedia.org/wiki/Graph_(discrete_mathematics))<jupyter_code>d = dataset_ngram.set_index('ngrams').T.to_dict('records')
import networkx as nx
from matplotlib import cm
import matplotlib.pyplot as plt
from google.colab import files # this will help us save our plot as an image
#from matplotlib.pyplot import figure

# Create network plot 
G = nx.Graph()

# Create connections between nodes
for k, v in d[0].items():
    G.add_edge(k[0], k[1], weight=(v * 5))
#    G.add_nodes_from(n_list)


fig, ax = plt.subplots(figsize=(10, 8), dpi = 90)

pos = nx.spring_layout(G, k=2)

# Plot networks
nx.draw_networkx(G, pos,
                 font_size=14,
                 width=3,
                 edge_color='grey',
                 edge_cmap=plt.cm.Blues,
                 node_color='#4CB1D7',
                 with_labels = False,
                 ax=ax
                 )

# Create offset labels to improve readability
for key, value in pos.items():
    x, y = value[0], value[1]
    ax.text(x, y,
            s=key,
            bbox=dict(facecolor='#5DC5C1', alpha=0.25),
            horizontalalignment='center', fontsize=12)

# If you want to download the graph, just uncomment the next two lines
#plt.savefig("network_graph_coursera.png")
#files.download("network_graph_coursera.png")    
plt.show()<jupyter_output><empty_output><jupyter_text>5.3 And Finally, A Better Word Cloud<jupyter_code>import wordcloud as w

word_dict = dict(zip(dataset_ngram["ngrams"].astype(str), dataset_ngram["count"]))
# word cloud needs a dictionary format if we're generating from frequencies

wordcloud = w.WordCloud(collocations=False, 
                        max_words=500, 
                        width=1000,
                        height=500).generate_from_frequencies(word_dict)

# Set plot size and output the graph
plt.figure(figsize=(15,10))
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()

# If you want to download the word cloud
#plt.savefig("ngram_coursera_wordcloud.png")
#files.download("ngram_coursera_wordcloud.png")    
# plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10119649979165427, 'fraction_numerical': 0.012560271444728852, 'mean_word_length': 5.071557643657391, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 81, 'https://': 14, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '357033', 'license': 'MIT', 'n_tokens_mistral': 5408, 'n_tokens_neox': 5105, 'n_words': 1832, 'path': 'bcirp2020_workshop_nlp_higher_ed.ipynb', 'repo_name': 'kevin-kaianalytics/coursera_course_reviews', 'size': '16832'}