<jupyter_start><jupyter_text>World Bank Text Analytics Tutorial <NAME> Senior Data Scientist, Pew Research Center I adapted this notebook from a workshop I did in 2018 with an old colleague of mine, <NAME> (now at IBM), for the New York chapter of the American Association for Public Opinion Research. The original repository can be found here: https://github.com/jugovich/NYAAPOR-Text-Analysis-2018   Loading in the data We'll use a sample from the Kaggle Amazon Fine Food Reviews dataset. The full dataset can be found here: https://www.kaggle.com/snap/amazon-fine-food-reviews<jupyter_code>import pandas as pd
sample = pd.read_csv("https://raw.githubusercontent.com/patrickvankessel/WorldBank-Text-Analysis-2019/master/kaggle_sample.csv")
print(len(sample))<jupyter_output>10000<jupyter_text>Let's see what's in here.<jupyter_code>sample.head()<jupyter_output><empty_output><jupyter_text>Examine the data Run the cell below a few times, let's take a look at our text and see what it looks like.  Always take a look at your raw data.<jupyter_code>sample.sample(10)['Text'].values<jupyter_output><empty_output><jupyter_text>Preprocess the text (clean it up!) I don't know about you, but I noticed some junk in our data - HTML and URLs.  Let's clear that out first.  We'll also take this opportunity to lemmatize the words - to do that, we'll install NLTK's WordNet library.<jupyter_code>import nltk
nltk.download('wordnet')
import re
from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

def clean_text(text):
    # First we'll use regular expressions to strip out links and HTML tags
    text = re.sub(r'http[a-zA-Z0-9\&\?\=\?\/\:\.]+\b', ' ', text)
    text = re.sub(r'\<[^\<\>]+\>', ' ', text)
    # Next, let's clear out all punctuation and replace it with whitespace
    text = re.sub(r'\W+', ' ', text) 
    # This isn't going to be perfect - ideally we expand contractions first
    # But it'll work well enough for now
    
    # Next, let's split on whitespace and then lemmatize each token
    tokens = text.split()
    tokens = [lemmatizer.lemmatize(x) for x in tokens]
    text = " ".join(tokens)
  
    return text

sample['Text'] = sample['Text'].map(clean_text)<jupyter_output><empty_output><jupyter_text>TF-IDF Vectorization (Feature Extraction) Just to be safe, let's add some additional words to a standard list of English stop words.<jupyter_code>from sklearn.feature_extraction import stop_words as sklearn_stop_words
stop_words = set(sklearn_stop_words.ENGLISH_STOP_WORDS).union(set([
    "www", "http", "https", "br", "amazon", "href", "wa", "ha"
]))<jupyter_output><empty_output><jupyter_text>Okay, now let's tokenize our text and turn it into numbers<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer

tfidf_vectorizer = TfidfVectorizer(
    max_df=0.9, 
    min_df=5, 
    ngram_range=(1, 1), 
    stop_words=stop_words, 
    max_features=2500
)
tfidf = tfidf_vectorizer.fit_transform(sample['Text'])
ngrams = tfidf_vectorizer.get_feature_names()
tfidf<jupyter_output><empty_output><jupyter_text>Because words are really big, by default we work with sparse matrices.  We can expand the sparse matrix with `.todense()` and compute sums like a normal dataframe.  Let's check out the top 20 words.<jupyter_code>ngram_df = pd.DataFrame(tfidf.todense(), columns=ngrams) 
ngram_df.sum().sort_values(ascending=False)[:20]<jupyter_output><empty_output><jupyter_text>Classification Let's make an outcome variable.  How about we try to predict 5-star reviews, and then maybe helpfulness?<jupyter_code>sample['good_score'] = sample['Score'].map(lambda x: 1 if x == 5 else 0)
sample['was_helpful'] = ((sample['HelpfulnessNumerator'] / sample['HelpfulnessDenominator']).fillna(0.0) > .80).astype(int)
column_to_predict = 'good_score'
from sklearn.model_selection import StratifiedKFold
from sklearn import svm
from sklearn import metrics

results = []
kfolds = StratifiedKFold(n_splits=5)<jupyter_output><empty_output><jupyter_text>We just created an object that'll split the data into fifths, and then iterate over it five times, holding out one-fifth each time for testing.  Let's do that now.  Each "fold" contains an index for training rows, and one for testing rows.  For each fold, we'll train a basic linear Support Vector Machine, and evaluate its performance.<jupyter_code>for i, fold in enumerate(kfolds.split(tfidf, sample[column_to_predict])):
    
    train, test = fold 
    print("Running new fold, {} training cases, {} testing cases".format(len(train), len(test)))
    
    clf = svm.LinearSVC(
        max_iter=1000,
        penalty='l2',
        class_weight='balanced',
        loss='squared_hinge'
    )
    # We picked some decent starting parameters, but encourage you to try out different ones
    # http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html 
    # If you're ambitious - check out the Scikit-Learn documentation and test out different models
    # http://scikit-learn.org/stable/supervised_learning.html
    
    training_text = tfidf[train]
    training_outcomes = sample[column_to_predict].loc[train]
    clf.fit(training_text, training_outcomes) # Train the classifier on the training data
    
    test_text = tfidf[test]
    test_outcomes = sample[column_to_predict].loc[test]
    predictions = clf.predict(test_text) # Get predictions for the test data
    
    precision, recall, fscore, support = metrics.precision_recall_fscore_support(
        test_outcomes, # Compare the predictions against the true outcomes
        predictions
    )
    
    results.append({
        "fold": i,
        "outcome": 0,
        "precision": precision[0],
        "recall": recall[0],
        "fscore": fscore[0],
        "support": support[0]
    })
    
    results.append({
        "fold": i,
        "outcome": 1,
        "precision": precision[1],
        "recall": recall[1],
        "fscore": fscore[1],
        "support": support[1]
    })
    
results = pd.DataFrame(results)<jupyter_output>Running new fold, 8000 training cases, 2000 testing cases
Running new fold, 8000 training cases, 2000 testing cases
Running new fold, 8000 training cases, 2000 testing cases
Running new fold, 8000 training cases, 2000 testing cases
Running new fold, 8000 training cases, 2000 testing cases<jupyter_text>How'd we do?<jupyter_code>print(results.groupby("outcome").mean()[['precision', 'recall']])
print(results.groupby("outcome").std()[['precision', 'recall']])<jupyter_output>precision    recall
outcome                     
0         0.639631  0.698769
1         0.816753  0.773207
         precision    recall
outcome                     
0         0.009594  0.014841
1         0.007677  0.007380<jupyter_text>Now we know that our model is pretty stable and reasonably performant, we can fit and transform the full dataset.<jupyter_code>clf.fit(tfidf, sample[column_to_predict])  
print(metrics.classification_report(sample[column_to_predict].loc[test], predictions))
print(metrics.confusion_matrix(sample[column_to_predict].loc[test], predictions))<jupyter_output>precision    recall  f1-score   support

           0       0.63      0.71      0.67       731
           1       0.82      0.76      0.79      1269

    accuracy                           0.74      2000
   macro avg       0.73      0.74      0.73      2000
weighted avg       0.75      0.74      0.75      2000

[[520 211]
 [299 970]]<jupyter_text>And now we can see what the most predictive features are.<jupyter_code>import numpy as np

ngram_coefs = sorted(zip(ngrams, clf.coef_[0]), key=lambda x: x[1], reverse=True)
ngram_coefs[:10]<jupyter_output><empty_output><jupyter_text>What happens if you change the outcome column to "was_helpful" and re-run it again?  Can you think of ways to improve this?  Add stopwords?  Bigrams?  Topic Modeling<jupyter_code>from sklearn.decomposition import NMF, LatentDirichletAllocation
def print_top_words(model, feature_names, n_top_words):
    for topic_idx, topic in enumerate(model.components_):
        print("Topic #{}: {}".format(
            topic_idx,
            ", ".join([feature_names[i] for i in topic.argsort()[:-n_top_words - 1:-1]])
        ))<jupyter_output><empty_output><jupyter_text>Let's find some topics.  We'll check out non-negative matrix factorization (NMF) first.<jupyter_code>nmf = NMF(n_components=10, random_state=42, alpha=.1, l1_ratio=.5).fit(tfidf)
# Try out different numbers of topics (change n_components)
# Documentation: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html
print("\nTopics in NMF model:")
print_top_words(nmf, ngrams, 10)<jupyter_output>Topics in NMF model:
Topic #0: like, taste, flavor, just, really, don, sugar, water, ve, tried
Topic #1: coffee, cup, strong, roast, bold, flavor, blend, keurig, dark, like
Topic #2: tea, green, bag, drink, cup, iced, stash, black, earl, taste
Topic #3: dog, treat, love, food, chew, small, bone, teeth, size, training
Topic #4: cat, food, eat, dry, wellness, canned, chicken, ingredient, love, vet
Topic #5: product, store, price, order, buy, local, time, grocery, shipping, box
Topic #6: great, love, snack, price, deal, healthy, taste, recommend, highly, tasting
Topic #7: chocolate, bar, dark, peanut, candy, snack, nut, protein, butter, sweet
Topic #8: chip, bag, salt, potato, kettle, vinegar, snack, salty, flavor, fat
Topic #9: good, really, price, pretty, taste, quality, tasting, fresh, quite, enjoy<jupyter_text>LDA is an other popular topic modeling technique<jupyter_code>lda = LatentDirichletAllocation(n_components=10, random_state=42).fit(tfidf)
# Documentation: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html
# doc_topic_prior (alpha) - lower alpha means documents will be composed of fewer topics (higher means a more uniform distriution across all topics)
# topic_word_prior (beta) - lower beta means topis will be composed of fewer words (higher means a more uniform distribution across all words)
print("\nTopics in LDA model:")
print_top_words(lda, ngrams, 10)<jupyter_output>Topics in LDA model:
Topic #0: price, store, great, product, buy, shipping, local, good, grocery, love
Topic #1: dog, treat, chew, love, toy, bone, pill, teeth, formula, training
Topic #2: coffee, tea, cup, green, flavor, taste, like, strong, bold, love
Topic #3: bar, taste, snack, like, great, good, flavor, love, chocolate, chip
Topic #4: chip, grey, earl, popchips, chocolate, tea, white, pork, best, popcorn
Topic #5: sauce, noodle, hot, soup, spicy, chicken, seasoning, love, good, dish
Topic #6: product, like, tea, taste, just, good, flavor, bag, time, great
Topic #7: food, cat, dog, love, treat, eat, product, chicken, dry, pet
Topic #8: coffee, cup, tea, flavor, taste, like, good, drink, great, strong
Topic #9: pasta, taste, cheese, like, product, bread, good, tasted, great, pretzel<jupyter_text>We can use the topic models the same way we did our classifier - everything in Scikit-Learn follows the same fit/transform paradigm.  So, let's get the topics for our documents.<jupyter_code>doc_topics = pd.DataFrame(lda.transform(tfidf))
doc_topics.head()
topic_column_names = ["topic_{}".format(c) for c in doc_topics.columns]
doc_topics.columns = topic_column_names<jupyter_output><empty_output><jupyter_text>Next we use Pandas to join the topics with the original sample dataframe<jupyter_code>sample_with_topics = pd.concat([sample, doc_topics], axis=1)<jupyter_output><empty_output><jupyter_text>Let's look for patterns by running some means and correlations<jupyter_code>sample_with_topics.groupby("Score").mean()
for topic in topic_column_names:
    print("{}: {}".format(topic, sample_with_topics[topic].corr(sample_with_topics['Score'])))<jupyter_output>topic_0: 0.09057467972739268
topic_1: -0.01960945176482254
topic_2: 0.03272564269106118
topic_3: 0.09310035155250233
topic_4: 0.023547152999123997
topic_5: 0.056220024146283236
topic_6: -0.1972726044480822
topic_7: 0.04712487342416328
topic_8: 0.029392534418780994
topic_9: -0.01027353589334897<jupyter_text>Here's an example of a linear regression<jupyter_code>from sklearn import datasets, linear_model
from sklearn.metrics import mean_squared_error, r2_score

training_data = sample_with_topics[topic_column_names[:-1]] # We're leaving a column out to prevent multicollinearity

regression = linear_model.LinearRegression()

# Train the model using the training sets
regression.fit(training_data, sample_with_topics['Score'])
coefficients = regression.coef_
for topic, coef in zip(topic_column_names[:-1], coefficients):
  print("{}: {}".format(topic, coef))<jupyter_output>topic_0: 0.8510971153838108
topic_1: 0.029369807250430147
topic_2: 0.4264011587663883
topic_3: 0.5430934041571058
topic_4: 0.5423901852913203
topic_5: 0.9001330001162867
topic_6: -0.25980997865674704
topic_7: 0.5483580339434101
topic_8: 0.4090444157982454<jupyter_text>Sadly Scikit-Learn doesn't make it easy to get p-values or a regression report like you'd normally expect of something like R or Stata.  Scikit-Learn is more about prediction than statistical analysis; for the latter, we can use Statsmodels.<jupyter_code>import statsmodels.api as sm

regression = sm.OLS(sample_with_topics['Score'], training_data)
results = regression.fit()
print(results.summary())<jupyter_output>OLS Regression Results                                
=======================================================================================
Dep. Variable:                  Score   R-squared (uncentered):                   0.905
Model:                            OLS   Adj. R-squared (uncentered):              0.905
Method:                 Least Squares   F-statistic:                          1.063e+04
Date:                Wed, 17 Jul 2019   Prob (F-statistic):                        0.00
Time:                        16:38:48   Log-Likelihood:                         -17145.
No. Observations:               10000   AIC:                                  3.431e+04
Df Residuals:                    9991   BIC:                                  3.437e+04
Df Model:                           9                                                  
Covariance Type:            nonrobust                                                  
================================[...]<jupyter_text>Clustering We can also check out other unsupervised methods like clustering.  I borrowed/modified some of this code from http://brandonrose.org/clustering  K-Means Clustering<jupyter_code>from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=10, max_iter=50, tol=.01)
# http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html
kmeans.fit(tfidf)
clusters = kmeans.labels_.tolist() # You can merge these back into the data if you want
centroids = kmeans.cluster_centers_.argsort()[:, ::-1] 
for i, closest_ngrams in enumerate(centroids):
    print("Cluster #{}: {}".format(i, np.array(ngrams)[closest_ngrams[:8]]))<jupyter_output>Cluster #0: ['product' 'price' 'great' 'store' 'order' 'good' 'box' 'time']
Cluster #1: ['cat' 'food' 'eat' 'love' 'treat' 'wellness' 'chicken' 'dry']
Cluster #2: ['taste' 'like' 'good' 'love' 'just' 'great' 'sugar' 'make']
Cluster #3: ['tea' 'green' 'taste' 'drink' 'bag' 'flavor' 'good' 'like']
Cluster #4: ['bar' 'chocolate' 'snack' 'like' 'taste' 'protein' 'nut' 'good']
Cluster #5: ['chip' 'potato' 'bag' 'flavor' 'kettle' 'salt' 'like' 'love']
Cluster #6: ['flavor' 'sauce' 'great' 'good' 'like' 'salt' 'oil' 'popcorn']
Cluster #7: ['coffee' 'cup' 'like' 'flavor' 'strong' 'good' 'roast' 'taste']
Cluster #8: ['dog' 'treat' 'food' 'love' 'chew' 'like' 'product' 'good']
Cluster #9: ['chocolate' 'dark' 'cooky' 'taste' 'hot' 'like' 'good' 'flavor']<jupyter_text>Agglomerative/Hierarchical Clustering<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>We're going to characterize how similar words are to each other, based on their co-occurrence within documents. To keep things manageable, we'll use a smaller set of 500 words.<jupyter_code>tfidf_vectorizer = TfidfVectorizer(
    max_df=0.9, 
    min_df=5, 
    ngram_range=(1, 1), 
    stop_words=stop_words, 
    max_features=500
)
tfidf = tfidf_vectorizer.fit_transform(sample['Text'])
ngrams = tfidf_vectorizer.get_feature_names()
from scipy.cluster.hierarchy import linkage, dendrogram
from sklearn.metrics.pairwise import cosine_similarity

# Uses cosine similarity to get word similarities based on document overlap
# To get this for document similarities in terms of word overlap, just drop the .transpose()!
similarities = cosine_similarity(tfidf.transpose()) 
distances = 1 - similarities # Converts to distances
clusters = linkage(distances, method='ward') # Run hierarchical clustering on the distances
fig, ax = plt.subplots(figsize=(15, 100))
ax = dendrogram(clusters, labels=ngrams, orientation="left", leaf_font_size=12, color_threshold=2.0)
plt.tight_layout()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11552261631668725, 'fraction_numerical': 0.042821010528792426, 'mean_word_length': 4.252394192153228, 'pattern_counts': {'":': 12, '<': 87, '<?xml version=': 0, '>': 88, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '252741', 'license': 'MIT', 'n_tokens_mistral': 6036, 'n_tokens_neox': 5407, 'n_words': 1602, 'path': 'Tutorial.ipynb', 'repo_name': 'patrickvankessel/WorldBank-Text-Analysis-2019', 'size': '17023'}