<jupyter_start><jupyter_text>Bag of Words and NBGaussian Jeopardy! solver<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import sklearn as sk

from pandas import Series, DataFrame

plt.style.use('ggplot')

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

from datasets import load_dataset

# Load jeopardy dataset
jeopardy = load_dataset('jeopardy')
# Load all of the Jeopardy! data
j = pd.DataFrame(jeopardy['train'])
# Remove extraneous data...
j = j.drop(columns=['air_date', 'show_number', 'round', 'value', 'category'])
j<jupyter_output><empty_output><jupyter_text>Preprocessing- Strip all punctuation- Make everything lowercase- Remove whitespace- Remove video clues (either have a URL to a video or say "clue crew" because they need a visual)- Remove answers that are empty- Remove stop words<jupyter_code>j.answer = (j.answer
                .str.strip()
                .str.lower()
           )
j.question = (j.question
                 .str.strip()
                 .str.lower()
             )
j
j.loc[j.question.str.contains('<br')]
# Remove `<br />` HTML tags
j.question = j.question.str.replace('<br />', '')
j
# Remove punctuation
regex = r'[-\,()\"#\/@;:<>\{\}!\-=~|\.\?\&+^%$_\'\*\\]'
j = j.replace(to_replace=regex, value="", regex=True)

# Drop video clues (contains clue crew or href)
j = j.loc[j.question.str.contains("clue crew") == False]
j = j.loc[j.question.str.contains("href") == False]

# example
j[j['answer'] == "jfk"]<jupyter_output><empty_output><jupyter_text>Processing for NLPI followed [this guide](https://medium.com/@datamonsters/text-preprocessing-in-python-steps-tools-and-examples-bf025f872908) by <NAME> on Medium and the NLP lecture from the CSCI303 TA's for most of this text clean up.<jupyter_code>import nltk
from nltk.stem import WordNetLemmatizer
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer

#nltk.download('punkt')
#nltk.download('wordnet')
from nltk.stem import PorterStemmer
stop_words = set(stopwords.words('english'))


def tokenize(entry):
    return word_tokenize(entry)


def remove_stop_words(entry):
    """ Removes stop words given a row from a dataframe"""
    return [i for i in entry if not i in stop_words]


def lemma(entry):
    """Lemmatize all of the words of each entry...defaulting to noun lemmatization"""
    lemmatizer = WordNetLemmatizer()
    return [lemmatizer.lemmatize(word) for word in entry]


def clean(entry):
    entry = tokenize(entry)
    entry = remove_stop_words(entry)
    entry = lemma(entry)
    
    # TODO: decide if stemming makes anything better...
    #entry = stem(entry)
    return " ".join(entry)


def stem(entry):
    ps = PorterStemmer()
    return [ps.stem(word) for word in entry]

# Tokenize, remove stop words, and lemmatize
j['clean'] = j.apply(lambda row: clean(row['question']), axis=1)
j<jupyter_output><empty_output><jupyter_text>Lemmatization constraintsThe NLTK WordNet Lemmatizer defaults to noun lemmatization, so conjugations of verbs (e.g., "becomes" and "became" --> "become") will not be lemmatized. In the future, I would like to explore other lemmatizer libraries (like Gensim) that work better with an unknown part of speech. --- Data exploration<jupyter_code>group = j.groupby('answer')
group.first()
# Count unique values
unique = j.answer.value_counts().to_frame()

# Compare common answers to the number of times they have been asked
unique.rename(columns={"answer":"answer frequency"})<jupyter_output><empty_output><jupyter_text>There are around 71,000 unique answers in the dataset, but many of them are rare: there are 209 clues about China, but only one clue for most of the answers in the set. For model fitting purpose, I will not train on clue answers that are rare.<jupyter_code>unique = unique.answer.value_counts().to_frame()
unique = unique.reset_index().rename(columns={"index":"clue_appearances", "answer":"number_of_clues"})
unique<jupyter_output><empty_output><jupyter_text>In the above dataframe, note that there are >43000 such clues that only have one sample. Removing clues that appear less than 5-10 times will increase the accuracy of the model at the expense of breadth.<jupyter_code>unique.plot(kind='scatter', x="clue_appearances", y="number_of_clues")
plt.title("Number of clues vs. Clue appearances")
plt.show()<jupyter_output><empty_output><jupyter_text>The above chart plots how many clues have been asked a certain number of times. For example, there is only one clue that has been asked 209 times, China (bottom right point), while there are over 40,000 clues that have been asked about only once (top left point).Most of the clues in this data set are unique (~43k answers have only been seen once), so these should be excluded from the dataset. Many of the clues that have only appeared once or twice are word play like "the good, the bad, and the ugly duckling" or rare answers that would be difficult to predict accurately with such a small sampling (like "things with chains," "grandma and grandpa"). The plot below removes some of the rarer clues to show a more accurate distribution of clue frequency vs. number of samples<jupyter_code>unique[unique.clue_appearances > 25].plot(kind='scatter', x="clue_appearances", y="number_of_clues")
plt.title("Number of clues vs. Clue appearances")
plt.show()<jupyter_output><empty_output><jupyter_text>Representing data with bag of wordsI followed a simple methodology for extracting meaning from each clue called bag of words. Bag of words is a form of feature extraction that represents the content of each clue by using the frequency of each word's appearance as an entry of a vector. For example, the sentence "the dog and the cat both like dog food" could be turned into the following vector representation, where the integer represents the frequency of that word in the sentence. ```pseudocode{ "the" : 2,  "dog" : 2,  "and" : 1,  "cat" : 1,  "both" : 1,  "like" : 1,  "food" : 1}  ```If we used a larger vocabulary (e.g., all of the words that have occured in a Jeopardy clue), we could apply this vectorization to each clue sequentially and represent features of the text. Each clue has to be represented using the same ordered vocabulary (the length of each vector == the number of words in vocabulary), so they will be sparse and contain a lot of zero scores.This is the framework I followed for my methodology from [NLP Project Using Bag-of-Words](https://in.springboard.com/blog/nlp-project-automated-question-answering-model/)>The bag-of-words model represents the text as a multiset of words. In essence, it’s exactly as the name suggests — it is a bag of individual words, without taking into account the word order or grammar. To perform retrieval using the bag-of-words in an NLP model, we typically follow these steps:>>- Break the dataset into individual words>- Make a dictionary of the words>- Print the dictionary>- Create a bag-of-words corpus, from which you can make a sparse representation for each sentence>- Get bag-of-words representation for a user query>- Loop over all the question vectors created and perform a cosine similarity function with the query vector>- When you use this model, you’ll be comparing the exact words in your query sentence and FAQs to identify similarities. This means that the bag-of-words model doesn’t process underlying meaning or context, resulting in inaccurate results.**Other references in this section**- https://machinelearningmastery.com/gentle-introduction-bag-words-model/<jupyter_code># Filter out the answers that don't appear frequently enough
# https://stackoverflow.com/questions/49137031/pandas-dataframe-delete-rows-with-low-frequency

min_samples = 10

j = j[j.groupby('answer')['answer'].transform('count').ge(min_samples)]
j = j.rename(columns={'question': 'original_question', 'answer':'original_answer', 'clean':'clean_text'})
j<jupyter_output><empty_output><jupyter_text>Here, I'll define the set of words that form the vocabulary and will be used to vectorize each clue.<jupyter_code># Define vocabulary 
vocabulary = set()
j.clean_text.str.split().apply(vocabulary.update)
vocabulary<jupyter_output><empty_output><jupyter_text>To increase the accuracy of the model, I want to combine all of the training clues into a single vector, so I am going to split my data into training and testing sets. With the training data, I want to concatenate each training clue into one prompt, prior to vectorizing.<jupyter_code># Train-test-split!
from sklearn.model_selection import train_test_split

train, test = train_test_split(j, test_size=0.1)
train.shape, test.shape
# Smash answers together to create lookup table with BoW for the train set
# e.g., all of the questions about China should be described by one vector
train = pd.DataFrame(train.groupby('original_answer')['clean_text'].apply(' '.join))
train<jupyter_output><empty_output><jupyter_text>TF-IDF for vectorization$\mathrm{TF-IDF}$ (or term frequency-inverse document frequency) is a measure that better accounts for the weight of rare words. The $\mathrm{TF-IDF}$ value for a word increases with its frequency (term frequency), but it is offset by its popularity in other clues (inverse document frequency).In other words, we can choose to give weight to more meaningful words using $\mathrm{TF-IDF}$, as opposed to just measuring term frequency. When two documents share a rare word (e.g., 'Mugabe'), it is a more meaningful link than sharing a common word (e.g., 'woman').<jupyter_code># Create BoW vectors for each set of data

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer


def create_vectors(df, col, vocabulary, is_dense=True, use_tfidf=True):
    """Creates vectors for a given column of a dataframe.
            Parameters:
                df (DataFrame): a dataframe
                col (string): name of a column in dataframe
                vocabulary (iterable): set of words for vectorization
                is_dense (boolean, optional): default returns a dense matrix, 
                        False returns csr_matrix
                use_tfidf (boolean, optional): default uses tf-idf for vectorization, 
                        False uses tf only
            Returns:
                vectors: a dataframe of sparse/dense vectors
    """
    
    if use_tfidf:
        vectorizer = TfidfVectorizer(vocabulary=vocabulary)
        X = vectorizer.fit_transform(df[col])
    else: 
        # Vectorize using CountVectorizer
        vectorizer = CountVectorizer(vocabulary=vocabulary)
        X = vectorizer.fit_transform(df[col])
        
    if is_dense:
        # output vectors should be dense
        vectors = pd.DataFrame(X.todense(), columns=vectorizer.get_feature_names())
    else: 
        vectors = pd.DataFrame(X, columns=['vec'])
    return vectors
# create bag of words for the Training set
train_vectors = create_vectors(train, 'clean_text', vocabulary)

# reset vectors on jeopardy so they align correctly
train = train.reset_index().rename(columns={'index':'q_id'})

# Concat the vectors to the right side of the train! dataframe
# pd.concat(..., ignore_index=True) isn't working how I expect?
train = pd.concat([train, train_vectors], axis=1)
# repeat above process for the test set
test_vectors = create_vectors(test, 'clean_text', vocabulary) 

test = test.reset_index().rename(columns={'index':'q_id'})

test = pd.concat([test, test_vectors], axis=1)
test = test.drop(columns=['clean_text', 'q_id'])
test<jupyter_output><empty_output><jupyter_text>Model classification using Gaussian Naive Bayes Naive Bayes classifiers use Bayes's theorem to describe the probability of events and assign class labels to samples. I chose to use the Gaussian Naive Bayes model because it works well with text data. While I don't go into the math behind it, it is pretty interesting.I originally tried to use cosine similarity to measure the similarity between vectors, but I faced challenges with using a sparse matrix representation, long runtime & running out of memory, and calculating meaningful differences in each word's vector representation. Cosine similarity is a widely accepted measurement for phrase similarity with bag of words vectors, but I couldn't get it to work.Framing this problem as a form of classification and using Gaussian Naive Bayes opened up the most room for finding and evaluating results. The cell below will train the classifier and make predictions, but it takes my computer around 10-15 minutes to train on this dataset. **References:**- [Text classification using the Bag Of Words Approach with NLTK and Scikit Learn](https://medium.com/swlh/text-classification-using-the-bag-of-words-approach-with-nltk-and-scikit-learn-9a731e5c4e2f)- [Naive Bayes classifier](https://en.wikipedia.org/wiki/Naive_Bayes_classifier)<jupyter_code>from sklearn.naive_bayes import GaussianNB
X_train = train.loc[:,"0":]
y_train = train.original_answer

X_test = test.loc[:,"0":]
y_test = test.original_answer
classifier = GaussianNB()
classifier.fit(X_train, y_train)

y_hat = classifier.predict(X_test)
from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_test, y_hat)
print("Model accuracy:", accuracy * 100)
from sklearn.metrics import classification_report
classification_report = classification_report(y_test, y_hat, output_dict=True, zero_division=0)
cr = pd.DataFrame(classification_report).T
cr = cr.reset_index()

precision_avg = cr.loc[559]['precision']
recall_avg = cr.loc[559]['recall']
f1_avg = cr.loc[559]['f1-score']


cr = cr.iloc[:-3]  ## drop metadata
print(precision_avg, recall_avg, f1_avg)
cr
plt.plot(cr.support, [precision_avg for i in cr.support], '-b', label='Mean')
plt.scatter(x=cr.support, y=cr.precision, label='Precision')

plt.xlabel('Support')
plt.ylabel('Precision')
plt.legend()
plt.show()
plt.plot(cr.support, [recall_avg for i in cr.support], '-b', label='Mean')
plt.scatter(x=cr.support, y=cr.recall, color='g', label='Recall')
plt.xlabel('Support')
plt.ylabel('Recall')
plt.legend()
plt.show()
plt.plot(cr.support, [f1_avg for i in cr.support], '-b', label='Mean')
plt.scatter(x=cr.support, y=cr['f1-score'], color='orange', label='Recall')
plt.xlabel('Support')
plt.ylabel('f1-score')
plt.legend()
plt.show()
# Confusion matrix
from sklearn.metrics import confusion_matrix

labels = labels=np.unique(y_test)
conf_mat = pd.DataFrame(confusion_matrix(y_test, y_hat, labels=labels), 
                        index=labels, 
                        columns=labels)
conf_mat
from seaborn import heatmap

plt.figure(figsize=(100,30))
heatmap(conf_mat.sample(n=30), square=True, cbar_kws={"orientation": "horizontal"})
plt.show()
# Drop vectors from test df now
test = test.loc[:, "original_answer":"original_question"]
test['class_label'] = y_hat

correct = test[test.original_answer == test.class_label]
incorrect = test[test.original_answer != test.class_label]
incorrect.loc[6]
incorrect.loc[2971].original_question
correct.loc[86].original_question<jupyter_output><empty_output><jupyter_text>Future work The nature of using bag of words on this large of a vocabulary means that I have very sparse vector representations. SciPy has a datatype that works very well for this format (`csr_matrix`), but this is incompatible with most of the machine learning algorithms provided in scikit-learn. In the future, I would really like to implement a classification algorithm by hand that supports `csr_matrix` natively because I expect this will rapidly improve runtime. Similarly, the high cost of computation using large vectors limited my ability to compare my model against other models (support-vector machines, cosine similarity, soft cosines, etc.), so this is something I would like to explore in detail.My accuracy hovers around 40% for this model, which isn't great (some of the best players have been more than 95% accurate). I want to explore different libraries for word lemmatization that perform better on unknown parts of speech. In addition, evaluating the performance of other models could help me choose a model that is possibly more accurate. While my sample size was fairly large (>200k samples), there are many questions that are archived online that I could use for training. I'm interested in writing a text mining program that can extract these questions and create a high quality dataset for future training. Finally, my goal for this project was to report probabilistic results (i.e., my model is 60% certain that a clue belongs to class X), but this is out of scope. There are other algorithms and paradigms that I can explore to come up with better results, perhaps a decision tree or random forest. With more time and effort, I can hopefully turn this model into a useful dashboard that can answer (with some degree of certainty) trivia questions from a show like Jeopardy.  Other classification algorithms!<jupyter_code>from sklearn.naive_bayes import MultinomialNB
classifier = MultinomialNB()
classifier.fit(X_train, y_train)
from sklearn.metrics import accuracy_score
mnb_yhat = classifier.predict(X_test)
mnb_acc = accuracy_score(y_test, mnb_yhat)
print("Model accuracy:", mnb_acc * 100)
tests = pd.read_csv('today.csv')
tests.head()
def process(df):
    df.answer = (df.answer
                .str.strip()
                .str.lower()
           )
    df.question = (df.question
                 .str.strip()
                 .str.lower()
             )
    
    df.question = df.question.str.replace('<br />', '')
    
    # Remove punctuation
    regex = r'[-\,()\"#\/@;:<>\{\}!\-=~|\.\?\&+^%$_\'\*\\]'
    df = df.replace(to_replace=regex, value="", regex=True)

    # Drop video clues (contains clue crew or href)
    df = df.loc[df.question.str.contains("clue crew") == False]
    df = df.loc[df.question.str.contains("href") == False]
    
    # Apply BoW
    df['clean'] = df.apply(lambda row: clean(row['question']), axis=1)
    
    return df
    
tests = process(tests)
today = create_vectors(tests, 'clean', vocabulary)
today = pd.concat([tests, today], axis=1)
X_today = today.loc[:,"0":]
y_today = today.answer
yhat_today = classifier.predict(X_today)
probs = classifier.predict_proba(X_today)
today['class_label'] = yhat_today
today[['question', 'answer', 'class_label']]
# pd.concat([today[['question', 'answer']], yhat_today])
today
probs = DataFrame(probs, columns=classifier.classes_, index=today.iloc[:,1])
probs
probs = probs.apply(lambda x: x / x.max())
probs.T.nlargest(5, "colorado", keep='all')
probs<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1001182922894935, 'fraction_numerical': 0.006774922034627379, 'mean_word_length': 4.613945064895865, 'pattern_counts': {'":': 8, '<': 60, '<?xml version=': 0, '>': 72, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '559934', 'license': 'MIT', 'n_tokens_mistral': 5659, 'n_tokens_neox': 5333, 'n_words': 2029, 'path': 'gaussian.ipynb', 'repo_name': 'cadecrandall/nlp-jeopardy', 'size': '18605'}