<jupyter_start><jupyter_text>Movie reviews sentiment analysis with Naive BayesProject 2, MS621*Spring 2021* GoalIn this project, you will build a multinomial naive bayes classifier to predict whether a movie review is positive or negative.  As part of the project, you will also learn to do *k*-fold cross validation testing.You will do your work in a `bayes`-*userid* repository. Please keep all of your files in the root directory of the repository. Do not add data to the repo!  Getting startedDownload and uncompress [polarity data set v2.0](https://www.cs.cornell.edu/people/pabo/movie-review-data/review_polarity.tar.gz) into the root directory of your repository, but do not add the data to git.  My directory looks like:```$ lsbayes.py   review_polarity  test_bayes.py```Download the [test_bayes.py](https://github.com/parrt/msds621/blob/master/projects/bayes/test_bayes.py) script into the root directory of your repository; you can add this if you want but I will overwrite it when testing. It assumes that the `review_polarity` directory is in the same directory (the root of the repository).Download the [bayes.py starter kit](https://github.com/parrt/msds621/blob/master/projects/bayes/bayes.py) into the root directory of your repository. Make sure to add this to the repo.See Naive Bayes discussion, p258 in [Introduction to Information Retrieval](https://nlp.stanford.edu/IR-book/).**Please do not add the data to your repository!**  Discussion  Naive bayes classifiers for text documentsA text classifier predicts to which class, $c$, an unknown document $d$ belongs. In our case, the predictions are binary: $c=0$ for negative movie review and $c=1$ for positive movie review. We can think about classification mathematically as picking the most likely class:$$c^*= \underset{c}{argmax} ~P(c|d)$$We can replace $P(c|d)$, using Bayes' theorem:$$P(c | d) = \frac{P(c)P(d|c)}{P(d)}$$to get the formula $$c^*= \underset{c}{argmax} \frac{P(c)P(d|c)}{P(d)}$$Since $P(d)$ is a constant for any given document $d$, we can use the following equivalent but simpler formula to find the most likely (even though the scalar $P(c)P(d|c)$ will not technically be a probability):$$c^*= \underset{c}{argmax} ~ P(c)P(d|c)$$Training then consists of estimating $P(c)$ and $P(c|d)$, which we'll get to shortly.  Representing documentsText classification requires a representation for document $d$. When loading a document, we first load the text and then tokenize the words, stripping away punctuation and stop words like *the*. The list of words is a fine representation for a document except that each document has a different length, which makes training most models problematic as they assume tabular data with a fixed number of features.  The simplest and most common approach is to create an overall vocabulary, $V$, created as a set of unique words across all documents in all classes. *Sort the unique words in the vocab alphabetically so we standardize which word is associated with which word vector index.* Then, the training features are those words. One way to represent a document then is with a binary word vector, with a 1 in each column if that word is present in the document. Something like this:<jupyter_code>import pandas as pd
df = pd.DataFrame(data=[[1,1,0,0],
                        [0,0,1,1]],
                  columns=['cat','food','hong','kong'])
df<jupyter_output><empty_output><jupyter_text>This tends to work well for very short strings/documents, such as article titles or tweets. For longer documents, using a binary presence or absence loses information. Instead, it's better to count the number of times each word is present.  For example, here are 3 documents and resulting word vectors:<jupyter_code>d1 = "cats food cats cats"
d2 = "hong kong hong kong"
d3 = "cats in hong kong"  # assume we strip stop words like "in"
df = pd.DataFrame(data=[[3,1,0,0],
                        [0,0,2,2],
                        [1,0,1,1]],
                  columns=['cat','food','hong','kong'])
df<jupyter_output><empty_output><jupyter_text>These word vectors with fixed lengths are how most models expect data, including sklearn's implementation.  Here's how to train a Naive Bayes model with sklearn using the trivial/toy `df` data and get the training set error:<jupyter_code>from sklearn.naive_bayes import MultinomialNB
import numpy as np

X = df.values
y = [0, 1, 1] # assume document classes
sknb = MultinomialNB()
sknb.fit(X, y)
y_pred = sknb.predict(X)
print(f"Correct = {np.sum(y==y_pred)} / {len(y)} = {100*np.sum(y==y_pred) / len(y):.1f}%")<jupyter_output>Correct = 3 / 3 = 100.0%
{'fraction_non_alphanumeric': 0.09731252709146077, 'fraction_numerical': 0.012787169484178587, 'mean_word_length': 4.580411124546554, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '111000', 'license': 'MIT', 'n_tokens_mistral': 1397, 'n_tokens_neox': 1313, 'n_words': 591, 'path': 'projects/bayes/bayes.ipynb', 'repo_name': 'Ming2010/msds621', 'size': '4614'}