<jupyter_start><jupyter_text>Text Classification In this chapter, we'll go over how to perform *text classification* with Python. As another form of natural language processing, text classification will allow us to automatically assign a set of pre-determined categories, or tags, to  unstructured texts.  Setup We'll get started by installing `matplotlib` and importing `pandas`.<jupyter_code>%matplotlib inline

import pandas as pd<jupyter_output><empty_output><jupyter_text>Data As an example, we'll be looking at a dataset containing [reviews](https://www.kaggle.com/zynicide/wine-reviews/data) from Wine Enthusiast magazine.<jupyter_code>#https://www.kaggle.com/zynicide/wine-reviews/data

wine_df = pd.read_csv('wine_reviews.csv')<jupyter_output><empty_output><jupyter_text>We can quickly get a sense of the distribution of reviews by looking at the `value_counts` associated with different `point` scores.<jupyter_code>wine_df['points'].value_counts()<jupyter_output><empty_output><jupyter_text>It could also be helpful for us to read over a few of the `descriptions` of the wines. However, since `pandas` defaults to a maximum column width of 50 characters, it's difficult to glean much from each review without specifying a longer maximum character width for the columns in our dataframe.<jupyter_code>wine_df['description'][:5]<jupyter_output><empty_output><jupyter_text>![google_search.png](images/google_search.png) We'll use the `set_options` function in `pandas` to set our maximum column width to 120 characters.<jupyter_code>pd.set_option('display.max_colwidth', 120)<jupyter_output><empty_output><jupyter_text>Now we'll have more text to work with when we're looking at the first five descriptions:<jupyter_code>wine_df['description'][:5]<jupyter_output><empty_output><jupyter_text>We can also use the `head` function to see all the keys included in the dataframe:<jupyter_code>wine_df.head()<jupyter_output><empty_output><jupyter_text>Classifying Texts in Python The following sections will walk us through a step-by-step guide of how to classify texts in Python.  Turning words into featuresIn order to classify our texts, we'll need to convert each text document in our corpus into a fixed-length *vector*. We can accomplish this using the [CountVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html?highlight=vectorizersklearn.feature_extraction.text.CountVectorizer) function available through the scikit-learn library. Using `CountVectorizer`, we'll be able to transform the texts in our wine review dataframe into a matrix of token counts.<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer<jupyter_output><empty_output><jupyter_text>1) Set up your model, fixing any parameters.`CountVectorizer` can take a number of parameters; while not an exhaustive list, some important parameters are described below.   `CountVectorizer` Parameters- **lowercase**: Converts all text to lower case. By default, `lowercase` is set to True.- **ngram_range**: This allows us to restrict the range of n-values for our n-grams. Formatted as a tuple, the first sets the minimum n-value, and the second sets the maximum n-value. By default, `n-gram range` is set to (1,1). (If we leave it alone, we'll only be looking at unigrams.) - **stop_words**: We can use this parameter to rule out words that occur 1) too frequently, 2) not frequently enough, and/or 3) fall outside of a threshold term frequency. This can be set the 'english' to use a pre-determined set of stopwords often found in texts written in the English language. We can also provide our own list of stopwords if we so choose. > - **max_df**: Allows us to set a maximum threshold on document frequency for our terms incorporated into our vocabulary. By default, `max_df` is set to 1.0. > - **min_df**: Allows us to set a minimum threshold on document frequency for our terms incorporated into our vocabulary. By default, `min_df` is set to 1.0. > - **max_features**: Allows us to build a vocabulary exclusively from high-frequency terms occuring throughout our corpus.  For this exercise, we'll convert our text to lower case, look only at unigrams, use "english" stopwords, and set a minimum document frequency for terms of .01.<jupyter_code>vectorizer = CountVectorizer(lowercase   = True,
                             ngram_range = (1,1),
                             stop_words  = 'english',
                             min_df      = .01,
                             max_features = None)<jupyter_output><empty_output><jupyter_text>2) Fit your modelWe'll `fit` the vectorizer we've created to the `description` key in our wine review dataframe. `model.fit(X)`<jupyter_code>vectorizer.fit(wine_df['description'])<jupyter_output><empty_output><jupyter_text>We can use the `len` function to see how many features were generated when applying the vectorizor to our corpus.<jupyter_code>len(vectorizer.get_feature_names())<jupyter_output><empty_output><jupyter_text>3) Create new data based on your model Now we can `transform` the text documents in the `description` key of our wine review dataframe into a document-term matrix.<jupyter_code>review_word_counts = vectorizer.transform(wine_df['description'])<jupyter_output><empty_output><jupyter_text>Let's take a look at the value counts for the dummy variable `rating` in our dataframe.<jupyter_code>wine_df['rating'].value_counts()<jupyter_output><empty_output><jupyter_text>4) Set up your (second) model, fixing any parameters. Multinomial NBAlso imported from the scikit-learn library, [MultinomialNB](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html?highlight=multinomialnbsklearn.naive_bayes.MultinomialNB) uses the naive Bayes algorithm to classify multinomially distributed data. This classifier is well-suited to the token counts we've previously generated with our vectorizer.<jupyter_code>from sklearn.naive_bayes import MultinomialNB
nb_classifier = MultinomialNB()<jupyter_output><empty_output><jupyter_text>5) Fit your (second) model Before we `fit` the classifier to our data, we'll have to specify a couple of **parameters**: - **X**: These are the *training* vectors for our model. For our wine review data, we'll use our `review_word_counts` as our training vectors.- **Y**: These are the *target* values for our model. For our wine review data, we'll use the `high_rating` key to determine the target values for our model.  `model.fit(X)` `model.fit(X, Y)`<jupyter_code>nb_classifier.fit(review_word_counts, wine_df['rating'])<jupyter_output><empty_output><jupyter_text>We can now view a list of coefficients associated with each of our features:<jupyter_code>nb_classifier.coef_[0]
coeficients = pd.Series(nb_classifier.coef_[0],
                        index = vectorizer.get_feature_names())<jupyter_output><empty_output><jupyter_text>`sort_values` lets us view the top terms used to describe the wines reviewed in our corpus.<jupyter_code>coeficients.sort_values(ascending=True)[:20]<jupyter_output><empty_output><jupyter_text>Classification Exercise 1Construct a model of UN speeches to distinquish between those before and after the collapse of the Soviet Union.  5) Create new data based on your (second) model Our classifier can now be used to `predict` whether a wine received a relatively `High` or `Low` rating, based on the words used to describe the wine.<jupyter_code>nb_classifier.predict(review_word_counts)
wine_df['prediction']  = nb_classifier.predict(review_word_counts)<jupyter_output><empty_output><jupyter_text>We can run a crosstab with `pandas` to quickly get a feel for how accurately our classifier predicts `High` and `Low` reviews:<jupyter_code>pd.crosstab(wine_df['rating'], wine_df['prediction'])<jupyter_output><empty_output><jupyter_text>For more detailed visualizations of prediction accuracy, we can import `classification_report`, `confusion_matrix`, and `accuracy_score` from scikit-learn.<jupyter_code>from sklearn.metrics import classification_report, confusion_matrix, accuracy_score<jupyter_output><empty_output><jupyter_text>With [accuracy_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html?highlight=accuracy_scoresklearn.metrics.accuracy_score), we can get a numerical indicator of prediction accuracy, rather than just eyeball using a crosstab. We'll need to specify the following:- **y_true**: The correct labels our model is trying to predict. For this example, we'll set `y_true` to `rating`.  - **y_pred**:  The labels predicted by our classifier. For this example, we'll set `y_pred` to `prediction`.<jupyter_code>accuracy_score(wine_df['rating'], wine_df['prediction'])<jupyter_output><empty_output><jupyter_text>Running a [confusion_matrix](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html?highlight=confusion_matrixsklearn.metrics.confusion_matrix) produces results similar to the crosstab we generated earlier. However, through the us of the [heatmap](https://seaborn.pydata.org/generated/seaborn.heatmap.html) function available through the seaborn library, we can create a visual representation of the degree of prediction accuracy.<jupyter_code>import seaborn as sns

cm = confusion_matrix(wine_df['rating'], wine_df['prediction'])
sns.heatmap(cm, annot=True, cmap="Greens", fmt='g')<jupyter_output><empty_output><jupyter_text>Finally, we can generate a [classification_report](https://scikit-learn.org/stable/modules/model_evaluation.htmlclassification-report) to return a text report with some helpful classification metrics.- **Precision**: % of selected items that are correct - **Recall**: % of correct items that are selected<jupyter_code>print(classification_report(wine_df['rating'], wine_df['prediction']))<jupyter_output>precision    recall  f1-score   support

        High       0.77      0.80      0.79     33635
         Low       0.87      0.85      0.86     51493

    accuracy                           0.83     85128
   macro avg       0.82      0.82      0.82     85128
weighted avg       0.83      0.83      0.83     85128<jupyter_text>We can also use our classifer to generate an array of probabilities that each of our features will occur in any given wine review, depending on whether a review is associated with a `High` or `Low` score:<jupyter_code>nb_classifier.predict_proba(review_word_counts)
predict_df = pd.DataFrame(nb_classifier.predict_proba(review_word_counts), 
                          columns=nb_classifier.classes_)
predict_df.head()
wine_df_prediction = pd.concat([wine_df, predict_df], axis = 1)
wine_df_prediction.sort_values('High', ascending=False)[['description','points']].head(15)
wine_df_prediction.sort_values('Low', ascending=False)[['description','points']].head(15)<jupyter_output><empty_output><jupyter_text>Classification Exercise 2 Which **post-1989** speech had the highest likelihood of being delivered during an earlier period?  What about overfitting? Up to this point in the chapter, we've done little to account for potentially overfitting our model to the data. When we overfit a model, we run the risk of describing the random errors in our dataset, rather than the relationships we're actually interested in analyizing. To get around this, we'll import [train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html?highlight=train_test_splitsklearn.model_selection.train_test_split), which allows us to set aside a random subset of texts in our corpus to test the fit of our model. We can indicate the proportion of the total body of texts we'd like to set aside for testing the model using the `test_size` parameter. In the following example, we'll set aside 20% of the wine reviews for testing by setting `test_size` to 0.2.<jupyter_code>from sklearn.model_selection import train_test_split

train, test = train_test_split(wine_df, test_size=0.2)
len(train)
len(test)<jupyter_output><empty_output><jupyter_text>Now that we've split up our data for training and testing, we can set the parameters of our vectorizer, then `fit` the vectorizer to our training data.<jupyter_code>vectorizer = CountVectorizer(lowercase   = True,
                             ngram_range = (1,2),
                             stop_words  = 'english',
                             min_df      = .01,
                             max_features = None)

vectorizer.fit(train['description'])<jupyter_output><empty_output><jupyter_text>We can use the `transform` method to return a document-term metrix for our training data.<jupyter_code>X_train = vectorizer.transform(train['description'])<jupyter_output><empty_output><jupyter_text>Using the MultinomialNB model from we imported earlier in the chapter, we can `fit` the matrix we've generated for descriptions of the wine reviews in the training data to the `rating` dummy variable indicating whether a wine received a high or low rating.<jupyter_code>nb_classifier.fit(X_train, train['rating'])<jupyter_output><empty_output><jupyter_text>The visualizations provided through scikitlearn's `classification_report`, `confusion_matrix`, and `accuracy_score` will come in handy as we compare the performance of our model when applied to our training data versus our test data.<jupyter_code>from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

print(accuracy_score(train['rating'],
                     nb_classifier.predict(X_train)))<jupyter_output>0.841875<jupyter_text>Before we can make this comparison, though, we'll have to `transform` our the raw documents in our test data into a document-term matrix. We'll also need to define our prediction model using the testing data.<jupyter_code>test_wf         = vectorizer.transform(test['description'])
test_prediction = nb_classifier.predict(test_wf)
print(accuracy_score(test['rating'], test_prediction))
vectorizer = CountVectorizer(lowercase=True,
                             ngram_range = (1,3),
                             stop_words = 'english',
                             max_df = .60,
                             min_df = 5,
                             max_features = None)
vectorizer.fit(train['description'])
print(len(vectorizer.get_feature_names()))
X_train = vectorizer.transform(train['description'])
nb_classifier.fit(X_train, train['rating'])

print(accuracy_score(train['rating'],
                     nb_classifier.predict(X_train)))



print(accuracy_score(test['rating'],
                     nb_classifier.predict(vectorizer.transform(test['description']))))<jupyter_output>0.818<jupyter_text>Classification Exercise 3What happens to your model if you change some of the parameters for your vectorizer? Be sure to split the data between train and test!  Logistic Regression In addition to MultinomialNB, we can also import a [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html?highlight=logisticregressionsklearn.linear_model.LogisticRegression) model from scikitlearn.<jupyter_code>from sklearn.linear_model import LogisticRegression
ln_classifier = LogisticRegression()<jupyter_output><empty_output><jupyter_text>We can define the parameters of our vectorizer, and fit the vectorizer to the training data in our wine dataframe. This will produce a total of 463 features.<jupyter_code>vectorizer = CountVectorizer(lowercase=True,
                             ngram_range = (1,2),
                             stop_words = 'english',
                             min_df = .01,
                             max_features = None)

vectorizer.fit(train['description'])

print(len(vectorizer.get_feature_names()))
ln_classifier.fit(vectorizer.transform(train['description']), train['rating'])<jupyter_output>463<jupyter_text>Let's look at the accuracy scores returned for the training and testing data when applying a logistic regression model:<jupyter_code>print(accuracy_score(train['rating'],
                     ln_classifier.predict(vectorizer.transform(train['description']))))
print(accuracy_score(test['rating'],
                     ln_classifier.predict(vectorizer.transform(test['description']))))<jupyter_output>0.8545<jupyter_text>We can also see the `heatmap` function from before to get a quick feel for how our model holds up when fit to our test data:<jupyter_code>test_prediction = ln_classifier.predict(vectorizer.transform(test['description']))

cm = confusion_matrix(test['rating'], test_prediction)
sns.heatmap(cm, annot=True, cmap="Greens", fmt='g')<jupyter_output><empty_output><jupyter_text>Classification Exercise 4What is the out sample accuracy of a logistic regression model on your data?`from sklearn.linear_model import LogisticRegression`  *K* Nearest Neighbors We can also [neighbors-based](https://scikit-learn.org/stable/modules/neighbors.htmlneighbors) classifiers from scikit-learn. Here, we'll import the [KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html), a classifier looking at the *k* nearest neighbors to each query point. In this context, *k* is a user-specified integer value.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

knn_classifier = KNeighborsClassifier(n_neighbors = 3)<jupyter_output><empty_output><jupyter_text>Rather than implement a CountVecotrizer as we have for our previous models in this chapter, we'll be importing the [TfidfVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html?highlight=tfidfvectorizersklearn.feature_extraction.text.TfidfVectorizer) for our *k* neighbors classifier. This way,  we'll be able to look at weighted word frequencies across documents. For a detailed discussion of td-idf, see the chapter on word frequencies.<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer<jupyter_output><empty_output><jupyter_text>The task of setting our parameters is nearly identical to what we encountered when setting up our `CountVectorizer`.<jupyter_code>tf_vector  = TfidfVectorizer(lowercase  =  True,
                             ngram_range = (1,2),
                             stop_words  = 'english',
                             max_df      = .60,
                             min_df      = .05,
                             max_features = None)<jupyter_output><empty_output><jupyter_text>We'll also reconfigure our `train_test_split`:<jupyter_code>train, test = train_test_split(wine_df, test_size=0.7)<jupyter_output><empty_output><jupyter_text>Now, 30% of our total data is allocated to training the model, and 70% to testing it.<jupyter_code>len(train)
len(test)<jupyter_output><empty_output><jupyter_text>We'll `transform` and `fit` as we have in the past:<jupyter_code>tf_vector.fit(train['description'])
review_tf = tf_vector.transform(train['description'])
knn_classifier.fit(review_tf, train['rating'])
knn_prediction = knn_classifier.predict(review_tf)

print(accuracy_score(train['rating'], knn_prediction))<jupyter_output>0.8431748766543974<jupyter_text>We can also generate classification reports and heatmaps as we have before to visualize the strength of our *k* nearest neighbors classifer:<jupyter_code>print(classification_report(train['rating'], knn_prediction))
cm = confusion_matrix(train['rating'], knn_prediction)
sns.heatmap(cm, annot=True, cmap="Greens", fmt='g')<jupyter_output><empty_output><jupyter_text>Classification Exercise 5What does a k-nearest neigbhor for your speech dataset look like? (Don't forget to shrink your dataframe). How does the accuracy compare? ![](images/knn2.png)<jupyter_code>knn_classifier = KNeighborsClassifier(n_neighbors = 15)<jupyter_output><empty_output><jupyter_text>But what's the best fitting model?  GridSearchCV Now that we know how to run a variety of text classification models in Python, we'd benefit from a discussion of how to determine which model best fits our data. To figure this out, we'll import [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html?highlight=gridsearchcvsklearn.model_selection.GridSearchCV) from scikit-learn, which allows us to perform an exhaustive grid search cross-validation of parameter values.<jupyter_code>from sklearn.model_selection import GridSearchCV<jupyter_output><empty_output><jupyter_text>This allows us to, for example, determine whether it makes the most sense to set `n_neighbors` in our *k* nearest neighbors classifier to 2, 3 (what it was set to in our original model), or 7.<jupyter_code># old model: knn_classifier = KNeighborsClassifier(n_neighbors = 3)

parameters = {'n_neighbors' : [2,3, 7],
              'weights'      : ['distance', 'uniform']}
              
grid = GridSearchCV(KNeighborsClassifier(), 
                    parameters, 
                    cv = 5)
grid<jupyter_output><empty_output><jupyter_text>Pipeline Using [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) along with GridSearchCV allows us to combine a list of *transforms* (estimators using the `transform` method) into a single estimator.<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV<jupyter_output><empty_output><jupyter_text>Below, we'll use Pipline to look at two transforms simultaneously: the `vectorizer` set up through `CountVectorizer`, and the `classifier` set up through `LogisticRegression`. We'll evaluate goodness of fit for models containing 300, 500, and 700 maximum features in the vectorizer.<jupyter_code>pipeline = Pipeline([
                     ('vectorizer' , CountVectorizer()),
                     ('classifier' , LogisticRegression())
                    ])

parameters = {'vectorizer__max_features' : [300, 500, 700],
               }<jupyter_output><empty_output><jupyter_text>Before we run GridSearchCV on our pipeline, let's take some time to set up the following parameters. While we've encountered `n_jobs` and `verbose` in preceding chapters, let's take some time to consider a new parameter:- **cv**: Allows us to choose one of several cross-validation strategies. By default, `cv` is set to 5, so that we perform a 5-fold cross-validation.<jupyter_code>grid_search = GridSearchCV(pipeline,
                           parameters,
                           n_jobs = -1,
                           cv = 5,
                           verbose = 1)<jupyter_output><empty_output><jupyter_text>Now we can fit our combined estimator to our 3 candidate models:<jupyter_code>grid_search.fit(wine_df['description'],
                wine_df['rating'])<jupyter_output>Fitting 5 folds for each of 3 candidates, totalling 15 fits<jupyter_text>We can use `grid_search` to determine the best accuracy score, the best estimator, and the best parameters among our candidates:<jupyter_code>grid_search.best_score_
grid_search.best_estimator_
grid_search.best_estimator_.get_params<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0865413827524847, 'fraction_numerical': 0.009721800269085543, 'mean_word_length': 4.963250517598344, 'pattern_counts': {'":': 0, '<': 190, '<?xml version=': 0, '>': 193, 'https://': 15, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '49', 'dir': 'jupyter-structured-clean-dedup', 'id': '498033', 'license': 'MIT', 'n_tokens_mistral': 6919, 'n_tokens_neox': 6537, 'n_words': 2080, 'path': 'Notebooks/Text-Classification.ipynb', 'repo_name': 'nealcaren/Text-Mining-with-Python-Notebooks', 'size': '23041'}