<jupyter_start><jupyter_text>The notebook shows an example how text files, labelled for sentiment (positive or negative), can be converted to a tabular format and then used to train a machine learning model capable of classifying new texts by sentiment.As an example, we will use a corpus of movie reviews first used in (Pang and Lee, 2004), for details see [here](http://www.cs.cornell.edu/people/pabo/movie-review-data/).<jupyter_code># setting logging to print only error messages of sklearnex
import logging
logging.basicConfig()
logging.getLogger("SKLEARNEX").setLevel(logging.ERROR)

import pandas as pd
import matplotlib.pyplot as plt

import time<jupyter_output><empty_output><jupyter_text>Load the dataIf the data has not been downloaded before, download it, and save to a folder called "datasets", alongside this notebook.<jupyter_code>import os
import urllib
import tarfile

url = "http://www.cs.cornell.edu/people/pabo/movie-review-data/review_polarity.tar.gz"

def download_data(url):
    """Download the data and extract the mo
    """
    
    # if the "datasets" folder does not exist, create it
    if not os.path.exists("datasets"):
        os.makedirs("datasets")
    
    # if the archived file does not exist, download it
    if not os.path.exists("datasets/review_polarity.tar.gz"):
        urllib.request.urlretrieve(url, "datasets/review_polarity.tar.gz")
    
    # if the unpacked file does not exist, unpack it
    if not os.path.exists("datasets/txt_sentoken"):
        infile = tarfile.open("datasets/review_polarity.tar.gz")
        infile.extractall(path="datasets")
        infile.close()

download_data(url)<jupyter_output><empty_output><jupyter_text>The code above will produce a folder inside "datasets", called "txt_sentoken". That folder will contain two subfolders - "pos" (containing 1000 files manually labelled as positive) and "neg" (containing 1000 files manually labelled as "negative"). Each file is a separate movie review, after minimal normalization (inserting spaces around punctuation symbols, lower-casing all words, etc).We will use Scikit-learn's `load_files` function to load these data into a format that can be input directly into other scikit-learn tools:<jupyter_code>from sklearn.datasets import load_files

# loading all files. 
movie = load_files("./datasets/txt_sentoken/", shuffle=True)
len(movie.data)
# target names ("classes") are automatically generated from subfolder names
movie.target_names
# First file seems to be about a Schwarzenegger movie. 
print(movie.data[0])
movie.target[0]<jupyter_output><empty_output><jupyter_text>Train-test splitThe data has been loaded by scikit-learn into a special data structure, which is neither a numpy array or a pandas dataframe. Nonetheless, we can use scikit-learn's `train_test_split` to split the data into the training and test parts:<jupyter_code>from sklearn.model_selection import train_test_split

docs_train, docs_test, y_train, y_test = train_test_split(movie.data, movie.target, 
                                                          test_size=0.2, random_state=7)
print(f"{len(docs_train)} train and {len(docs_test)} test instances")<jupyter_output>1600 train and 400 test instances<jupyter_text>Data transformation Create Bag-of-Words representations`CountVectorizer` is a convenient facility to create bag-of-words representations of documents into a numpy array. Setting the arguments of the constructor class, we can configure the basic linguistic preprocessing steps that should be applied to it: supplying a custom tokenizer, stopword list, ngram range, etc.<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

count_vectorizer = CountVectorizer(
    strip_accents="unicode", # convert accented chars to non-accented versions
    lowercase=True,
    tokenizer=None,        # None - use the default tokenizer
    preprocessor=None,     # None - use the default preprocessor
    stop_words="english",
    ngram_range=(1,1),     # min and max range of ngrams
    analyzer="word",       # split the document into words, rather than e.g. characters
    max_df=1.0,            # ignore words with df greater than the value (int represents count, 
                           # float represents proportion of documents)
    min_df=1               # ignore words the df lower than the value (int represents count, 
                           # float represents proportion)
)<jupyter_output><empty_output><jupyter_text>We first create an instance of `CountVectorizer` and then call its `fit_transform` method. It first "fits" on the data, i.e., extracts individual features from each document, as specified by the arguments to the constructor method (e.g., extract bigrams), and then "transforms" the data, i.e. creates numpy arrays where rows are documents, columns are features extracted from the documents, and values in the cells are counts of each feature in each document:<jupyter_code># fit and tranform using training text 
docs_train_counts = count_vectorizer.fit_transform(docs_train)
docs_train_counts.shape<jupyter_output><empty_output><jupyter_text>Thus, there 1600 rows, one for each document, and 36304 columns, one for each feature.Let's check the datatype of the data:<jupyter_code>type(docs_train_counts)<jupyter_output><empty_output><jupyter_text>Note the data is stored in a special data structure called "sparse matrix", specifically Compressed Sparse Row. Because there are many zeros in each row (in the first row, there are 139 non-zero values, out of the total of 36034), the sparse matrix stores the data more efficiently and thus is able to represent relatively large text collections, without running into memory errors.We can, however, convert a sparse matrix to a normal, "dense", numpy array, if needed:<jupyter_code>docs_train_counts[0, :].toarray()<jupyter_output><empty_output><jupyter_text>The fitted `count_vectorizer` stores also the vocabulary of the text collection on which it was fitted.<jupyter_code>count_vectorizer.vocabulary_<jupyter_output><empty_output><jupyter_text>The vocabulary is just a dictionary, where keys are original words and values are their indices in the produced matrix. The vocabulary will be used internally by the fitted vectorizer, when we transform the test data, deriving a document-by-feature matrix, which has the same columns in the same order as the training data.Let's apply the fitted vectorizer also to the test data. Note we call the `transform` method, not `fit` or `fit_transform`, because the vectorizer has been fitted already:<jupyter_code>docs_test_counts = count_vectorizer.transform(docs_test)
docs_test_counts.shape<jupyter_output><empty_output><jupyter_text>It has 400 rows (one per document), but the same number of columns as the training data, 36034.  TF-IDF weightingFurther, we can transform the observed counts into TF-IDF weights in order to reflect the importance of every word in the document. This is achieved with `TfidfTransformer`:<jupyter_code>from sklearn.feature_extraction.text import TfidfTransformer

tfidf_transformer = TfidfTransformer()<jupyter_output><empty_output><jupyter_text>First the transformer needs to be "fitted" on the training data and then the fitted transformer should be used to transform both the training and the test data:<jupyter_code># fit and transform the training set with "fit_transform()"
docs_train_tfidf = tfidf_transformer.fit_transform(docs_train_counts)
# transform test
docs_test_tfidf = tfidf_transformer.transform(docs_test_counts)<jupyter_output><empty_output><jupyter_text>Standardize featuresFit a scaler on the training set and use it to transform both the training and test sets. We'll use a `MaxAbsScaler` as it is capable of dealing with sparse matrices.<jupyter_code>from sklearn.preprocessing import MaxAbsScaler 

scaler = MaxAbsScaler()

X_train = scaler.fit_transform(docs_train_tfidf)
X_test = scaler.transform(docs_test_tfidf)
X_train<jupyter_output><empty_output><jupyter_text>Train models<jupyter_code>from sklearn.model_selection import GridSearchCV<jupyter_output><empty_output><jupyter_text>BaselineClassifying everything as "pos"*pos*:* precision: 800/1600 = 0.5* recall: 800/800 = 1.0* f-score: 2/(1/p + 1/r) = 0.66*neg*:* precision: 0/0 = 0.0* recall: 0/800 = 0.0* f-score: 2/(1/p + 1/r) = 0.0*Macro-averaged*:* precision: 0.25* recall: 0.5* **f-score: 0.33**  Decision Trees<jupyter_code>from sklearn.tree import DecisionTreeClassifier

dtree = DecisionTreeClassifier(random_state=7)

# specify the hyperparameters and their values
# 3 x 3 = 9 combinations in the grid
param_grid = {
    'max_depth': [15, 30, 50],
    'min_samples_split': [10, 20, 50],
}

# we'll use 5-fold cross-validation
grid_search = GridSearchCV(dtree, param_grid, cv=5,
                           scoring='f1_macro', 
                           return_train_score=True) 

start = time.time()
grid_search.fit(X_train, y_train)
end = time.time() - start
print(f"Took {end} seconds")
cv_results = pd.DataFrame(grid_search.cv_results_)[['params', 'mean_train_score', 'mean_test_score']]
cv_results["diff, %"] = 100*(cv_results["mean_train_score"]-cv_results["mean_test_score"]
                                                     )/cv_results["mean_train_score"]

pd.set_option('display.max_colwidth', 100)
cv_results.sort_values('mean_test_score', ascending=False)<jupyter_output><empty_output><jupyter_text>Random Forest<jupyter_code>from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(random_state=7)

# specify the hyperparameters and their values
# 4 x 3 x 3 = 36 combinations in the grid
param_grid = {
    'n_estimators': [100, 200, 500, 1000],
    'max_depth': [5, 15, 30],
    'min_samples_split': [5, 10, 20]
}

# we'll use 5-fold cross-validation
grid_search = GridSearchCV(rf, param_grid, cv=5,
                           scoring='f1_macro', 
                           return_train_score=True) 

start = time.time()
grid_search.fit(X_train, y_train)
end = time.time() - start
print(f"Took {end} seconds")
cv_results = pd.DataFrame(grid_search.cv_results_)[['params', 'mean_train_score', 'mean_test_score']]
cv_results["diff, %"] = 100*(cv_results["mean_train_score"]-cv_results["mean_test_score"]
                                                     )/cv_results["mean_train_score"]

pd.set_option('display.max_colwidth', 100)
cv_results.sort_values('mean_test_score', ascending=False)<jupyter_output><empty_output><jupyter_text>Let's view the most important features:<jupyter_code># put them into a separate variable for convenience
feature_importances = grid_search.best_estimator_.feature_importances_

# the order of the features in `feature_importances` is the same as in count_vectorizer.get_feature_names(),
# so we can "zip" the two and print the first 50 in the descending order:
for k, v in sorted(zip(feature_importances, count_vectorizer.get_feature_names_out()), reverse=True)[:50]:
    print(f"{v}: {k}")
from joblib import dump

# create a folder where all trained models will be kept
if not os.path.exists("models"):
    os.makedirs("models")
    
dump(grid_search.best_estimator_, 'models/rf-clf.joblib')<jupyter_output><empty_output><jupyter_text>Linear SVM<jupyter_code>from sklearn.svm import LinearSVC

lsvm = LinearSVC(random_state=7, max_iter=10000)

# specify the hyperparameters and their values
# 7 combinations in the grid
param_grid = {
    'C': [0.0001, 0.001, 0.01, 0.1, 1, 5, 10]
}

# we'll use 5-fold cross-validation
grid_search = GridSearchCV(lsvm, param_grid, cv=5,
                           scoring='f1_macro', 
                           return_train_score=True) 

start = time.time()
grid_search.fit(X_train, y_train)
end = time.time() - start
print(f"Took {end} seconds")
cv_results = pd.DataFrame(grid_search.cv_results_)[['params', 'mean_train_score', 'mean_test_score']]
cv_results["diff, %"] = 100*(cv_results["mean_train_score"]-cv_results["mean_test_score"]
                                                     )/cv_results["mean_train_score"]

cv_results.sort_values('mean_test_score', ascending=False)<jupyter_output><empty_output><jupyter_text>Save the best model to disk:<jupyter_code>dump(grid_search.best_estimator_, 'models/svm-linear-clf.joblib')<jupyter_output><empty_output><jupyter_text>Polynomial SVM<jupyter_code>from sklearn.svm import SVC

svm_poly = SVC(kernel="poly", degree=2, random_state=7)

# specify the hyperparameters and their values
# 5 x 4 = 20 combinations in the grid
param_grid = {
    'C': [0.01, 0.1, 1, 10, 100],
    'gamma': ["scale", 0.1, 0.5, 0.9],
}

# we'll use 5-fold cross-validation
grid_search = GridSearchCV(svm_poly, param_grid, cv=5,
                           scoring='f1_macro', 
                           return_train_score=True) 

start = time.time()
grid_search.fit(X_train, y_train)
end = time.time() - start
print(f"Took {end} seconds")
cv_results = pd.DataFrame(grid_search.cv_results_)[['params', 'mean_train_score', 'mean_test_score']]
cv_results["diff, %"] = 100*(cv_results["mean_train_score"]-cv_results["mean_test_score"]
                                                     )/cv_results["mean_train_score"]

pd.set_option('display.max_colwidth', 100)
cv_results.sort_values('mean_test_score', ascending=False)<jupyter_output><empty_output><jupyter_text>RBF kernel<jupyter_code>from sklearn.svm import SVC

svm_poly = SVC(kernel="rbf", gamma="scale", random_state=7)

# specify the hyperparameters and their values
# 5 x 4 = 20 combinations in the grid
param_grid = {
    'C': [0.01, 0.1, 1, 10, 100],
    'gamma': ["scale", 0.1, 0.5, 0.9],
}

# we'll use 5-fold cross-validation
grid_search = GridSearchCV(svm_poly, param_grid, cv=5,
                           scoring='f1_macro', 
                           return_train_score=True) 

start = time.time()
grid_search.fit(X_train, y_train)
end = time.time() - start
print(f"Took {end} seconds")
cv_results = pd.DataFrame(grid_search.cv_results_)[['params', 'mean_train_score', 'mean_test_score']]
cv_results["diff, %"] = 100*(cv_results["mean_train_score"]-cv_results["mean_test_score"]
                                                     )/cv_results["mean_train_score"]

pd.set_option('display.max_colwidth', 100)
cv_results.sort_values('mean_test_score', ascending=False)<jupyter_output><empty_output><jupyter_text>Evaluate on test<jupyter_code>from joblib import load
from sklearn.metrics import precision_recall_fscore_support

lsvm = load("models/svm-linear-clf.joblib")

yhat = lsvm.predict(X_test)

# micro-averaged precision, recall and f-score
p, r, f, s = precision_recall_fscore_support(y_test, yhat, average="macro")
print(f"Precision: {p}")
print(f"Recall: {r}")
print(f"F score: {f}")<jupyter_output>Precision: 0.8518541797611565
Recall: 0.8515978293638831
F score: 0.8517206064375876<jupyter_text>The accuracy scores on the test set turn out to be very similar to the one achieved during cross-validation.Plot a confusion matrix:<jupyter_code>from sklearn.metrics import ConfusionMatrixDisplay

ConfusionMatrixDisplay.from_predictions(y_test, yhat, labels=lsvm.classes_,
                                        xticks_rotation="vertical", normalize="true",
                                        cmap=plt.cm.Blues)<jupyter_output><empty_output><jupyter_text>Classify previously unseen reviewsSome reviews from the Rotten Tomatoes website:<jupyter_code>new_reviews = ["""Never let the audience off the hook, Hitchcock once said. 
               Ho achieves this in spades and with more than one hook.""",
               """The smartest mainstream film about class made in many 
               years, Bong Joon-ho's Parasite lays bare the lie that hard 
               work can bring anyone closer to their dreams.""",
               """[A] funny, inventive, scary film."""
              ]

# preprocess and transform
new_reviews_counts = count_vectorizer.transform(new_reviews)
new_reviews_tfidf = tfidf_transformer.transform(new_reviews_counts)
new_scaled = scaler.transform(new_reviews_tfidf)

# predict labels
predictions = lsvm.predict(new_scaled)

# print results
for review, category in zip(new_reviews, predictions):
    print(f'{review} => {movie.target_names[category]}\n')<jupyter_output>Never let the audience off the hook, Hitchcock once said. 
               Ho achieves this in spades and with more than one hook. => neg

The smartest mainstream film about class made in many 
               years, Bong Joon-ho's Parasite lays bare the lie that hard 
               work can bring anyone closer to their dreams. => pos

[A] funny, inventive, scary film. => pos
{'fraction_non_alphanumeric': 0.09516625429968016, 'fraction_numerical': 0.018888419528091244, 'mean_word_length': 4.403325725464623, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 98, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '610860', 'license': 'Apache-2.0', 'n_tokens_mistral': 5384, 'n_tokens_neox': 5034, 'n_words': 1486, 'path': 'NLP/Document-Level Sentiment Analysis.ipynb', 'repo_name': 'vpekar/bd4dm', 'size': '16571'}