<jupyter_start><jupyter_text>Naive Bayes and Logistic RegressionNaive Bayes is popular in text classification tasks and sentiment analysis.  Imports<jupyter_code>import sys
import altair as alt
alt.renderers.enable("mimetype")
alt.data_transformers.enable('data_server')

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (
    GridSearchCV,
    RandomizedSearchCV,
    cross_validate,
    train_test_split,
)
from sklearn.naive_bayes import BernoulliNB, MultinomialNB
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier<jupyter_output><empty_output><jupyter_text>1: Classifying happy moments We will use [HappyDB](https://www.kaggle.com/ritresearch/happydb) corpus which contains about 100,000 happy moments classified into 7 categories: *affection, exercise, bonding, nature, leisure, achievement, enjoy_the_moment*. The data was crowd-sourced via [Amazon Mechanical Turk](https://www.mturk.com/). The ground truth label is not available for all examples, and in this project, we'll only use the examples where ground truth is available (~15,000 examples).<jupyter_code>df = pd.read_csv("../downloads/cleaned_hm.csv", index_col=0)
sample_df = df.dropna()
sample_df.head()
sample_df = sample_df.rename(
    columns={"cleaned_hm": "moment", "ground_truth_category": "target"}
)
sample_df
train_df, test_df = train_test_split(sample_df, test_size=0.3, random_state=123)
X_train, y_train = train_df["moment"], train_df["target"]
X_test, y_test = test_df["moment"], test_df["target"]<jupyter_output><empty_output><jupyter_text>1.1 Distribution of target values<jupyter_code>alt.Chart(pd.DataFrame(y_train)).mark_bar().encode(y="target", x="count()")<jupyter_output><empty_output><jupyter_text>Achievement and Affection seem to be the target the most often and Exercise and Nature the least.   1.2 DummyClassifier We report mean cross-validation scores for `DummyClassifier`. The low mean cross-validation score can be attributed to the distribution of targets. Since `DummyClassifier` uses the most frequent target (affection in this case) as prediction, if we find its frequency we will have our CV score.<jupyter_code># from MDS notes
def mean_std_cross_val_scores(model, X_train, y_train, **kwargs):
    """
    Returns mean and std of cross validation

    Parameters
    ----------
    model :
        scikit-learn model
    X_train : numpy array or pandas DataFrame
        X in the training data
    y_train :
        y in the training data

    Returns
    ----------
        pandas Series with mean scores from cross_validation
    """

    scores = cross_validate(model, X_train, y_train, **kwargs)

    mean_scores = pd.DataFrame(scores).mean()
    std_scores = pd.DataFrame(scores).std()
    out_col = []

    for i in range(len(mean_scores)):
        out_col.append((f"%0.3f (+/- %0.3f)" % (mean_scores[i], std_scores[i])))

    return pd.Series(data=out_col, index=mean_scores.index)
mean_std_cross_val_scores(DummyClassifier(), X_train, y_train, return_train_score=True)
# We get the same score calculated by DummyClassifier this way
y_train.value_counts()["affection"] / y_train.shape[0]<jupyter_output><empty_output><jupyter_text>1.3 Different classifiers For each model in the starter code below, we carry out 5-fold cross-validation and report mean cross-validation scores, mean train scores, and mean fit and score times. > Note that in the given text, we use CountVectorizer with `binary=False` argument to extract binary features. The binary features (0/1) could be passed to `BernoulliNB`.> `alpha` is by default set to 1 to apply smoothing on the final prediction of the model (to overcome the division over zero problem).<jupyter_code>models = {
    "Decision Tree": DecisionTreeClassifier(),
    "RBF SVM": SVC(),
    "BernoulliNB": BernoulliNB(alpha=1),
    "MultinomialNB": MultinomialNB(alpha=1),
    "Logistic Regression": LogisticRegression(max_iter=2000, multi_class="ovr"),
}
results = pd.DataFrame()
for key, value in models.items():
    if key == "BernoulliNB":
        pipe = make_pipeline(CountVectorizer(stop_words='english', binary=True), value)
    else:
        pipe = make_pipeline(CountVectorizer(stop_words='english',), value)
    results[key] = mean_std_cross_val_scores(
        pipe, X_train, y_train, return_train_score=True
    )

results<jupyter_output><empty_output><jupyter_text>1.3 Discussion 1. Logistic regression has the highest CV score and the gap between CV and train score are small, showing that the model is performing well. After that are RBF SVM followed closely by Multinomial Naive Bayes both having lower CV scores, but small gaps between CV and train score. Among these Multinomial and Bernoulli Naive Bayes have the fastest fit time and score time. The very high trains score and low cross validation score for the decision tree implies overfitting.2. Our target in this case has multiple classes. This is how each model in the list above might be tackling multi-class classification.>Decision Tree: Same as binary classification, the end of each leaf could be a different class.  >RBF SVM: Through the "one-vs-one" approach for multi-class classification a total of n * (n - 1) / 2 classifiers are constructed and each one trains data from two classes.   >Naive Bayes: We can use the similar concept used for binary classification, but this time condition on multiple classes.  >Logistic Regression: Multi-class classification is done using "one-vs-rest" or "one-vs-one" methods.   1.4 Hyperparameter optimization We will:1. Define a pipeline with `CountVectorizer` and `LogisitcRegression` with `max_iter=2000` and `multi_class='ovr'`.2. Using `RandomizedSearchCV`, jointly optimize `C` and `max_features` of logistic regression.<jupyter_code>from scipy.stats import uniform, loguniform

hyper_pipe = make_pipeline(
    CountVectorizer(stop_words="english"),
    LogisticRegression(
        max_iter=2000,
        multi_class="ovr",  # one-vs-rest
    ),
)

param_distributions = {
    "countvectorizer__max_features": np.arange(5000, 10000, 100),
    "logisticregression__C": loguniform(1e-1, 1e2),
}

random_search = RandomizedSearchCV(
    hyper_pipe,
    param_distributions,
    n_iter=20,
    random_state=2021,
    return_train_score=True,
)
random_search.fit(X_train, y_train)
random_search.best_score_
random_search.best_params_
pd.DataFrame(random_search.cv_results_)[
    ["param_countvectorizer__max_features",
     "param_logisticregression__C",
     "mean_test_score",
     "mean_train_score",
     "rank_test_score"]
].sort_values("rank_test_score")<jupyter_output><empty_output><jupyter_text>We are getting similar average cross validation score compared to the default values.The best model has slightly higher score compared to the next couple of scores."   2: Model interpretation One of the primary advantage of linear models is their ability to interpret models in terms of important features by exploring the weights learned by the logistic regression classifier.   2.1 Most informative wordsWe will use the best estimator from 3.7 to get    - first 5 words that are positively associated and first 5 words which are negatively associated with the class "affection".     - first 5 words that are positively associated and first 5 words which are negatively associated with the class "exercise". > This information is exposed by the `coef_` attribute of [LogisticRegression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) object. For multiclass classification, for each class the models learns coefficients per feature for that class. > The vocabulary (mapping from feature indices to actual words) is obtained by calling `get_feature_names_out()` on the `CountVectorizer` object.<jupyter_code>vocabs = random_search.best_estimator_["countvectorizer"].get_feature_names_out()
coeffs = random_search.best_estimator_["logisticregression"].coef_
targets = random_search.classes_
random_search_df = pd.DataFrame(coeffs, columns=vocabs).set_index(targets).T

pos_affection = (
    random_search_df.sort_values("affection", ascending=False).head().index.tolist()
)
neg_affection = random_search_df.sort_values("affection").head().index.tolist()

pos_exercise = (
    random_search_df.sort_values("exercise", ascending=False).head().index.tolist()
)
neg_exercise = random_search_df.sort_values("exercise").head().index.tolist()

print("Positively associated with 'affection':")
print("   ", ", ".join(pos_affection))
print("Negatively associated with 'affection':")
print("   ", ", ".join(neg_affection))

print("Positively associated with 'exercise':")
print("   ", ", ".join(pos_exercise))
print("Negatively associated with 'exercise':")
print("   ", ", ".join(neg_exercise))<jupyter_output>Positively associated with 'affection':
    husband, wife, brother, son, daughter
Negatively associated with 'affection':
    friend, unforgettable, exam, friends, police
Positively associated with 'exercise':
    gym, workout, exercise, yoga, exercised
Negatively associated with 'exercise':
    dog, daughter, just, got, movie<jupyter_text>2.2 Interpretation - The positively associated words for affection and exercise make sense. However, it is hard to justify the negative words. It is very helpful to see what features our model is using to predict. For example, we have both `friend` and `friends` in our negatively associated affection list. Knowing this we would be able to do some further data transformation to minimize this nuisance and improve performance. The other thing that we could do by looking at these features is whether the features with highest weights really make sense.  - It would be very hard to get the most informative features in other classifier. We could say that the top most point of a decision tree (root) is the most important feature, but that is not very informative in most of the cases. For SVM the most important features are multiple observations (arrays) which do not give any intuitive information. For Naive Bayes models, it would be very hard to find the features that contribute the most to the model especially if we have a multiclass target.   3: Test score and final evaluationLet's examine the performance of our best model on the test set.   3.1 Evaluation on test data<jupyter_code>best_maxfeature = list(random_search.best_params_.values())[0]
best_C = list(random_search.best_params_.values())[1]
optimized_pipe = make_pipeline(
    CountVectorizer(
        stop_words="english", 
        max_features=best_maxfeature
    ),
    LogisticRegression(
        C=best_C,
        max_iter=2000,
        multi_class="ovr",
    ),
)

optimized_pipe.fit(X_train, y_train)
optimized_pipe.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>The test score is very close to the cross validation score. Considering the large amount of data that has been used to train the model, we are confident that this model will be able to perform well for the population that the information was collected from.   3.2 Evaluation using probability scoresUsing this model, we will try to find the moments in the test set: 1. where the model is most confident that the moment belongs to class "achievement" (i.e., an example with highest predicted probability for class "achievement")2. where the model is most confident that the moment belongs to class "nature" (i.e., an example with the highest predicted probability of being "nature")In each case, print out the moment and the associated probability score.<jupyter_code>prob_df = pd.DataFrame(random_search.predict_proba(X_test), columns=targets)

max_achiev = prob_df["achievement"].idxmax()
max_achiev_prob = prob_df["achievement"].max()

print("Most confident for class 'achievement':")
print("Prob score:", max_achiev_prob)
X_test.iloc[max_achiev]
max_nature = prob_df["nature"].idxmax()
max_nature_prob = prob_df["nature"].max()
print("Most confident for class 'nature':")
print("Prob score:", max_nature_prob)
X_test.iloc[max_nature]<jupyter_output>Most confident for class 'nature':
Prob score: 0.9475748027062482<jupyter_text>3.3 Fake moments We can test the best model on some fake moments.<jupyter_code>test_moments = [
    "On the weekend, I spent some quality time with my best friend.",
    "I love my kids!",
    "I went for a hike in the forest.",
    "I did yoga this morning.",
    "I am still breathing and I am alive!",
    "I am going to sleep",
]
optimized_pipe.predict(test_moments)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08593627374038612, 'fraction_numerical': 0.009653115680426934, 'mean_word_length': 5.188926663428849, 'pattern_counts': {'":': 10, '<': 39, '<?xml version=': 0, '>': 47, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '348793', 'license': 'MIT', 'n_tokens_mistral': 3794, 'n_tokens_neox': 3608, 'n_words': 1314, 'path': '4.HappyDB_naive_bayes_logistic_reg/naive_bayes_logistic_reg.ipynb', 'repo_name': 'artanzand/Kaggle_projects', 'size': '12742'}