<jupyter_start><jupyter_text>Improving M000 (log loss: 0.320) by changing MultinomialNB hyperparameters / vectorization<jupyter_code>import sys
sys.path.append('..')

from __future__ import division, print_function 
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, TfidfTransformer
from sklearn.naive_bayes import BernoulliNB, MultinomialNB
from sklearn.multiclass import OneVsRestClassifier
from sklearn.feature_selection import chi2, mutual_info_classif, SelectKBest
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import binarize 
from sklearn.metrics import log_loss
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from collections import defaultdict 
%matplotlib inline
%load_ext autoreload
%autoreload 2

from evaluation import cross_validate_multilabel, multilabel_results<jupyter_output>The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload<jupyter_text>Define classes and load test/train data, some input text is "N/A" so turn na_filter off to prevent this being converted to NaN<jupyter_code>toxic_classes = [
    'toxic', 'severe_toxic', 'obscene', 
    'threat', 'insult', 'identity_hate' 
]

df = pd.read_csv('../data/train.csv', na_filter=False)
X_text = df['comment_text'].values
Y = df[toxic_classes].values
ids = df['id']<jupyter_output><empty_output><jupyter_text>[1] Feature reductionTo improve our model accuracy we will first try reducing the number of features.Each feature is a single token (word) found within the training data, the feature value is the count of that token within the  document record, i.e. how many times a word appeared in a particular commentThe majority of these words will not be useful in making predictions, and whilst these non-discriminative features should affect each class equally, their inclusion can still result in negative performance. [1a] Drop low frequency tokensFirst we will remove words that only appear in a small number of documents, as their class conditional probabilities will be less reliable.Plotting histogram showing the token frequency distribution (x-axis: number of documents that token appears in, y-axis: number of tokens with given document frequency)<jupyter_code>X = CountVectorizer(binary=True).fit_transform(X_text) # binarize features for easy histogram
document_frequency = np.asarray(X.sum(axis=0)).flatten()
plt.hist(document_frequency, bins=1000)
plt.yscale('log') 
plt.show()<jupyter_output><empty_output><jupyter_text>It is quite clear that the mast majority of features only appear in a small number of documents (~1) so we will model performance when setting a minimum document frequency (expressed as an integer document count).<jupyter_code>results = []
min_dfs = [1, 10, 100, 1000, 5000, 10000, 20000, 30000, 40000, 50000, 60000]

for min_df in min_dfs:
    cvec = CountVectorizer(min_df=min_df)
    X = cvec.fit_transform(X_text)
    cv_scores = cross_validate_multilabel(MultinomialNB(), X, Y, cv=10, scoring='neg_log_loss') 
    results.append(list(cv_scores))
    print('min_df:', min_df, '|', 'number of features:', len(cvec.vocabulary_))
    
multilabel_results(results, toxic_classes, min_dfs)<jupyter_output>min_df: 1 | number of features: 139171
min_df: 10 | number of features: 16887
min_df: 100 | number of features: 3806
min_df: 1000 | number of features: 584
min_df: 5000 | number of features: 114
min_df: 10000 | number of features: 59
min_df: 20000 | number of features: 21
min_df: 30000 | number of features: 12
min_df: 40000 | number of features: 7
min_df: 50000 | number of features: 2
min_df: 60000 | number of features: 1<jupyter_text>Suprisingly, our best result comes from the model which has features with a minumum document frequent of 60,000. In this model we only have one feature...<jupyter_code>cvec = CountVectorizer(min_df=60000)
X = cvec.fit_transform(X_text) 
cvec.vocabulary_<jupyter_output><empty_output><jupyter_text>The token 'the' is unlikely to discrimate between classes, as it is so common. The most likely explanation here is that we are simply getting a "null" model which has a probability equal to the class prior: the likelihood of observing an example  in a class (class marginal probability)In other words for models with only very frequent, in the Bayes numerator `P(x|y)P(y)` then `P(x|y)` will be `~P(x)` (essentially independant of class) and the so `P(y)` will become the dominant term.Our single feature model goes one step further and `P(x|y)` = 1 for both classes (sum of feature / sum of all features = 1) and so we should get a constant probability for all records (equal to the class prior)<jupyter_code>mnb = MultinomialNB()
mnb.fit(X, Y[:, 0])
probs = mnb.predict_proba(X)[:,1]
probs.min() == probs.max()<jupyter_output><empty_output><jupyter_text>Checking the proportion of records which have each class attribute<jupyter_code>Y.sum(axis=0) / Y.shape[0]<jupyter_output><empty_output><jupyter_text>So we are dealing with very imbalanced classes, and in our class specific (one vs rest) models the class prior for the negative case will always dominate. We are essentially just prediction "not toxic"" for every record and getting the best result because "not toxic" and getting a *good* result because that is the most likely class.This model be our new baseline (log_loss: **0.141**)  [1b] Select "best" featuresRather than dropping features which have a low rate of occurence, we will try to select the features which will best discriminate between classes prior to fitting the model.We use a chi2 test for independance which compares the joint probability `P(x|y)` with `P(x)` and ranks the features which have the highest different between the conditional and marginal probabilities.For example if:* `P('idiot')` = 0.1 and `P('idiot'|'toxic')` = 0.6* `P('dog')` = 0.01 and `P('idiot'|'toxic')` = 0.15Then **idiot** will have a larger chi2 value.(The actual chi2 calculation actually considers an average across all combinations of `P('idiot'|'toxic'), P('idiot'|'not_toxic'), P('not_idiot'|'toxic'), P('not_idiot'|'not_toxic')`)<jupyter_code>def kbest_feature_names(class_labels, kbest_per_class, cvec):
    # finding names of top kbest features
    kbest_features = defaultdict(list) 
    for toxic_class, kbest in zip(class_labels, kbest_per_class):
        indices = kbest.get_support(indices=True)
        scores = kbest.scores_
        sorted_indices = [index for index, score in sorted(zip(indices, scores), key=lambda x: x[1])]  
        for index in sorted_indices:
            kbest_features[toxic_class].append(cvec.get_feature_names()[index])   
            
    return pd.DataFrame(kbest_features) 
cvec = CountVectorizer()
X = cvec.fit_transform(X_text)
kbest_per_class = [SelectKBest(chi2, k=15).fit(X, y) for y in Y.T]<jupyter_output><empty_output><jupyter_text>Before fitting any models here are the top features per class according to chi2 value<jupyter_code>kbest_feature_names(toxic_classes, kbest_per_class, cvec)<jupyter_output><empty_output><jupyter_text>Most of these results seem logical and are tokens you would normally associate with toxic comments, however we are getting a few spurious results likely caused by low frequency tokens so we will rerun using a small cut-off for rare tokens.<jupyter_code>cvec = CountVectorizer(min_df=10)
X = cvec.fit_transform(X_text)
kbest_per_class = [SelectKBest(chi2, k=15).fit(X, y) for y in Y.T]
kbest_feature_names(toxic_classes, kbest_per_class, cvec)<jupyter_output><empty_output><jupyter_text>These results look more robust. Whilst we see some of the same tokens being the best feature for multiple classes, there does seem to be a general theme in most cases. For example most of the top features in "identity_hate" target ethnicity / orientation etc. Will now try fitting models with only k best features retained, note that chi2 will be calculated within the cross-validation folds to avoid information from the full set leaking into the model.<jupyter_code>def kbest_pipeline(k, classifier):
    return Pipeline([
        ('kbest', SelectKBest(chi2, k=k)),
        ('classifier', classifier) 
    ])
results = []
kbest = [1,5,10,20,30,40,50,60,70,80,90,100]
cvec = CountVectorizer(min_df=10)
X = cvec.fit_transform(X_text)

for k in kbest:
    cv_scores = cross_validate_multilabel(kbest_pipeline(k, MultinomialNB()), X, Y, cv=10, scoring='neg_log_loss') 
    results.append(list(cv_scores))
    print('kbest:', k)
multilabel_results(results, toxic_classes, kbest)<jupyter_output><empty_output><jupyter_text>Whilst selecting top features gave us a minor improvement (k=30) the performance appears to still impacted by the class imbalance. This is most noticeable in the classes which have the smallest proportion of positive samples (e.g. identity hate) we are actually getting a worse result by adding token features to the model. This is because the class prior is most influential in these cases and will still achieve the best result by essentially assuming all cases are negative.Will try re-running the last test with uniform class priors, although will likely get a worse result as we assuming the marginal `P(yi)` = `P(not_yi)` (n.b. Using larger k values as we no longer have reliable prior)<jupyter_code>results = []
kbest = [50,100,200,300,500,750,1000]
cvec = CountVectorizer(min_df=10)
X = cvec.fit_transform(X_text)

for k in kbest:
    cv_scores = cross_validate_multilabel(kbest_pipeline(k, MultinomialNB(fit_prior=False)), X, Y, cv=10, scoring='neg_log_loss') 
    results.append(list(cv_scores))
    print('kbest:', k)
multilabel_results(results, toxic_classes, kbest)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0849221890137071, 'fraction_numerical': 0.024425435432340513, 'mean_word_length': 5.7248787248787245, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '123348', 'license': 'MIT', 'n_tokens_mistral': 3159, 'n_tokens_neox': 2908, 'n_words': 1089, 'path': 'notebooks/d001_naive_bayes_improved.ipynb', 'repo_name': 'david-macleod/kaggle-toxic-comments', 'size': '9703'}