<jupyter_start><jupyter_text>For now we want to have a glance at a baseline model, so we are not using nothing besides the comment_text with TF-IDF and a Logistic Regression with no tunning- Not using the features created in 'EDA.ipynb'<jupyter_code>def import_train_data():
    Xtrain = pd.read_csv('../.data/train_new_features.csv',index_col='id')
    Xtr = Xtrain[['comment_text']]
    ytr = Xtrain[['toxic', 'severe_toxic', 'obscene', 'threat', 'insult','identity_hate']]   
    return Xtr,ytr

def feature_extraction(Xtr,n_grams=(1,1),max_features=10000):
    word_vectorizer = TfidfVectorizer(
        sublinear_tf=True,
        use_idf=True,
        strip_accents='unicode',
        stop_words='english',
        min_df=20,
        ngram_range=n_grams,
        max_features=max_features)
    
    start = time.time()
    print('Extracting fetures...')
    train_features = word_vectorizer.fit_transform(Xtr['comment_text'])
    print(f'Time elapsed: {time.time() - start} seconds. ')
    param = word_vectorizer.get_params()
    return train_features,param

def train_model(Xtr,ytr,df_results,param_extractor,c_weight='balanced'):
    
    clf = LogisticRegression(class_weight=c_weight) #'balanced'
    param_classifier = clf.get_params()
    score_df = pd.DataFrame(columns=ytr.columns)
    scores = []
    
    for col in ytr.columns:
        cv_score = np.round(np.mean(cross_val_score(clf,
                                           Xtr,
                                           ytr[col],
                                           cv=3,
                                           scoring='roc_auc')),4)
        scores.append(cv_score)
        print(f'CV score for label {col} is {cv_score}.')
        
    scores.append(param_extractor)
    scores.append(param_classifier)
    df_results = df_results.append(dict(zip(df_results.columns,scores)), ignore_index = True)
    
    return df_results
x_train, y_train = import_train_data()
list_cols = y_train.columns.to_list()
list_cols.append('parameters_extraction')
list_cols.append('parameters_classifier')
score_df = pd.DataFrame(columns=list_cols)
#x_train_transformed,parameters_feat_ext = feature_extraction(x_train)<jupyter_output><empty_output><jupyter_text>Test different class weights<jupyter_code>weights = [{0:1,1:1},{0:1,1:10},{0:1,1:100}]
for w in weights:
    score_df = train_model(x_train_transformed,y_train,
                           score_df,parameters_feat_ext,c_weight=w)
    
display(score_df)<jupyter_output><empty_output><jupyter_text>Conclusion: Attributing different class_weights does not seem to be improving our scores so far.  Search n_gram_range<jupyter_code>list_cols = y_train.columns.to_list()
list_cols.append('parameters_extraction')
list_cols.append('parameters_classifier')
score_df = pd.DataFrame(columns=list_cols)

for n_gram_range in [(1,2),(1,3),(1,4),(2,3)]:
    x_train_transformed,parameters_feat_ext = feature_extraction(x_train,n_gram_range)
    score_df = train_model(x_train_transformed,y_train,
                           score_df,parameters_feat_ext)
display(score_df)<jupyter_output><empty_output><jupyter_text>Conclusion: We found no evidence that increasing n_gram range could improve our scores.We could add more to max_features instead.  Search max_features<jupyter_code>list_cols = y_train.columns.to_list()
list_cols.append('parameters_extraction')
list_cols.append('parameters_classifier')
score_df = pd.DataFrame(columns=list_cols)

for max_feats in [1.5e4,3e4,5e4]:
    print(max_feats)
    x_train_transformed,parameters_feat_ext = feature_extraction(x_train,
                                                                 max_features=int(max_feats))
    score_df = train_model(x_train_transformed,y_train,
                           score_df,parameters_feat_ext)
display(score_df)<jupyter_output><empty_output><jupyter_text>Conclusion: Now we seem to reach something, but the model does not seem to show any improvement with more than 15k terms.We could try now work with both max_features and n_gram_range.  Search max_features and n_gram_range<jupyter_code>list_cols = y_train.columns.to_list()
list_cols.append('parameters_extraction')
list_cols.append('parameters_classifier')
score_df = pd.DataFrame(columns=list_cols)
# 3e4 and (1,2)
for max_feats in [1.5e4,3e4]:
    for n_gram_range in [(1,2),(1,3)]:
    
        x_train_transformed,parameters_feat_ext = feature_extraction(x_train,
                                                                     n_gram_range,
                                                                     int(max_feats))
        score_df = train_model(x_train_transformed,y_train,
                               score_df,parameters_feat_ext)
display(score_df)<jupyter_output><empty_output><jupyter_text>Conclusion: Found a peak at max_feats = 3e4, n_gram_range = (1,2), row 2. We will keep these hyperparameters for now.  Increase feature space with EDA notebook features.<jupyter_code>from sklearn.preprocessing import MinMaxScaler
from sklearn.tree import DecisionTreeClassifier


x_train, y_train = import_train_data()
list_cols = y_train.columns.to_list()
list_cols.append('parameters_extraction')
list_cols.append('parameters_classifier')
score_df = pd.DataFrame(columns=list_cols)

max_feats = 1.5e4
n_gram_range = (1,2)

x_train_vectorized,parameters_feat_ext = feature_extraction(x_train,
                                                             n_gram_range,
                                                             int(max_feats))

# text statistics from EDA notebook
text_statistics_features = text_stats_features(x_train)
norm_text_stats_feats = MinMaxScaler().fit_transform(text_statistics_features)
x_train_features = sp.hstack([x_train_vectorized,norm_text_stats_feats],format='csr')
score_df = train_model(x_train_features,y_train,
                               score_df,parameters_feat_ext)

#display(score_df)<jupyter_output>CV score for label toxic is 0.9697.
CV score for label severe_toxic is 0.9855.
CV score for label obscene is 0.9829.
CV score for label threat is 0.9792.
CV score for label insult is 0.975.
CV score for label identity_hate is 0.972.<jupyter_text>Conclusion: Statistics from the text concerning word_counts, marks such as question and exclamation ones, as well as symbols and smiles improved our CV score for most labels. Only _insult_ that presented a decrement in its CV score.  Learning Curves : Diagnostics<jupyter_code>clf1 = LogisticRegression(class_weight='balanced') #'balanced'
plot_curve(clf1, x_train_features, y_train)
clf1 = LogisticRegression(class_weight='balanced') #'balanced'
clf2 = DecisionTreeClassifier(max_depth=4)
clf3 = MultinomialNB()
clf = VotingClassifier(
    estimators=[('lr', clf1),
                ('dt', clf2),
                ('mnb', clf3)], voting='soft')
    
plot_curve(clf, x_train_features, y_train)<jupyter_output><empty_output><jupyter_text>In Threat more than any other the model seems to be overfitting, we could think of a stronger regularization for it.In half of the labels we do have more than 2 points differences between mean training and validation score. We can focus on mitigating those learning issues. Decisions:    Adding more training instances. [Not that feasible]    Adding more features. [Possible]    Feature selection. [Possible]    Hyperparameter optimization. [Possible] Notes : voting 'soft' requires classifiers that have their output probabilities well calibrated. Conclusion: Although not presenting the worst score the 'threat' label presents a large standard deviation from the mean for the 5-fold CV score. As well the gap between mean train and test scores is the largest across labels.We could interpret it as the concept of 'threat' to be learned is not as easily generalizable as the rest of the labels. Another aspect we could question is the labeling process, what if the definition of 'threat' is not homogeneous across the group of people labelling the sentences? Not to be forgotten, we are dealing with an imbalanced problem and that we can verify by ourselves based on the data distributions we have.<jupyter_code>y_train.apply(pd.value_counts)<jupyter_output><empty_output><jupyter_text>Conclusion: 'Threat' label in fact presents the least ammount of examples, almost a third of the second most imbalanced label. When we split it in train and test in fact would make sense that it fitted well the train set while presenting large variance in the test set.   Let's check the calibration curve first<jupyter_code># Calibrating:
from sklearn.calibration import calibration_curve

# not working yet
fraction_of_positives, mean_predicted_value = \
            calibration_curve(y_test, prob_pos, n_bins=10)

plt.plot(mean_predicted_value, fraction_of_positives, "s-",
                 label="%s (%1.3f)" % (name, clf_score))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08280614550384094, 'fraction_numerical': 0.012313601446000903, 'mean_word_length': 3.808799565453558, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '285568', 'license': 'Apache-2.0', 'n_tokens_mistral': 2755, 'n_tokens_neox': 2632, 'n_words': 687, 'path': 'toxic_comment_challenge/Notebooks/2.0_Modeling.ipynb', 'repo_name': 'prbronze/Kaggle', 'size': '8852'}