<jupyter_start><jupyter_text>Cosine Similarity FeatureThis model will add a cosine similarity feature in addition to NMF features. The previous model showed some misclassifed pairs had a very high cosine similarity.**Pipeline**1. Stack questions2. Lemmatize questions3. Tfidf4. NMF5. Unstack questions6. Add cosine similarity7. XGBClassifier<jupyter_code># data manipulation
import utils
import pandas as pd
import numpy as np

# modeling
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF
from sklearn.preprocessing import FunctionTransformer
from sklearn.model_selection import cross_validate, StratifiedKFold

from xgboost import XGBClassifier
X_train = utils.load('X_train')
y_train = utils.load('y_train')
try:
    X_train_lemma = utils.load('X_train_lemma') 
except:
    pipe_cos_sim = Pipeline(
        [
            ('stack', FunctionTransformer(utils.stack_questions, validate=False)),
            ('lemma', FunctionTransformer(utils.cleanup_text, validate=False)),
        ]
    )

    X_train_lemma = pipe_cos_sim.transform(train_df)
    utils.save(train_lemma, 'X_train_lemma') # save as it can take 10 minutes to lemmatize the entire corpus
pipe_transform = Pipeline(
    [
        ('tfidf', TfidfVectorizer()),
        ('nmf', NMF(n_components=5)),
        ('unstack', FunctionTransformer(utils.unstack_questions, validate=True)),
        ('cos_sim', FunctionTransformer(utils.calc_cos_sim, validate=True))
    ]
)

X_train_transform = pipe_transform.fit_transform(X_train_lemma)
skf = StratifiedKFold(n_splits=3, random_state=42)
cv = cross_validate(XGBClassifier(n_estimators=500, random_state=42, n_jobs=-1), 
               X_train_transform, 
               y_train, 
               cv=skf, 
               n_jobs=-1, 
               scoring=('accuracy', 'precision', 'recall', 'f1', 'roc_auc', 'neg_log_loss'))
results_df = utils.load('results')

results_df = results_df.drop(index='cos_sim_model', errors='ignore')
results_df = results_df.append(utils.log_scores(cv, 'cos_sim_model'))
results_df
utils.save(results_df, 'results')<jupyter_output><empty_output><jupyter_text>ResultsAdding the cosine similarity metric to the model made a marginal improvement in the training statistics, and is possibly a good candidate to hyper tune via cross validation. Let's now take a look at where the classifier was wrong.<jupyter_code>xgb = XGBClassifier(n_estimators=500, random_state=42, n_jobs=-1)
xgb.fit(X_train_transform, y_train)
y_probs = xgb.predict_proba(X_train_transform)[:, 1]
class_errors_df = utils.ground_truth_analysis(y_train, y_probs)
class_errors_df.head()
X_train_transform.shape<jupyter_output><empty_output><jupyter_text>Top false negative examples<jupyter_code>fn_idx = class_errors_df.sort_values('diff', ascending = False).head().index
for idx in fn_idx:
    print('Prob:', y_probs[idx])
    print()
    print(X_train.iloc[idx].question1)
    print(X_train.iloc[idx].question2)
    print()
    print('Lemma-------')
    print(X_train_lemma[idx*2])
    print(X_train_lemma[idx*2+1])
    print()
    print('Cos sim------')
    print(X_train_transform[idx, -1])
    print()
    print('Vecs-----------')
    print(X_train_transform[idx, :5])
    print(X_train_transform[idx, 5:-1])
    print('-------------------------------------------')
    print()<jupyter_output>Prob: 0.023252033

How can I get unlimited Ola Credits? Please help. I know there's a hack for that.
What is the best Ola hack to get unlimited Ola Credits?

Lemma-------
good ola hack unlimited ola credit
unlimited ola credit help know hack

Cos sim------
0.0015347087514404256

Vecs-----------
[0.         0.         0.00169648 0.01552974 0.00111447]
[0.01473608 0.00013005 0.         0.         0.00031791]
-------------------------------------------

Prob: 0.024024863

What is the best Ola hack to get unlimited Ola Credits?
How can I get unlimited Ola credits? I know there's a hack for that.

Lemma-------
unlimited ola credit know hack
good ola hack unlimited ola credit

Cos sim------
0.001481936862938256

Vecs-----------
[0.01473608 0.00013005 0.         0.         0.00031791]
[0.         0.         0.00018338 0.01586184 0.00109255]
-------------------------------------------

Prob: 0.03743049

How do I train myself to build mental visualisation skills like Nikola Tesla?
How do I trai[...]<jupyter_text>Top false positive examples<jupyter_code>fp_idx = class_errors_df.sort_values('diff').head().index
for idx in fp_idx:
    print('Prob:', y_probs[idx])
    print()
    print(X_train.iloc[idx].question1)
    print(X_train.iloc[idx].question2)
    print()
    print('Lemma-------')
    print(X_train_lemma[idx*2])
    print(X_train_lemma[idx*2+1])
    print()
    print('Cos sim------')
    print(X_train_transform[idx, -1])
    print()
    print('Vecs-----------')
    print(X_train_transform[idx, :5])
    print(X_train_transform[idx, 5:-1])
    print('-------------------------------------------')
    print()<jupyter_output>Can I make money online?
I am willing to work hard for money but how I can make money?
Prob: 0.96103376

Lemma-------
willing work hard money money
money online

Cos sim------
0.9993590913043952

Vecs-----------
[0.        0.        0.1067102 0.        0.       ]
[0.         0.         0.06381616 0.00228588 0.        ]
-------------------------------------------

How can you earn money from Quora as a user?
How does Quora make money?
Prob: 0.95245737

Lemma-------
quora money
earn money quora user

Cos sim------
0.9873872317045038

Vecs-----------
[0.         0.04005779 0.06285799 0.         0.        ]
[0.         0.06401524 0.07205664 0.         0.        ]
-------------------------------------------

How do I earn money through Quora?
How can I use Quora to make money?
Prob: 0.9491555

Lemma-------
earn money quora
use quora money

Cos sim------
0.9810484046375674

Vecs-----------
[0.         0.05729245 0.06246081 0.         0.00039522]
[0.         0.04775823 0.07836525 0.         0[...]
{'fraction_non_alphanumeric': 0.16279840848806365, 'fraction_numerical': 0.07161803713527852, 'mean_word_length': 2.694427434170239, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '214120', 'license': 'MIT', 'n_tokens_mistral': 2258, 'n_tokens_neox': 1969, 'n_words': 510, 'path': 'py_files/05. Cos Similarity Feature Model.ipynb', 'repo_name': 'jason-sa/question_intent_classification', 'size': '6032'}