<jupyter_start><jupyter_text>Text categorization with k nearest neighbors<jupyter_code>import numpy as np 
import scipy
import matplotlib.pyplot as plt
import pandas as pd
from skmultilearn.adapt import MLkNN
from skmultilearn.problem_transform import BinaryRelevance
from sklearn.neighbors import KNeighborsClassifier as kNN
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn import metrics
import sys
sys.path.append('../ThresholdFunctionLearning')    ## Append path to the ThresholdFunctionLearning directory to the interpreters
                                                   ## search path
plt.style.use('mypaper.mplstyle')
from threshold_learning import predict_test_labels_binary    ## Import the 'predict_test_labels_binary()' function from the 

pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)<jupyter_output><empty_output><jupyter_text>Load and pre-process data<jupyter_code>#import data
df = pd.read_csv("../../dataset/content_paragraphs_ready.csv", header=0)

#For the label "education", all values are 0. Thus, it is dropped from the label list. 
label_list = ["threats/impacts", "responses/actions", "severity", 
        "susceptibility", "self-efficacy", "external-efficacy", "response efficacy", 
        "public health", "economy", "political evaluation", "racial conflict", 
        "international ralations/foreign policies", "positive", "negative"] 

print("Maximum value for each label")
display(df[label_list].max(axis=0))

print("Minimum value for each label")
display(df[label_list].min(axis=0))

feature_list = list(df.columns[25:])

df = df[["para_id"] + label_list + feature_list]

df.head()<jupyter_output>Maximum value for each label<jupyter_text>For the label "education", all values are 0. Thus, it is dropped from the label list.<jupyter_code>#Remove columns with no labels and split into training and test sets. 
X = df[feature_list].to_numpy().astype(float)
Y = df[feature_list].to_numpy().astype(float)

Y_gz = Y > 0
no_labels_id = np.where(~Y_gz.any(axis=1))[0]
no_labels_id

np.random.seed(123)
df_train_features, df_test_features, df_train_labels, df_test_labels = train_test_split(df[feature_list], df[label_list], test_size = 0.2) # 80-20 train-test split
df_train_features.shape<jupyter_output><empty_output><jupyter_text>Dimensionality reductionWith a train-test split of 80-20, we have 232 training examples. It is recommended online [https://www.quora.com/When-should-I-use-K-Nearest-Neighbors-When-is-it-a-better-algorithm-than-NN-SVM-etc (better citations needed)] that kNN works for a lot of data in low-dimenional space. Thus, we need to shrink the dimension space to as low as possible. So, we try to do a PCA analysis to understand if this is possible up to $p=n$.<jupyter_code>pca = PCA(n_components=df_train_features.shape[0])
pca.fit(df_train_features[feature_list])

PC_values = np.arange(pca.n_components_) + 1

#Scree plot and the cumulative variance tax.
fig = plt.figure(figsize=(14, 6))
ax1 = fig.add_subplot(121)
ax1.plot(PC_values, pca.explained_variance_ratio_, 'ro-', linewidth=2)
ax1.set_xlabel("Principal components")
ax1.set_ylabel("Variance")

cum_var = np.cumsum(pca.explained_variance_ratio_) #cumulative variance.
target_cum_var = 0.9 #target cumulative variance.
PC_retained = np.where(cum_var>target_cum_var)[0][0] + 1 #Number of principal components retained. 

ax2 = fig.add_subplot(122)
ax2.plot(PC_values, cum_var, 'ro-', linewidth=2)
ax2.set_xlabel("Principal components")
ax2.set_ylabel("Cumulative Variance")
ax2.set_xlim()
ax2.set_ylim()
ax2.hlines(target_cum_var, ax2.get_xlim()[0], ax2.get_xlim()[1], linestyles="--", colors=plt.cm.Greys(200))
ax2.vlines(PC_retained, ax2.get_ylim()[0], ax2.get_ylim()[1], linestyles="--", colors=plt.cm.Greys(200))

print(f"Target cumulative variance: {target_cum_var:.2f}")
print(f"Number of principal components: {PC_retained}")

PC_list = [f"PC-{comp}" for comp in np.arange(1, PC_retained+1)]
#Append the data to the main dataframe and create a component dataframe. 
component_df = pd.DataFrame(pca.components_[:PC_retained],columns=feature_list, index = PC_list)
display(component_df.T)
df_train_features[PC_list] = pca.transform(df_train_features[feature_list])[:, :PC_retained]
df_test_features[PC_list] = pca.transform(df_test_features[feature_list])[:, :PC_retained]<jupyter_output>Target cumulative variance: 0.90
Number of principal components: 166<jupyter_text>We notice that 165 principal components can describe 90% variance of the data. Thus, dimensionality reduction is possible. The total feature space of 2094 can be reduced to 165 principal components while retaining 90% variance in the data.  Next, we consider the predictive power of each principal component for each label by considering the difference between the means of positive (1) and negative (0) principal component values to understand if the feature space can be shrunk further. For each principal component, we compute the maximum of this absolute value among all the labels. The largest value of this absolute difference is 3.0. In order to shrink the dimensional space as much as possible for the kNN model to work well, we consider a value below which we will we will ignore the principal components.  For example, let us consider this target value as 0.3 which is 10% of the maximum 3.0. For the principal component 150, the maximum of the absolute value of the difference between the positive and negative labels is 0.1. Since this value is lesser than 3.0, principal component 150 is ignored.<jupyter_code>#New dataframe storing the rows as the principal components and columns as the labels with values indicating the difference between the positive and negative means.
df_diff = pd.DataFrame(index=PC_list, columns=label_list)
for label in label_list:
    df_diff[label] = df_train_features.loc[df_train_labels[label]==1, PC_list].mean(axis=0).values - \
                                df_train_features.loc[df_train_labels[label]==0, PC_list].mean(axis=0).values
df_diff["mod_max_diff"] = np.abs(df_diff[label_list]).max(axis=1)

target_difference_cutoff = 0.3 
print("Number of principal components with max absolute difference between 1 and 0 label means for cutoff " + 
      f"{target_difference_cutoff} : {df_diff[df_diff['mod_max_diff']<target_difference_cutoff].shape[0]}")

separable_PC_list = df_diff.loc[df_diff['mod_max_diff']>target_difference_cutoff].index.values

#Plotting the principal components to understand if any of them are particularly useless and can be dropped.
fig = plt.figure(figsize=(14,6))
ax = fig.add_subplot(121)
ax.scatter(np.arange(1, df_diff.shape[0]+1), df_diff["mod_max_diff"])
ax.set_xlabel("Principal components")
ax.set_ylabel("Max absolute difference between 1 and 0 means")
ax.set_xlim()
ax.hlines(target_difference_cutoff, ax.get_xlim()[0], ax.get_xlim()[1], linestyles="--", color=plt.cm.Greys(200))

ax2 = fig.add_subplot(122)
ax2.hist(df_diff[df_diff["mod_max_diff"]<target_difference_cutoff]["mod_max_diff"])
ax2.set_xlabel("Max absolute difference between 0 and 1 means")
ax2.set_ylabel("Frequency")<jupyter_output>Number of principal components with max absolute difference between 1 and 0 label means for cutoff 0.3 : 103<jupyter_text>Considering a cutoff of 0.3, we find that the 103 principal components can be ignored, reducing the feature space to 62. These principal components are further considered for analysis.   Binary Relevance Classification with KNN<jupyter_code>## Implement a binary relevance model using KNN classifiers (Naive approach to be compared with ML-KNN, later)
X_train = df_train_features[separable_PC_list].values
y_train = df_train_labels[label_list].values
X_test = df_test_features[separable_PC_list].values
y_test = df_test_labels[label_list].values

br_classifier = BinaryRelevance(
    classifier = kNN()
)

br_classifier.fit(X_train, y_train)

br_train_preds = br_classifier.predict(X_train).toarray()
br_test_preds = br_classifier.predict(X_test).toarray()

print (f"The Hamming loss for the training data is {metrics.hamming_loss(y_train, br_train_preds):.3f}")
print (f"The Hamming loss for the test data is {metrics.hamming_loss(y_test, br_test_preds):.3f}")
%%capture
from sklearn.model_selection import GridSearchCV

parameters_br = {'n_neighbors': range(1,5)}  
#By default, the Hamming loss as an option is not provided in the scoring string options. So, we will make the Hamming loss funciton as a scorer and use that. 
hamming_scorer = metrics.make_scorer(metrics.hamming_loss)

clf_br = GridSearchCV(kNN(), parameters_br, scoring = hamming_scorer, cv = 5, verbose = 1)
clf_br.fit(X_train, y_train)
best_index_br = np.argmin(clf_br.cv_results_["mean_test_score"])
best_parameters_br = clf_br.cv_results_["params"][best_index_br]

df_CV_br = pd.DataFrame(columns=["Params", "Mean out-of-bag Hamming loss"])
df_CV_br["Params"] = clf_br.cv_results_["params"]
df_CV_br[ "Mean out-of-bag Hamming loss"] = clf_br.cv_results_["mean_test_score"]
display(df_CV_br)
print(f"Best parameters: {best_parameters_br}, Best Hamming loss: {np.min(clf_br.cv_results_['mean_test_score'])}")
#Threshold learning
#using the best parameters from the cross-validation with original threshold. 
classifier_best_br = BinaryRelevance(
    classifier = kNN(n_neighbors = 3)
)
classifier_best_br.fit(X_train, y_train)
y_train_pred_best_br = classifier_best_br.predict(X_train)
y_train_pred_best_array_br = y_train_pred_best_br.toarray()
y_test_pred_best_br = classifier_best_br.predict(X_test)
y_test_pred_best_array_br = y_test_pred_best_br.toarray()

print (f"Best parameters: The Hamming loss training data is {metrics.hamming_loss(y_train, y_train_pred_best_array_br):.3f}")
print (f"Best parameters: The Hamming loss test data is {metrics.hamming_loss(y_test, y_test_pred_best_array_br):.3f}")

#using the best parameters
y_train_pred_proba = classifier_best_br.predict_proba(X_train)
y_train_pred_proba_array = y_train_pred_proba.toarray()
y_test_pred_proba = classifier_best_br.predict_proba(X_test)
y_test_pred_proba_array = y_test_pred_proba.toarray()

t_range = (0, 1)

test_labels_binary, threshold_function = predict_test_labels_binary(y_train_pred_proba_array, y_train, y_test_pred_proba_array, t_range)
print (f"Best parameters with threshold function learning: Hamming loss Test set is {metrics.hamming_loss(y_test, test_labels_binary)}")<jupyter_output>Best parameters: The Hamming loss training data is 0.110
Best parameters: The Hamming loss test data is 0.202
Best parameters with threshold function learning: Hamming loss Test set is 0.2684729064039409<jupyter_text>ML-kNNWith the dimensionality reduced from 2094 to 62 components, we can now apply the ML-kNN model to understand its prediction performance.<jupyter_code>X_train = df_train_features[separable_PC_list].values
y_train = df_train_labels[label_list].values
X_test = df_test_features[separable_PC_list].values
y_test = df_test_labels[label_list].values

classifier = MLkNN(k=3)
classifier.fit(X_train, y_train)
y_train_pred = classifier.predict(X_train)
y_train_pred_array = y_train_pred.toarray()
y_test_pred = classifier.predict(X_test)
y_test_pred_array = y_test_pred.toarray()

print (f"The Hamming loss for the training data is {metrics.hamming_loss(y_train, y_train_pred_array):.3f}")
print (f"The Hamming loss for the test data is {metrics.hamming_loss(y_test, y_test_pred_array):.3f}")<jupyter_output>/Users/architdatar/anaconda3/envs/virtual_workspace/lib/python3.7/site-packages/sklearn/utils/validation.py:72: FutureWarning: Pass n_neighbors=3 as keyword args. From version 1.0 (renaming of 0.25) passing these as positional arguments will result in an error
  "will result in an error", FutureWarning)<jupyter_text>Obtaining the test error rate as 19.6% is an encouraging result. Even with k=3, the model does better than the stipid model which always predicts an error rate of 23% (as per Bobby's calculations).   Model improvementThe next steps are to understand the best values of k by performing cross-validation, learning a suitable threshold, and weighting to adjust for the skew (optional).<jupyter_code>%%capture
from sklearn.model_selection import GridSearchCV

parameters = {'k': range(1,5), 's': [1.0, 1.5, 2.5, 5.0, 10.0]}  
#By default, the Hamming loss as an option is not provided in the scoring string options. So, we will make the Hamming loss funciton as a scorer and use that. 
hamming_scorer = metrics.make_scorer(metrics.hamming_loss)

clf = GridSearchCV(MLkNN(), parameters, scoring=hamming_scorer, cv=5, verbose=1)
clf.fit(X_train, y_train)
best_index = np.argmin(clf.cv_results_["mean_test_score"])
best_parameters = clf.cv_results_["params"][best_index]

df_CV = pd.DataFrame(columns=["Params", "Mean out-of-bag Hamming loss"])
df_CV["Params"] = clf.cv_results_["params"]
df_CV[ "Mean out-of-bag Hamming loss"] = clf.cv_results_["mean_test_score"]
display(df_CV)
print(f"Best parameters: {best_paramters}, Best Hamming loss: {np.min(clf.cv_results_['mean_test_score'])}")

#We can make a plot of this later. 
#Threshold learning
#using the best parameters from the cross-validation with original threshold. 
classifier_best = MLkNN(k=3, s=2.5)
classifier_best.fit(X_train, y_train)
y_train_pred_best = classifier_best.predict(X_train)
y_train_pred_best_array = y_train_pred_best.toarray()
y_test_pred_best = classifier_best.predict(X_test)
y_test_pred_best_array = y_test_pred_best.toarray()

print (f"Best parameters: The Hamming loss training data is {metrics.hamming_loss(y_train, y_train_pred_best_array):.3f}")
print (f"Best parameters: The Hamming loss test data is {metrics.hamming_loss(y_test, y_test_pred_best_array):.3f}")

#using the best parameters
classifier_CV = MLkNN(k=3, s=2.5)
classifier_CV.fit(X_train, y_train)
y_train_pred_proba = classifier_CV.predict_proba(X_train)
y_train_pred_proba_array = y_train_pred_proba.toarray()
y_test_pred_proba = classifier_CV.predict_proba(X_test)
y_test_pred_proba_array = y_test_pred_proba.toarray()

t_range = (0, 1)

test_labels_binary, threshold_function = predict_test_labels_binary(y_train_pred_proba_array, y_train, y_test_pred_proba_array, t_range)
print (f"Best parameters with threshold function learning: Hamming loss Test set is {metrics.hamming_loss(y_test, test_labels_binary)}")<jupyter_output>/Users/architdatar/anaconda3/envs/virtual_workspace/lib/python3.7/site-packages/sklearn/utils/validation.py:72: FutureWarning: Pass n_neighbors=3 as keyword args. From version 1.0 (renaming of 0.25) passing these as positional arguments will result in an error
  "will result in an error", FutureWarning)
{'fraction_non_alphanumeric': 0.09810061951119886, 'fraction_numerical': 0.018381101504527198, 'mean_word_length': 6.514066496163683, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '182893', 'license': 'MIT', 'n_tokens_mistral': 5096, 'n_tokens_neox': 4847, 'n_words': 1305, 'path': 'codes/ML-kNN/ML-kNN_paragraph_classification.ipynb', 'repo_name': 'architdatar/NewsArticleClassification', 'size': '14689'}