<jupyter_start><jupyter_text>**Preprocessing and Model evaluation**

---

  **Some Note-worthy Points**
*   The features are mostly PC components(28 out of 30) which we an not evaluate anything.
*   We found out in EDA that the dataset is highly imbalanced, so using accuracy as evaluation metric is not a clever idea, the performance metrics should be Confusion Matrix and PR(precision-recall) curve, or maybe Area under PR curve if a single-number metric is needed.<jupyter_code># Basic Libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from google.colab import drive
drive.mount('/content/drive', force_remount = True)
df = pd.read_csv("/content/drive/MyDrive/Datasets/creditcard.csv") 
df.head()<jupyter_output><empty_output><jupyter_text>**Data Split** Test Train and Validate data set splitting

Two classes of fruadulent and not being fraudulent are really imbalanced<jupyter_code>#test-train and scaling data set
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score

x = df.drop('Class', axis = 1)
y = df['Class']
scaler = StandardScaler()
scaled_x = scaler.fit_transform(x)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.25, stratify=y,random_state=7)
x_train, x_validate, y_train, y_validate = train_test_split(x_train, y_train, test_size=0.333, stratify=y_train,random_state=7)
x_train.head()
y_train.head()
x_validate.head()
from imblearn.over_sampling import SMOTE
sm = SMOTE(random_state=4001,kind='regular')
X_res, y_res = sm.fit_sample(x_train, y_train)
pd.value_counts(y_res)<jupyter_output>/usr/local/lib/python3.6/dist-packages/sklearn/externals/six.py:31: FutureWarning: The module is deprecated in version 0.21 and will be removed in version 0.23 since we've dropped support for Python 2.7. Please rely on the official version of six (https://pypi.org/project/six/).
  "(https://pypi.org/project/six/).", FutureWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.neighbors.base module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.neighbors. Anything that cannot be imported from sklearn.neighbors is now part of the private API.
  warnings.warn(message, FutureWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
  warnings.warn(msg, category=FutureWarning)<jupyter_text>Prediction and Evaluation of Logistic Regression
 **We will use PR AUC over ROC AUC cause PR does not account for true negatives therefore more suited for imbalanced classification**  Original<jupyter_code>from sklearn.linear_model import LogisticRegressionCV
logCV = LogisticRegressionCV(Cs=[0.01,0.1,1,10,100], scoring='average_precision',verbose=0,max_iter = 700, random_state=70)
logCV.fit(x_train,y_train)
print("The best parameter C is",logCV.C_[0])
from sklearn.metrics import average_precision_score, precision_recall_curve
pred_prob = logCV.predict_proba(x_validate)
y_score = pred_prob[:,1]
average_precision = average_precision_score(y_validate, y_score)
original_precision, original_recall, original_thresholds = precision_recall_curve(y_validate, y_score)

#Plot
sns.lineplot(original_recall, original_precision, color='red', alpha=0.5,linewidth=2,label='Original logistic')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall curve')
plt.legend(loc='upper left', bbox_to_anchor=(1,1), fontsize = 'large')<jupyter_output>/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>Above lineplot shows how our model performerd and we can choose an optimised cutoff for best precisionand recall rate.  Resampled<jupyter_code>from sklearn.linear_model import LogisticRegression
log_res = LogisticRegression(C=0.1, verbose=0, random_state=70)
log_res.fit(X_res,y_res)
predict_prob = log_res.predict_proba(x_validate)
resampled_score = predict_prob[:,1]
average_precision = average_precision_score(y_validate, y_score)
resampled_precision, resampled_recall, resampled_thresholds = precision_recall_curve(y_validate, resampled_score)

#Plot
sns.lineplot(original_recall, original_precision, color='red', alpha=0.5,linewidth=2.5,label='Original logistic')
sns.lineplot(resampled_recall, resampled_precision, color='green', alpha=0.5,linewidth=2.5,label='Resampled logistic')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall curve')
plt.legend(loc='upper left', bbox_to_anchor=(1,1), fontsize = 'large')<jupyter_output>/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>Prediction and Evaluation Random Forest  Resampled<jupyter_code>from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(max_depth=4, n_estimators=150, bootstrap=True, random_state=70)
rf.fit(X_res,y_res)
pred_prob = rf.predict_proba(x_validate)
rf_score = pred_prob[:,1]
average_precision = average_precision_score(y_validate, y_score)
rf_precision, rf_recall, rf_thresholds = precision_recall_curve(y_validate, rf_score)

#Plot
sns.lineplot(original_recall, original_precision, color='red', alpha=0.5,linewidth=2.5,label='Original logistic')
sns.lineplot(resampled_recall, resampled_precision, color='green', alpha=0.5,linewidth=2.5,label='Resampled logistic')
sns.lineplot(rf_recall, rf_precision, color='violet', alpha=0.5,linewidth=3,label='Resampled random forest')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Random Forest Precision-Recall curve')
plt.legend(loc='upper left', bbox_to_anchor=(1,1), fontsize = 'large')<jupyter_output>/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>Original<jupyter_code>from sklearn.ensemble import RandomForestClassifier
rf1 = RandomForestClassifier(max_depth=4, n_estimators=150, bootstrap=True, random_state=70)
rf1.fit(x_train,y_train)
pred_prob = rf1.predict_proba(x_validate)
rf1_score = pred_prob[:,1]
average_precision = average_precision_score(y_validate, y_score)
rf1_precision, rf1_recall, rf1_thresholds = precision_recall_curve(y_validate, rf1_score)

#Plot
sns.lineplot(original_recall, original_precision, color='red', alpha=0.5,linewidth=2.5,label='Original logistic')
sns.lineplot(resampled_recall, resampled_precision, color='green', alpha=0.5,linewidth=2.5,label='Resampled logistic')
sns.lineplot(rf1_recall, rf1_precision, color='purple', alpha=0.5,linewidth=3,label='Original random forest')
sns.lineplot(rf_recall, rf_precision, color='blue', alpha=0.5,linewidth=3,label='Resampled random forest')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Random Forest Precision-Recall curve')
plt.legend(loc='upper left', bbox_to_anchor=(1,1), fontsize = 'large')<jupyter_output>/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.6/dist-package[...]<jupyter_text>Cluster Before Prediction on Original Set<jupyter_code>from sklearn.cluster import KMeans
x_train_minority, x_train_majority = x_train[y_train==1], x_train[y_train==0]
kmeans = KMeans(n_clusters=20, random_state=70).fit(x_train_majority)
prediction = [0]*len(x_validate)
for i in range(0,20):
        majority_segment = x_train_majority[kmeans.labels_==i]
        X_train_segment = pd.concat([majority_segment,x_train_minority])
        y_train_segment = [0]*len(majority_segment)+[1]*len(x_train_minority)
        RF = RandomForestClassifier(max_depth=4, n_estimators=100, bootstrap=True,random_state=701)
        RF.fit(X_train_segment,y_train_segment)
        pred_prob = RF.predict_proba(x_validate)
        prediction = np.add(prediction, pred_prob[:,1])
prediction = prediction/20
y_score = prediction
average_precision = average_precision_score(y_validate, y_score)
cls_precision, cls_recall, cls_thresholds = precision_recall_curve(y_validate, y_score)
# Final Plot
sns.lineplot(original_recall, original_precision, color='red', alpha=0.5,linewidth=2.5,label='Original logistic')
sns.lineplot(resampled_recall, resampled_precision, color='green', alpha=0.5,linewidth=2.5,label='Resampled logistic')
sns.lineplot(rf1_recall, rf1_precision, color='purple', alpha=0.5,linewidth=3,label='Original random forest')
sns.lineplot(rf_recall, rf_precision, color='blue', alpha=0.5,linewidth=3,label='Resampled random forest')
sns.lineplot(cls_recall, cls_precision, color='orange', alpha=0.5,linewidth=3,label='Clustered random forest')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Random Forest Precision-Recall curve')
plt.legend(loc='upper left', bbox_to_anchor=(1,1), fontsize = 'large')<jupyter_output>/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.6/dist-package[...]<jupyter_text>**It is pretty clear that *Original Random Forest* have maximum area under the curve, its pretty high in precision compared to any other but it may be overfitted so for more realistic prediction lets go with *Clustered Random Forest***

**When Recall Rate is in range 0.0-0.5, *Clustered Random Forest* is most promising**  **Threshold and Confusion Matrix**<jupyter_code>error = []
for recall, precision in zip(cls_recall, cls_precision):
    err = (recall-0.90)**2 + (precision-0.83)**2
    error.append(err)
error = np.asarray(error)
min_index = np.where(error==min(error))
opt_threshold = cls_thresholds[min_index]
"The optimised threshold is: {0:0.3f}".format(opt_threshold[0])
from sklearn.metrics import confusion_matrix
y_pred = [1 if x > opt_threshold else 0 for x in y_score]
conf = confusion_matrix(y_validate,y_pred)
conf = pd.DataFrame(conf, range(2),range(2))
sns.heatmap(conf, annot=True, fmt='g', annot_kws={"size": 21})
plt.title('Confusion matrix for validation set')<jupyter_output><empty_output><jupyter_text>Lets Now **Predict on Test set**<jupyter_code>pred_prob = RF.predict_proba(x_test)
y_score = pred_prob[:,1]
y_pred = [1 if x > opt_threshold else 0 for x in y_score]
conf = confusion_matrix(y_test,y_pred)
conf = pd.DataFrame(conf, range(2),range(2))
sns.heatmap(conf, annot=True, fmt='g', annot_kws={"size": 21})
plt.title('Confusion matrix for prediction on test set')<jupyter_output><empty_output><jupyter_text>**Results are Pretty good on our chosen model and threshold set and we have similar results in validation set too.** 

*   In actual situations, accuracy in imbalanced datasets is less prior to choosing an optimised point. We can even lower our precision for better recall.
*   Loss of not identifying positive ones is much much more than false positives.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10046518086509755, 'fraction_numerical': 0.021106713879052975, 'mean_word_length': 6.386666666666667, 'pattern_counts': {'":': 2, '<': 35, '<?xml version=': 0, '>': 37, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '192957', 'license': 'MIT', 'n_tokens_mistral': 4741, 'n_tokens_neox': 4555, 'n_words': 1251, 'path': 'Preprocessing_and_Models.ipynb', 'repo_name': 'kaustubhadixit/Credit-Card-Fraud', 'size': '14403'}