<jupyter_start><jupyter_text>Hash Encode some of categorical features and predict Here, an attempt to see how the data can be fitted by hash encoding the features that seemed most important for prediction from exploration of the data set, combined with random forest classifier<jupyter_code># import useful packages

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline

import seaborn as sns
sns.set()

mpl.rcParams['lines.linewidth']=2.0
mpl.rcParams['xtick.labelsize']=13
mpl.rcParams['ytick.labelsize']=13
mpl.rcParams['axes.labelsize']=15
mpl.rcParams['axes.labelweight']='heavy'
mpl.rcParams['axes.titlesize']=18
mpl.rcParams['axes.titleweight']='heavy'
mpl.rcParams['legend.fontsize']=12



from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn import metrics
from sklearn.metrics import accuracy_score

import category_encoders as ce

import os
import sys
src_dir = os.path.join(os.path.dirname(os.getcwd()),'adfraud')
sys.path.append(src_dir)

data_dir = os.path.join(os.path.dirname(os.getcwd()),'data')

data_filename = 'train_sample.csv'
data_location = os.path.join(data_dir,data_filename)
ad_data = pd.read_csv(data_location)
feats_df  = ad_data[['ip','app','channel']]
labels_df = ad_data['is_attributed'].values
#labels = ad_data['is_attributed'].values


x_train,x_test,y_train,y_test = train_test_split(feats_df,labels_df,test_size=0.3)<jupyter_output><empty_output><jupyter_text>Encode the Channel and App fields<jupyter_code>hasher = ce.hashing.HashingEncoder(n_components=50,cols=['app','channel'])<jupyter_output><empty_output><jupyter_text>Here, I've used a hashing encoder, hashing the categorical fields into a smaller number of columns than would be found with a one hot encoder.We can treat the number of components as a hyperparameter of the model.<jupyter_code>hasher.fit(x_train)
x_train_hashed = hasher.transform(x_train)
x_train_hashed.head(5)<jupyter_output><empty_output><jupyter_text>Note that I have not hashed ip, juts the app and channel fields.Some **issues** to have in mind for this model: - out-of-sample test data. If we have app and channel fields in the test set that do not appear in the training set, this will obviously cuase problems.  - ip is categorical really. In a real-time system - would want to classify based on times seen in previous x minutes/hours or something maybe. Since we will| use random forest classification, leaving ip in it's raw format may be acceptable for this problem.<jupyter_code>clf = RandomForestClassifier(n_estimators=20, max_depth=5,class_weight={0:0.001,1:0.999})
clf.fit(x_train_hashed.values,y_train)
print(np.shape(x_train_hashed.values))
print(np.shape(y_train))
x_test_hashed = hasher.transform(x_test)
preds = clf.predict_proba(x_test_hashed.values)
fpr, tpr, thresholds = metrics.roc_curve(y_test, preds[:,1])
auc = metrics.roc_auc_score(y_test, preds[:,1])
print(auc)
plt.plot(fpr,tpr,'r*', label='area={:.4f}'.format(auc))
plt.plot(np.linspace(0,1,20),np.linspace(0,1,20),'k--')
plt.xlabel('FPR')
plt.ylabel('TPR')
plt.title('ROC')

plt.legend(frameon=False)

plt.xlim([0.0,1.0])
plt.ylim([0.0,1.0])
plt.show()<jupyter_output><empty_output><jupyter_text>Let's confirm what happens when the hasher recieves a set of features that don't appear in the training:<jupyter_code>atest = x_train.head(1).copy().reset_index()
atest = atest.drop(['index'],axis=1)
atest<jupyter_output><empty_output><jupyter_text>reset the elements of the simple checking data sample to out-of-sample values<jupyter_code>atest.loc[0,'ip']     =1234567
atest.loc[0,'app']    =1234567
atest.loc[0,'channel']=1234567
atest
hasher.transform(atest)<jupyter_output><empty_output><jupyter_text>So, it will  hash to something the random forest can classify.<jupyter_code>clf.predict_proba(hasher.transform(atest))<jupyter_output><empty_output><jupyter_text>How does the accuracy of the model vary with the threshold?<jupyter_code>def pred_from_prob(a,thrshld):
    bin_preds = np.zeros((np.size(a,0),))
    bin_preds[np.where(a[:,1]>thrshld)]=1.0
    return bin_preds
binary_preds = pred_from_prob(preds,0.61)
print(accuracy_score(y_test,binary_preds))
print(accuracy_score(y_test[np.where(y_test==1)],binary_preds[np.where(y_test==1)]))
print(accuracy_score(y_test[np.where(y_test==0)],binary_preds[np.where(y_test==0)]))


#thrshs = np.linspace(0.01, 1.0, 50)
acc_tot = []
acc_0 = []
acc_1 = []
for ind,th in enumerate(thresholds):
    binary_preds = pred_from_prob(preds,th)
    acc_tot.append(accuracy_score(y_test,binary_preds))
    acc_1.append(accuracy_score(y_test[np.where(y_test==1)],binary_preds[np.where(y_test==1)]))
    acc_0.append(accuracy_score(y_test[np.where(y_test==0)],binary_preds[np.where(y_test==0)]))


mpl.rcParams['figure.figsize']=[15.0,5.0]

plt.subplot(1,2,1)
plt.plot(fpr,tpr,'r*', label='roc auc={:.4f}'.format(auc))
plt.plot(np.linspace(0,1,20),np.linspace(0,1,20),'k--')

#plt.plot(fpr,acc_tot,'b*-',label='accuracy')
#plt.plot(fpr,acc_0,'go--',label='accuracy - 0',alpha=0.3)
plt.plot(fpr,acc_1,'y--',alpha=0.9,label='accuracy - cases:1')

plt.xlabel('FPR')
plt.ylabel('TPR')
plt.title('ROC')

plt.legend(frameon=False)

plt.xlim([0.0,1.0])
plt.ylim([0.0,1.0])


plt.subplot(1,2,2)

#plt.plot(thresholds,acc_tot,'b*-',label='accuracy')
plt.plot(thresholds,acc_0,'go--',label='cases: 0',alpha=0.3)
plt.plot(thresholds,acc_1,'yo--',label='cases: 1')
plt.plot(thresholds[::5],fpr[::5],'c*',alpha=0.2,label='fpr')

plt.xlim([0,1])
plt.xlim([0,1])

plt.xlabel('threshold')
plt.ylabel('accuracy')
plt.title('accuracy')

plt.legend(frameon=False)

plt.show()<jupyter_output><empty_output><jupyter_text>This is starting to look like a good model for prediction - at least on this small subset of the training data. We have yet to include all the features for the prediction, which may give extra ROC AUC. The model has not yet been tuned either, tuning hyperparameters should give an improvement to the model too.In order to develop these models further, the model will be put into a pipeline in future codebase - allowing for easier hyperparameter adjustment and optimization, as well as for comparison between different variations of the model. Slightly surprise dby accuracy of the model, quickly check the accuracy with some home built logic for sanity check....<jupyter_code>acc_check=[]
acc_check_0=[]
acc_check_1=[]
for ind,th in enumerate(thresholds):
    binary_preds = pred_from_prob(preds,th)
    checker = (y_test-0.5)*(binary_preds-0.5)
    if ind==0:
        print(np.sum(checker))
        print(len(checker))
        print(np.where(checker>0.0))
        print(checker[np.where(checker>0.0) and np.where(y_test==0)])
        print(checker[np.where(checker>0.0) and np.where(y_test==1)])
    acc_check.append(4.0*np.sum(checker[np.where(checker>0.0)])/len(checker))
    checker_0 = checker[np.where(y_test==0)]
    acc_check_0.append(4.0*np.sum(checker_0[checker_0>0.0])/len(checker_0))
    checker_1 = checker[np.where(y_test==1)]
    acc_check_1.append(4.0*np.sum(checker_1[checker_1>0.0])/len(checker_1))
plt.plot(acc_check,'r')
plt.plot(acc_tot,'k--')
plt.plot(acc_check_1,'y--')
plt.plot(acc_1,'m-.')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14577018462386332, 'fraction_numerical': 0.026453568476164233, 'mean_word_length': 6.673361522198731, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '596190', 'license': 'Apache-2.0', 'n_tokens_mistral': 2858, 'n_tokens_neox': 2717, 'n_words': 537, 'path': 'notebooks/simple_encode_categoricals.ipynb', 'repo_name': 'pdghawk/adfraud', 'size': '7258'}