<jupyter_start><jupyter_text>This notebook is meant to do build upon the insights derived from basic data exploration on FanAI DC datasets. The goal is to create new features using the data upon which a robust classification model can be built.  Let's import relevant packages.<jupyter_code>%matplotlib inline 
import numpy as np

import matplotlib
import matplotlib.pyplot as plt
from matplotlib import cm
from scipy.stats import chisquare
import os
import seaborn as sns
#sns.set()

import pandas as pd
from pandas import datetime
import pandas_profiling
from pandas import Series


# Load scikit's random forest classifier library
from sklearn.ensemble import RandomForestClassifier
import sklearn.preprocessing as preprocessing
from imblearn.over_sampling import SMOTE
from collections import Counter<jupyter_output><empty_output><jupyter_text>Read all the data files<jupyter_code>data_dir = '/Users/aprakash/LA_IDX_2019/FanAI/data/'

customers = pd.read_csv(data_dir+ 'customers.csv')
final_sales = pd.read_csv(data_dir+ 'final_sales.csv')
products = pd.read_csv(data_dir+ 'products.csv')
transactions = pd.read_csv(data_dir+ 'transactions.csv')<jupyter_output><empty_output><jupyter_text>Write functions which will assign numerical values to encoded variables and create new features<jupyter_code>#feature funtions
def age_group_2_age(age_group):
    age_group = age_group.strip()
    if age_group =='A':
        Age = 20.0
    elif age_group =='B':
        Age = 27
    elif age_group =='C':
        Age = 32
    elif age_group =='D':
        Age = 37
    elif age_group =='E':
        Age = 42
    elif age_group =='F':
        Age = 47
    elif age_group =='G':
        Age = 52
    elif age_group =='H':
        Age = 57
    elif age_group =='I':
        Age = 62
    elif age_group =='J':
        Age = 67
    elif age_group =='K':
        Age = 75
    else:
        Age = 75
    return Age

def zip_2_distance(zip_group):
    zip_group = zip_group.strip()
    if zip_group =='E':
        Distance = 0
    elif zip_group =='F':
        Distance = 0
    else:
        Distance = 1
    return Distance


def roc_plot(fpr, tpr):
    plt.plot(fpr, tpr, color='Red', label='ROC')
    plt.plot([0, 1], [0, 1], color='blue', linestyle='--')
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver Operating Characteristic (ROC) Curve')
    plt.grid()
    plt.minorticks_on()
    plt.legend()
    plt.show()<jupyter_output><empty_output><jupyter_text>Create binary feature "Far_Distance" which is 0 for customers living closer (E,F) to the store and 1 for customers living farther away (A, B, C, D, G, H). Assuming H (unknown) is also far away   Assign numerical value to Age_group in a new column Age<jupyter_code>customers['Age'] = customers['age_group'].apply(age_group_2_age)
customers['Far_Distance'] = customers['residence_area_code'].apply(zip_2_distance)
customers.head()<jupyter_output><empty_output><jupyter_text>Since the goal is to build a model which would predict if the customer will spend >1500 based on Final_sales, I create another binary feature "high_spender" which is 0 for customers spending  1500. This is the target variable for classification.<jupyter_code>final_sales['high_spender'] = (final_sales['sales_price'] >= 1500).astype('int')
final_sales.head()<jupyter_output><empty_output><jupyter_text>Assemble the data in one data frame by joining of common keys<jupyter_code>trans_w_product  = pd.merge(transactions,products, on='product_id', how='left')
trans_n_customer = pd.merge(trans_w_product,customers, on='customer_id', how='left')
trans_n_customer.head()<jupyter_output><empty_output><jupyter_text>Using this dataset I can derive new features on customer purchasing behavior First, I combine the sales by day for each customer since everything purchased on the same date is considered part of the same order. Second, I also sum the number of items bought on the same day for each customer<jupyter_code>e = trans_n_customer.groupby(['customer_id', 'transaction_date'])['sales_price'].sum()
f =trans_n_customer.groupby(['customer_id', 'transaction_date'])['quantity'].sum()

c = pd.merge(e,f,left_index=True,right_index=True)
c.sort_values(['customer_id', 'transaction_date'], ascending=[True, True])

c.reset_index()<jupyter_output><empty_output><jupyter_text>fixing the mistake pointed by Harry. Dropping the transactions in the final sales files while creating new features.<jupyter_code>d = c.merge(final_sales.drop_duplicates(), on=['customer_id','transaction_date'], 
                   how='left', indicator=True)
d.sort_values(['customer_id', 'transaction_date'], ascending=[True, True])

d_new = d[d['_merge'] == 'left_only']
d_slim = d_new.drop(['sales_price_y','high_spender', '_merge'], axis=1)
d_slim<jupyter_output><empty_output><jupyter_text>This new dataframe gives us a time series for each customer where for a given date we have the aggregate purchasing price and total number of items. Using these time series for each customer, I can create new features which is average speding by a customer on a day and average number of items purchased in a day. Rename the columns to reflect their true value.<jupyter_code>g = d_slim.groupby(['customer_id'])['sales_price_x'].mean()
h = d_slim.groupby(['customer_id'])['quantity'].mean()
dd = pd.merge(g,h,left_index=True,right_index=True)
dd.reset_index()
dd.rename(columns={'sales_price_x': 'average_spending', 'quantity': 'average_number_items'}, inplace=True)<jupyter_output><empty_output><jupyter_text>Combine customer demographic data with average spending data-frame<jupyter_code>ii = pd.merge(dd,customers, on='customer_id', how='inner')
ii.head()<jupyter_output><empty_output><jupyter_text>Final dataframe is merged with final_sales containing target variable("high_spender") which is estimated based on if sales_price >1500  redundant columns are dropped since their feature has been extracted in opther column<jupyter_code>jj = pd.merge(ii,final_sales, on='customer_id', how='inner')
jj.head()
final_df= jj.drop(['age_group', 'residence_area_code','sales_price'], axis=1)
final_df.head()<jupyter_output><empty_output><jupyter_text>At this point data is ready to be modeled split the dataset in training : test (80:20) usimng random sampling.<jupyter_code>np.random.seed(42)
final_df['is_train'] = np.random.uniform(0, 1, len(final_df)) <= .80
train, test = final_df[final_df['is_train']==True], final_df[final_df['is_train']==False]
print('Number of observations in the training data:', len(train))
print('Number of observations in the test data:',len(test))<jupyter_output>Number of observations in the training data: 16315
Number of observations in the test data: 4073<jupyter_text>Create a list of fetures (independent variables) and define target variable<jupyter_code>Indep_var =final_df.columns[:7]

# View features
Indep_var
target_var = train['high_spender']

target_var<jupyter_output><empty_output><jupyter_text>Create a simple Random forest classifier using sufficiently large number of trees  and ensure that bootstrapped subsmaples are balanced (since the data is inherently unbalanced) Using SMOTE to oversample and balance datasets.  train the classifier on the trainign data and test it on test data.<jupyter_code>sm = SMOTE(sampling_strategy='float', random_state=12, ratio = 1.0)
x_train_res, y_train_res = sm.fit_sample(train[Indep_var], target_var)
print('Resampled dataset shape %s' % Counter(y_train_res))

clf = RandomForestClassifier(n_estimators = 100, random_state=0, class_weight= 'balanced_subsample')

clf.fit(x_train_res, y_train_res)

# Apply the Classifier we trained to the test data (which, remember, it has never seen before)
preds = clf.predict(test[Indep_var])<jupyter_output><empty_output><jupyter_text>View the predicted probabilities of the observations<jupyter_code>predicted_prob = clf.predict_proba(test[Indep_var])
predicted_prob<jupyter_output><empty_output><jupyter_text>Create confusion matrix<jupyter_code>pd.crosstab(test['high_spender'], preds, rownames=['Actual spending'], colnames=['Predicted Spending'])<jupyter_output><empty_output><jupyter_text>View the list of features and their importance scores (a very useful and important feature of Random forrest) plot the feature score for convenience<jupyter_code>list(zip(train[Indep_var], clf.feature_importances_))
print(preds[0:15])
print(test['high_spender'][0:15])
plt.figure(figsize=(15,10))
plt.plot(Indep_var, clf.feature_importances_,'gP')
plt.ylabel('Feature Importance', fontsize=20)
plt.yticks(fontsize=15)
plt.xticks(fontsize=12)
plt.grid()
plt.minorticks_on()
plt.xlabel('Feature', fontsize=20)<jupyter_output><empty_output><jupyter_text>Estimate different metrics of performace of the model using confusion matrix<jupyter_code>from sklearn.metrics import confusion_matrix 
from sklearn.metrics import accuracy_score 
from sklearn.metrics import classification_report 
from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score

actual = test['high_spender'] 
predicted = preds
results = confusion_matrix(actual, predicted) 
print('Confusion Matrix :')
print(results) 

acc_score = accuracy_score(actual, predicted)
print('Accuracy Score :',accuracy_score(actual, predicted))
print('Report : ')
print(classification_report(actual, predicted))<jupyter_output>Confusion Matrix :
[[2996  363]
 [ 446  268]]
Accuracy Score : 0.8013749079302726
Report : 
              precision    recall  f1-score   support

           0       0.87      0.89      0.88      3359
           1       0.42      0.38      0.40       714

    accuracy                           0.80      4073
   macro avg       0.65      0.63      0.64      4073
weighted avg       0.79      0.80      0.80      4073<jupyter_text>ROC Analysis<jupyter_code>fpr, tpr, thresholds = roc_curve(actual,predicted_prob[:,1],pos_label=1)
roc_plot(fpr, tpr)
auc = roc_auc_score(actual,predicted_prob[:,1])
print('AUC: %.2f' %auc)<jupyter_output>AUC: 0.74
{'fraction_non_alphanumeric': 0.1047466453034248, 'fraction_numerical': 0.020528740236330865, 'mean_word_length': 5.5061889250814335, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '543693', 'license': 'MIT', 'n_tokens_mistral': 3373, 'n_tokens_neox': 3171, 'n_words': 885, 'path': 'FanAI-DC-Modeling.ipynb', 'repo_name': 'abp15/Data_challenges', 'size': '9986'}