<jupyter_start><jupyter_text>Project Overall ObjectsThis notebook file would approach and achieve the objects highlighted in **black**. Those in *green* have already been accomplished in earlier analysis. -  Clean dirty log data and transform it for analytics. -  Exploratory Data Analysis (EDA). -  Find the conversion rate of users, identify key factors that bottleneck the conversion rate. -  Propose any hypothesis and test through analyzing features.- Build machine learning models to predict user behaviors, including but not limited to signup, churn, etc.- Discover interesting insights in the dataset and suggest how to improve the user signup rate.This notebook file is to develop machine learning models to predict user behaviors, based on selected features from the previous analyses. We would build and compare the results of Logistic Regression Models, Decision Trees (Single and Bagged), K-Nearest Neighbors (Single and Bagged), Random Forest, Gradient Boosting Tree, MLP Neural Network, and Linear & Non-Linear SVM. Lastly, we apply a hyper-parametering tuning on a selected model.  Load data and browse data<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

% matplotlib inline
# Always make it pretty.
plt.style.use('ggplot')
# Load data from file
df = pd.read_csv('data/selected_data.csv')
# Browse dataset
df.head(10)
# Show summary stats
df.describe()<jupyter_output><empty_output><jupyter_text>Build Logistic Regression Model  Encode categorical columns to numeric values<jupyter_code>df.columns
col_category = ['model_bin','os_bin','CT_bin','PRO_bin','source','content','medium']
df_dummies = pd.get_dummies(df[col_category], columns=col_category)
df_dummies.head()
df = df.join(df_dummies)
df.head()
df.columns
df['avg_st'] = df['avg_st'].fillna(0)
# Use scatter_matrix from Pandas
from pandas.plotting import scatter_matrix
scatter_matrix(df[[u'signup', u'Ttoend', u'count',u'avg_st']], 
               alpha=0.2, figsize=(16, 16), diagonal='kde')
plt.savefig('timescatter_feature_selection.png',bbox_inches="tight",dpi=500)
plt.show()<jupyter_output><empty_output><jupyter_text>Define Features and Target<jupyter_code>df['log_st'] = np.log((df.avg_st+1)/1000) 
df['log_Ttoend'] = np.log(df.Ttoend+1) 
df['log_ileave'] = np.log(df.ileave+1) 
df['log_npage'] = np.log(df.npage+1) 
df['log_nclick'] = np.log(df.nclick+1) 
selected_features = [u'count',u'worktime',u'weekend',
                     u'log_st',
                     u'model_bin_mac',u'model_bin_others', u'model_bin_pc', u'os_bin_macosx', u'os_bin_others',
                     u'os_bin_windows', u'CT_bin_Beijing', u'CT_bin_Guangzhou',
                     u'CT_bin_Shanghai', u'CT_bin_others', u'PRO_bin_BJ', u'PRO_bin_GD',
                     u'PRO_bin_SH', u'PRO_bin_ZJ', u'PRO_bin_others', u'source_baidu',
                     u'source_others', u'source_others',u'content_data',u'content_others',u'content_ukcontent',
                     u'medium_cpc', u'medium_others', u'medium_ukmedium',
                     u'log_npage', u'log_nclick', u'log_ileave', 
                     u'log_Ttoend',
                     u'is_fd', u'is_ft', u'CN']
target = u'signup'
X = df[selected_features].values
y = df['signup'].values
X.shape<jupyter_output><empty_output><jupyter_text>Let's Train-test split the data!<jupyter_code># import train test split function from sklearn
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)<jupyter_output><empty_output><jupyter_text>Logistic Regression model using sklearn<jupyter_code># Import logistic regression from sklearn
from sklearn.linear_model import LogisticRegression

# Initialize model by providing parameters
# http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
clf = LogisticRegression(C=1.0, penalty='l1')

# Fit a model by providing X and y from training set
clf.fit(X_train, y_train)

# Make prediction on the training data
y_train_pred = clf.predict(X_train)
p_train_pred = clf.predict_proba(X_train)[:,1]

# Make predictions on test data
y_test_pred = clf.predict(X_test)
p_test_pred = clf.predict_proba(X_test)[:,1]<jupyter_output><empty_output><jupyter_text>Calculate the metric scores for the model<jupyter_code># Import metrics functions from sklearn
from sklearn.metrics import precision_score, accuracy_score, recall_score, f1_score, roc_auc_score
# Helper method to print metric scores    
def get_performance_metrics(y_train, y_train_pred, y_test, y_test_pred, threshold=0.5):
    metric_names = ['AUC','Accuracy','Precision','Recall','f1-score']
    metric_values_train = [roc_auc_score(y_train, y_train_pred),
                    accuracy_score(y_train, y_train_pred>threshold),
                    precision_score(y_train, y_train_pred>threshold),
                    recall_score(y_train, y_train_pred>threshold),
                    f1_score(y_train, y_train_pred>threshold)
                   ]
    metric_values_test = [roc_auc_score(y_test, y_test_pred),
                    accuracy_score(y_test, y_test_pred>threshold),
                    precision_score(y_test, y_test_pred>threshold),
                    recall_score(y_test, y_test_pred>threshold),
                    f1_score(y_test, y_test_pred>threshold)
                   ]
    all_metrics = pd.DataFrame({'metrics':metric_names,
                                'train':metric_values_train,
                                'test':metric_values_test},columns=['metrics','train','test']).set_index('metrics')
    print(all_metrics)
from sklearn.metrics import roc_curve, auc

def plot_roc_curve(y_train, y_train_pred, y_test, y_test_pred, title):
    roc_auc_train = roc_auc_score(y_train, y_train_pred)
    fpr_train, tpr_train, _ = roc_curve(y_train, y_train_pred)

    roc_auc_test = roc_auc_score(y_test, y_test_pred)
    fpr_test, tpr_test, _ = roc_curve(y_test, y_test_pred)
    plt.figure()
    lw = 2
    plt.plot(fpr_train, tpr_train, color='green',
             lw=lw, label='ROC Train (AUC = %0.4f)' % roc_auc_train)
    plt.plot(fpr_test, tpr_test, color='darkorange',
             lw=lw, label='ROC Test (AUC = %0.4f)' % roc_auc_test)
    plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver operating characteristic example')
    plt.legend(loc="lower right")
    plt.savefig('ROC_' + title + '.png', bbox_inches="tight",dpi=500)
    plt.show()
# print model results
get_performance_metrics(y_train, p_train_pred, y_test, p_test_pred)
plot_roc_curve(y_train, p_train_pred, y_test, p_test_pred, 'LgReg')<jupyter_output>train      test
metrics                      
AUC        0.951113  0.953284
Accuracy   0.961562  0.965557
Precision  0.625000  0.763636
Recall     0.230548  0.280000
f1-score   0.336842  0.409756<jupyter_text>recall=tp/(tp+fn): for this problem, sign up rate is low, it is expected that we see low tp compared with fn precision=tp/(tp+fp): this metric is more importatn, low precision might due to small number of data amount Understanding the Estimated Coefficients<jupyter_code>df_coeffs = pd.DataFrame(list(zip(selected_features, clf.coef_.flatten()))).sort_values(by=[1], ascending=False)
df_coeffs.columns = ['feature', 'coeff']
df_coeffs
from pylab import rcParams
#plt.barh(y_pos, performance, align='center', alpha=0.5)
ax = df_coeffs.plot.barh(align='center',width=0.35)
t = np.arange(X.shape[1])
ax.set_yticks(t)
ax.set_yticklabels(df_coeffs['feature'])
rcParams['figure.figsize'] = 12, 12
plt.savefig('features.png',bbox_inches="tight",dpi=500)
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>model evaluation  confusion matrix<jupyter_code>from sklearn.metrics import confusion_matrix, classification_report, roc_curve
confusion_matrix(y_test, y_test_pred)
# Helper method to plot confusion matrix
def plot_confusion_matrix(y_true, y_pred):
    '''
    Code from sklearn example.
    '''
    cm = confusion_matrix(y_true, y_pred)
    print(cm)

    # Show confusion matrix in a separate window
    plt.matshow(cm)
    plt.title('Confusion matrix')
    plt.colorbar()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    plt.show()
plot_confusion_matrix(y_train, y_train_pred)
plot_confusion_matrix(y_test, y_test_pred)
print("Area Under Curve (AUC) of the Logistic Regression is: {}".format(roc_auc_score(y_test, y_test_pred)))<jupyter_output>Area Under Curve (AUC) of the Logistic Regression is: 0.6380672019030628<jupyter_text>Bagging Single Tree<jupyter_code>from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(max_depth=20,min_samples_leaf=10)

# Fit a model by providing X and y from training set
clf.fit(X_train, y_train)

# Make prediction on the training data
y_train_pred = clf.predict(X_train)
p_train_pred = clf.predict_proba(X_train)[:,1]

# Make predictions on test data
y_test_pred = clf.predict(X_test)
p_test_pred = clf.predict_proba(X_test)[:,1]

# print model results
get_performance_metrics(y_train, p_train_pred, y_test, p_test_pred)
plot_roc_curve(y_train, p_train_pred, y_test, p_test_pred, 'Single_Tree')
# define function to perform train, test, and get model performance
def train_test_model(clf, X_train, y_train, X_test, y_test, title):
    # Fit a model by providing X and y from training set
    clf.fit(X_train, y_train)

    # Make prediction on the training data
    y_train_pred = clf.predict(X_train)
    p_train_pred = clf.predict_proba(X_train)[:,1]

    # Make predictions on test data
    y_test_pred = clf.predict(X_test)
    p_test_pred = clf.predict_proba(X_test)[:,1]

    # print model results
    get_performance_metrics(y_train, p_train_pred, y_test, p_test_pred)
    plot_roc_curve(y_train, p_train_pred, y_test, p_test_pred, title)<jupyter_output><empty_output><jupyter_text>Bagged Trees<jupyter_code># http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html 
from sklearn.ensemble import BaggingClassifier

base_classifier = DecisionTreeClassifier(max_depth=20,min_samples_leaf=10)

# Choose some parameter combinations to try
parameters = {
              'base_estimator':base_classifier,
              'n_estimators': 50,
              'n_jobs': -1
              }

clf = BaggingClassifier(**parameters)

# Train test model
train_test_model(clf, X_train, y_train, X_test, y_test, 'Bagged_Tree')<jupyter_output>train      test
metrics                      
AUC        0.996754  0.980669
Accuracy   0.986089  0.982921
Precision  0.960474  0.924528
Recall     0.700288  0.653333
f1-score   0.810000  0.765625<jupyter_text>bagging is a very useful and effective methods for low amount of data points.  Single KNN<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
parameters = {
    #'weights':'distance',
    'n_neighbors':3,
    'leaf_size':10
}
base_classifier = KNeighborsClassifier(**parameters)
clf = base_classifier

# Train test model
train_test_model(clf, X_train, y_train, X_test, y_test, 'Single_KNN')<jupyter_output>train      test
metrics                      
AUC        0.986976  0.789707
Accuracy   0.974253  0.962141
Precision  0.865591  0.607595
Recall     0.463977  0.320000
f1-score   0.604128  0.419214<jupyter_text>Bagged KNN<jupyter_code>from sklearn.ensemble import BaggingClassifier

# Choose some parameter combinations to try
parameters = {
              'base_estimator':base_classifier,
              'n_estimators': 30,
              'n_jobs': -1
              }

clf = BaggingClassifier(**parameters)

# Train test model
train_test_model(clf, X_train, y_train, X_test, y_test, 'Bagged_KNN')<jupyter_output>train      test
metrics                      
AUC        0.995139  0.866518
Accuracy   0.975595  0.963279
Precision  0.934911  0.652174
Recall     0.455331  0.300000
f1-score   0.612403  0.410959<jupyter_text>Random Forest<jupyter_code># http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
from sklearn.ensemble import RandomForestClassifier

# Choose some parameter combinations to try
parameters = {'n_estimators': 50,
              'max_features': 'auto',
              'criterion': 'gini',
              'max_depth': 20,
              'min_samples_split': 2,
              'min_samples_leaf': 20,
              'random_state': 0,
              'n_jobs': -1
              }

clf = RandomForestClassifier(**parameters)

# Fit a model by providing X and y from training set
clf.fit(X_train, y_train)

# Train test model
train_test_model(clf, X_train, y_train, X_test, y_test, 'Random_Forest')<jupyter_output>train      test
metrics                      
AUC        0.987776  0.972267
Accuracy   0.969494  0.968688
Precision  1.000000  1.000000
Recall     0.279539  0.266667
f1-score   0.436937  0.421053<jupyter_text>Gradient Boosting Trees<jupyter_code># http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html
from sklearn.ensemble import GradientBoostingClassifier

# Choose some parameter combinations to try
parameters = {
    'n_estimators': 50,
    'max_depth': 5,
    'learning_rate': 0.2,
    'random_state': 42
}

# parameters = {
#     'n_estimators': 50,
#     'max_depth': 5,
#     'learning_rate': 0.2,
#     'subsample': 0.7,
#     'max_features':0.8,
#     'random_state': 42
# }

clf = GradientBoostingClassifier(**parameters)

# Train test model
train_test_model(clf, X_train, y_train, X_test, y_test, 'Gradient_Boosting')
df_importance = pd.DataFrame(list(zip(selected_features, clf.feature_importances_.flatten()))).sort_values(by=[1], ascending=False)
df_importance.columns = ['feature', 'importance']
df_importance<jupyter_output><empty_output><jupyter_text>Neural Network<jupyter_code># http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier
from sklearn.neural_network import MLPClassifier

# Choose some parameter combinations to try
parameters = {
    'solver':'adam', 
    'activation':'relu',
    'alpha':1e-5, #increase alpha->increase penalty :: http://scikit-learn.org/stable/auto_examples/neural_networks/plot_mlp_alpha.html#sphx-glr-auto-examples-neural-networks-plot-mlp-alpha-py
    'hidden_layer_sizes':(5,5), 
    'learning_rate':'adaptive',
    'random_state':1
    }
clf = MLPClassifier(**parameters)

# Train test model
train_test_model(clf, X_train, y_train, X_test, y_test, 'MLP_NN')<jupyter_output>train      test
metrics                      
AUC        0.993069  0.989047
Accuracy   0.986699  0.983205
Precision  0.964844  0.925234
Recall     0.711816  0.660000
f1-score   0.819237  0.770428<jupyter_text>SVM Linear SVM<jupyter_code>from sklearn.svm import LinearSVC

# Choose some parameter combinations to try
clf = LinearSVC()

# Fit a model by providing X and y from training set
clf.fit(X_train, y_train)

# No predict_proba for LinearSVC

# Make prediction on the training data
p_train_pred = clf.predict(X_train)

# Make predictions on test data
p_test_pred = clf.predict(X_test)

# print model results
get_performance_metrics(y_train, p_train_pred, y_test, p_test_pred)
plot_roc_curve(y_train, p_train_pred, y_test, p_test_pred, 'Linear_SVM')<jupyter_output>train      test
metrics                      
AUC        0.653453  0.667324
Accuracy   0.966687  0.966695
Precision  0.760563  0.739130
Recall     0.311239  0.340000
f1-score   0.441718  0.465753<jupyter_text>NonLinear SVM<jupyter_code># http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html
from sklearn.svm import SVC

# Choose some parameter combinations to try
parameters = {
    'probability':True, # get simulated probability
    'max_iter':2000
    }
clf = SVC(**parameters)    

# Train test model
train_test_model(clf, X_train, y_train, X_test, y_test, 'NonLinear_SVM')<jupyter_output>/Users/leoli/anaconda3/lib/python3.6/site-packages/sklearn/svm/base.py:218: ConvergenceWarning: Solver terminated early (max_iter=2000).  Consider pre-processing your data with StandardScaler or MinMaxScaler.
  % self.max_iter, ConvergenceWarning)<jupyter_text>HyperParameter Tuning: Grid Search<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import make_scorer, roc_auc_score, accuracy_score
from sklearn.model_selection import GridSearchCV

# Choose the type of classifier. 
clf = RandomForestClassifier()

# Choose some parameter combinations to try
param_grid = {'n_estimators': [100,200], 
              'max_features': ['auto'], 
              'criterion': ['gini'],
              'max_depth': [15,20,25], 
              'min_samples_split': [2],
              'min_samples_leaf': [2,10,20],
              'n_jobs':[-1]
             }

# Type of scoring used to compare parameter combinations
acc_scorer = make_scorer(roc_auc_score)

# Run the grid search
# read theory
grid_obj = GridSearchCV(clf, param_grid, cv=5, scoring=acc_scorer)
grid_obj = grid_obj.fit(X_train, y_train)

# Set the clf to the best combination of parameters
clf = grid_obj.best_estimator_

# Fit the best algorithm to the data. 
clf.fit(X_train, y_train)
# Train test model
train_test_model(clf, X_train, y_train, X_test, y_test, 'selected_RF_tuned')<jupyter_output>train      test
metrics                      
AUC        0.999978  0.976666
Accuracy   0.987309  0.977227
Precision  1.000000  1.000000
Recall     0.700288  0.466667
f1-score   0.823729  0.636364
{'fraction_non_alphanumeric': 0.10366653361464333, 'fraction_numerical': 0.04236756571819581, 'mean_word_length': 4.144617189791727, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 72, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '41632', 'license': 'MIT', 'n_tokens_mistral': 6632, 'n_tokens_neox': 6086, 'n_words': 1310, 'path': '3_Supervised_Learnings.ipynb', 'repo_name': 'alexlovescoding/User-Behavior-Analysis-and-Prediction', 'size': '17537'}