<jupyter_start><jupyter_text>Intuit Quickbooks Upgrade* Team-lead GitLab userid:* Group name:* Team member names:  SetupPlease complete this python notebook with your group by answering the questions in `intuit-redux.pdf`. Create a Notebook and HTML file with all your results and comments and push both the Notebook and HTML file to GitLab when your team is done. All results MUST be reproducible (i.e., the TA and I must be able to recreate the HTML file from the Jupyter Notebook without changes or errors). This means that you should NOT use any python-packages that are not part of the rsm-msba-spark docker container.This is the second group assignment for MGTA 455 and you will be using Git and GitLab. If two people edit the same file at the same time you could get what is called a "merge conflict". This is not something serious but you should realize that Git will not decide for you who's change to accept so the team-lead will have to determine the edits to use. To avoid merge conflicts, **always** "pull" changes to the repo before you start working on any files. Then, when you are done, save and commit your changes, and then push them to GitLab. Make "pull first" a habit!If multiple people are going to work on the assignment at the same time I recommend you work in different notebooks. You can then `%run ...`  these "sub" notebooks from the main assignment file. You can seen an example of this in action below for the `model1.ipynb` notebookSome group work-flow tips:* Pull, edit, save, stage, commit, and push* Schedule who does what and when* Try to avoid working simultaneously on the same file * If you are going to work simultaneously, do it in different notebooks, e.g.,     - model1.ipynb, model2.ipynb, model3.ipynb* Use the `%run ... ` command to bring different pieces of code together into the main jupyter notebook* Put python functions in modules that you can import from your notebooks. See the example below for the `example` function defined in `utils/functions.py`A graphical depiction of the group work-flow is shown below:![](images/git-group-workflow-wbg.png)Tutorial videos about using Git, GitLab, and GitGadget for group assignments:* Setup the MSBA server to use Git and GitLab: https://youtu.be/zJHwodmjatY* Dealing with Merge Conflicts: https://youtu.be/qFnyb8_rgTI* Group assignment practice: https://youtu.be/4Ty_94gIWeA<jupyter_code>import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import pyrsm as rsm
import statsmodels.formula.api as smf
from sklearn import preprocessing
from statsmodels.genmod.families import Binomial
from statsmodels.genmod.families.links import logit

# increase plot resolution
# mpl.rcParams["figure.dpi"] = 150
## loading the data - this dataset must NOT be changed
intuit75k = pd.read_pickle("data/intuit75k.pkl")
intuit75k["res1_yes"] = (intuit75k["res1"] == "Yes").astype(int)
intuit75k.head()
# show dataset description
rsm.describe(intuit75k)
# run python code from another notebook
%run ./sub-notebooks/model1.ipynb
# importing functions from a module/package
from utils import functions
functions.example()<jupyter_output>You just accessed a function from your first python packages!
Change the code in utils/function.py to whatever you need for this assignment
Use 'from utils import functions' to get access to your code
You can add modules to import from by adding additional .py files to the 'utils' directory
Note: If you make changes to the content of this file you will have to restart the notebook kernel to get the updates<jupyter_text>1. Explore the data again<jupyter_code>import seaborn as sns
import matplotlib.pyplot as plt
f, axes = plt.subplots(3, 4, figsize=(16, 12))#, sharex=True)

sns.countplot(intuit75k["zip_bins"], color="skyblue", ax=axes[0, 0])
sns.countplot(intuit75k['sex'], color="olive", ax=axes[0, 1])
sns.countplot(intuit75k['bizflag'], color="gold", ax=axes[0, 2])
sns.countplot(intuit75k['numords'], color="teal", ax=axes[0, 3])
sns.histplot(intuit75k['dollars'], color="teal", ax=axes[1, 0])
sns.histplot(intuit75k['last'], color="gold", ax=axes[1, 1])
sns.histplot(intuit75k['sincepurch'], color="olive", ax=axes[1, 2])
sns.countplot(intuit75k['version1'], color="skyblue", ax=axes[1, 3])
sns.countplot(intuit75k['owntaxprod'], color="skyblue", ax=axes[2, 0])
sns.countplot(intuit75k['upgraded'], color="olive", ax=axes[2, 1])
sns.countplot(intuit75k['res1'], color="gold", ax=axes[2, 2])
sns.countplot(intuit75k['training'], color="teal", ax=axes[2, 3])


plt.show()
f, axes = plt.subplots(3, 4, figsize=(16, 12))#, sharex=True)

sns.countplot(x='res1', hue='zip_bins', data=intuit75k, color="skyblue", ax=axes[0, 0])
sns.countplot(x='res1', hue='sex', data=intuit75k, color="olive", ax=axes[0, 1])
sns.countplot(x='res1', hue='bizflag', data=intuit75k, color="gold", ax=axes[0, 2])
sns.countplot(x='res1', hue='numords', data=intuit75k, color="teal", ax=axes[0, 3])
sns.barplot(x='res1', y='dollars', data=intuit75k, color="teal", ax=axes[1, 0])
sns.barplot(x='res1', y='last', data=intuit75k, color="gold", ax=axes[1, 1])
sns.barplot(x='res1', y='sincepurch', data=intuit75k, color="olive", ax=axes[1, 2])
sns.countplot(x='res1', hue='version1', data=intuit75k, color="skyblue", ax=axes[1, 3])
sns.countplot(x='res1', hue='owntaxprod', data=intuit75k, color="skyblue", ax=axes[2, 0])
sns.countplot(x='res1', hue='upgraded', data=intuit75k, color="olive", ax=axes[2, 1])
sns.countplot(x='res1', hue='res1_yes', data=intuit75k, color="gold", ax=axes[2, 2])
sns.countplot(x='res1', hue='training', data=intuit75k, color="teal", ax=axes[2, 3])


plt.show()<jupyter_output><empty_output><jupyter_text>2. Adjust input variables<jupyter_code>#zip bins & dollars
intuit75k['zip_bins_1'] = (intuit75k['zip_bins'] == 1).astype(int)
intuit75k['lg_dollars'] = np.log(intuit75k['dollars'])
intuit75k['zip_801'] = (intuit75k['zip'] =='00801').astype(int)
intuit75k['zip_804'] = (intuit75k['zip'] =='00804').astype(int)

f, axes = plt.subplots(1, 2, figsize=(10, 5))
sns.histplot(intuit75k['lg_dollars'], color="teal", ax=axes[0])
sns.histplot(intuit75k['dollars'], color="teal", ax=axes[1])
plt.show()
# Create 3-factor label
intuit75k['only_ver02'] = np.where((intuit75k['upgraded'] == 0) & (intuit75k['version1'] == 0), 1, 0)
intuit75k['from_v1_to_v2'] = np.where((intuit75k['upgraded'] == 1) & (intuit75k['version1'] == 0), 1, 0)
intuit75k['only_ver01'] = np.where((intuit75k['upgraded'] == 0) & (intuit75k['version1'] == 1), 1, 0)
#breakeven & split the data
breakeven_rate = 1.41/60

train_cols = ['numords' , 'zip_bins_1', 'zip_801', 'zip_804', 'lg_dollars', 'last', 'only_ver02', 'from_v1_to_v2', 'only_ver01' ]
rvar = "res1_yes"
idvar = "id"

train_non = intuit75k[intuit75k['training'] == 1]
test_non = intuit75k[intuit75k['training'] == 0]

Xnon_train = train_non[train_cols].values
y_train = train_non[rvar].values
Xnon_test = test_non[train_cols].values
y_test = test_non[rvar].values

Xnon = intuit75k[train_cols].values
from sklearn.preprocessing import MinMaxScaler

evar_to_transfer = intuit75k[train_cols]#.values
scaler = MinMaxScaler()
intuit75k[train_cols] = scaler.fit_transform(evar_to_transfer)
train = intuit75k[intuit75k['training'] == 1]
test = intuit75k[intuit75k['training'] == 0]

X = intuit75k[train_cols].values
y = intuit75k[rvar].values

X_train = train[train_cols].values
X_test = test[train_cols].values

test_result = test[['id', 'res1']].copy() #store the model results
def pref_calc_ad(perc_mail, rep_rate):
    nr_mail = (22500-1103) * perc_mail
    nr_resp = nr_mail* rep_rate *0.5
    revenue = nr_resp * 60 
    mail_cost = nr_mail * 1.41
    profit = revenue  - mail_cost
    ROME = profit / mail_cost 
    return ( f"""the number of businesses to mail is {int(nr_mail):,} ({round((100 * perc_mail), 2)}%). The response rate for the selected customers is predicted to be {round((100 * rep_rate), 2)}% or {int(nr_resp):,} buyers. The revenue is equal to ${int(revenue):,}. The expected profit is ${int(profit):,}. The marginal cost is estimated to be ${int(mail_cost):,} with a ROME of {round((100 * ROME), 2)}%""")<jupyter_output><empty_output><jupyter_text>3. Redo Logistic Regression<jupyter_code>lr_mod= smf.glm(
    formula="res1_yes ~ numords  + zip_bins_1 + zip_801 + zip_804 + lg_dollars + last + only_ver02 + from_v1_to_v2 + only_ver01  ",
    family=Binomial(link=logit()),
    data=intuit75k[intuit75k['training']==1],
)
lr_sel = lr_mod.fit()
lr_sel.summary()
pred_lr = lr_sel.predict(test)

test_result['prob_lg'] = pred_lr
test_result['pred_lg'] = (pred_lr/2 > breakeven_rate)

perc = test_result['pred_lg'].mean()
resp = functions.response_rate(test_result[test_result['pred_lg']==True]['res1'], lev='Yes')

print(f"""By logistic regression model, {pref_calc_ad(perc, resp)}""")<jupyter_output>By logistic regression model, the number of businesses to mail is 6,260 (29.26%). The response rate for the selected customers is predicted to be 11.36% or 355 buyers. The revenue is equal to $21,339. The expected profit is $12,512. The marginal cost is estimated to be $8,827 with a ROME of 141.76%<jupyter_text>Logistic regression resultThe expected profit (446,995) is higher than our highest expected profit of logistic regression last time (321k)<jupyter_code>#last weeks function
functions.pref_calc_logit(perc, resp)<jupyter_output><empty_output><jupyter_text>4. Keras Neural Network<jupyter_code>from keras.layers import Dense
from keras.models import Sequential
from keras.utils.np_utils import to_categorical
from keras.callbacks import EarlyStopping
predictors = X_train.copy()
target = to_categorical(train['res1_yes'])
n_cols = predictors.shape[1]

predict_data = X_test.copy()
test_label = to_categorical(test['res1_yes'])<jupyter_output><empty_output><jupyter_text>Neural Network Model 1 (100 nodes, 6 layers)<jupyter_code>from numpy.random import seed
seed(1234)
import tensorflow
tensorflow.random.set_seed(122)


model = Sequential()
model.add(Dense(100, activation='relu', input_shape = (n_cols,)))
model.add(Dense(100, activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(2, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy',
              metrics=['accuracy'])

early_stopping_monitor = EarlyStopping(patience=2)

model.fit(predictors, target, validation_split=0.3, epochs=20,           
          callbacks = [early_stopping_monitor])
predictions_k1 = model.predict(predict_data)
probability_true_k1 = predictions_k1[:,1]

test_result['prob_k1'] = probability_true_k1
test_result['pred_k1'] = probability_true_k1/2 > breakeven_rate

perc_nn = test_result['pred_k1'].mean()
rep_nn = functions.response_rate(test_result[test_result['pred_k1']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['pred_k1'], test_result['res1'])
conf_nn
print(f"""By Keras neural network model 1, with 100 nodes, {pref_calc_ad(perc_nn, rep_nn)}""")<jupyter_output>By Keras neural network model 1, with 100 nodes, the number of businesses to mail is 5,123 (23.95%). The response rate for the selected customers is predicted to be 12.69% or 325 buyers. The revenue is equal to $19,514. The expected profit is $12,289. The marginal cost is estimated to be $7,224 with a ROME of 170.1%<jupyter_text>Neural Network Model 2 (200 nodes, 6 layers)<jupyter_code>from numpy.random import seed
seed(122)
import tensorflow
tensorflow.random.set_seed(122)

model = Sequential()
model.add(Dense(200, activation='relu', input_shape = (n_cols,)))
model.add(Dense(200, activation='relu'))
model.add(Dense(200, activation='relu'))
model.add(Dense(200, activation='relu'))
model.add(Dense(200, activation='relu'))
model.add(Dense(2, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy',
              metrics=['accuracy'])

early_stopping_monitor = EarlyStopping(patience=2)

model.fit(predictors, target, validation_split=0.3, epochs=20,           
          callbacks = [early_stopping_monitor])
predictions_k2 = model.predict(predict_data)
probability_true_k2 = predictions_k2[:,1]

test_result['prob_k2'] = probability_true_k2
test_result['pred_k2'] = probability_true_k2/2 > breakeven_rate


perc_nn = test_result['pred_k2'].mean()
rep_nn = functions.response_rate(test_result[test_result['pred_k2']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['pred_k2'], test_result['res1'])
conf_nn
print(f"""By neural network model 2, with 200 nodes, {pref_calc_ad(perc_nn, rep_nn)}""")<jupyter_output>By neural network model 2, with 200 nodes, the number of businesses to mail is 8,904 (41.62%). The response rate for the selected customers is predicted to be 9.09% or 404 buyers. The revenue is equal to $24,278. The expected profit is $11,722. The marginal cost is estimated to be $12,555 with a ROME of 93.36%<jupyter_text>Neural Network Model 3 (200 nodes, 5 layers)<jupyter_code>from numpy.random import seed
seed(122)
import tensorflow
tensorflow.random.set_seed(122)

model = Sequential()
model.add(Dense(200, activation='relu', input_shape = (n_cols,)))
model.add(Dense(200, activation='relu'))
model.add(Dense(200, activation='relu'))
model.add(Dense(200, activation='relu'))
model.add(Dense(2, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy',
              metrics=['accuracy'])

early_stopping_monitor = EarlyStopping(patience=2)

model.fit(predictors, target, validation_split=0.3, epochs=20,           
          callbacks = [early_stopping_monitor])
predictions_k3 = model.predict(predict_data)
probability_true_k3 = predictions_k3[:,1]

test_result['prob_k3'] = probability_true_k3
test_result['pred_k3'] = probability_true_k3/2 > breakeven_rate


perc_nn = test_result['pred_k3'].mean()
rep_nn = functions.response_rate(test_result[test_result['pred_k3']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['pred_k3'], test_result['res1'])
conf_nn
print(f"""By neural network model 3, with 200 nodes, {pref_calc_ad(perc_nn, rep_nn)}""")<jupyter_output>By neural network model 3, with 200 nodes, the number of businesses to mail is 8,006 (37.42%). The response rate for the selected customers is predicted to be 9.67% or 387 buyers. The revenue is equal to $23,222. The expected profit is $11,934. The marginal cost is estimated to be $11,288 with a ROME of 105.72%<jupyter_text>Neural Network Model 4 (100 nodes, 7 layers)<jupyter_code>from numpy.random import seed
seed(122)
import tensorflow
tensorflow.random.set_seed(122)

model = Sequential()
model.add(Dense(100, activation='relu', input_shape = (n_cols,)))
model.add(Dense(100, activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(2, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy',
              metrics=['accuracy'])

early_stopping_monitor = EarlyStopping(patience=2)

model.fit(predictors, target, validation_split=0.3, epochs=20,           
          callbacks = [early_stopping_monitor])
predictions_k4 = model.predict(predict_data)
probability_true_k4 = predictions_k4[:,1]

test_result['prob_k4'] = probability_true_k4
test_result['pred_k4'] = probability_true_k4/2 > breakeven_rate


perc_nn = test_result['pred_k4'].mean()
rep_nn = functions.response_rate(test_result[test_result['pred_k4']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['pred_k4'], test_result['res1'])
conf_nn
print(f"""By neural network model 4, with 100 nodes, {pref_calc_ad(perc_nn, rep_nn)}""")<jupyter_output>By neural network model 4, with 100 nodes, the number of businesses to mail is 5,027 (23.5%). The response rate for the selected customers is predicted to be 12.79% or 321 buyers. The revenue is equal to $19,285. The expected profit is $12,196. The marginal cost is estimated to be $7,089 with a ROME of 172.04%<jupyter_text>Neural Network Model 5 (200 nodes, 4 layers)<jupyter_code>from numpy.random import seed
seed(122)
import tensorflow
tensorflow.random.set_seed(122)

model = Sequential()
model.add(Dense(200, activation='relu', input_shape = (n_cols,)))
model.add(Dense(200, activation='relu'))
model.add(Dense(200, activation='relu'))
model.add(Dense(2, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy',
              metrics=['accuracy'])

early_stopping_monitor = EarlyStopping(patience=2)

model.fit(predictors, target, validation_split=0.3, epochs=20,           
          callbacks = [early_stopping_monitor])
predictions_k5 = model.predict(predict_data)
probability_true_k5 = predictions_k5[:,1]

test_result['prob_k5'] = probability_true_k5
test_result['pred_k5'] = probability_true_k5/2 > breakeven_rate


perc_nn = test_result['pred_k5'].mean()
rep_nn = functions.response_rate(test_result[test_result['pred_k5']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['pred_k5'], test_result['res1'])
conf_nn
print(f"""By neural network model 5, with 100 nodes, {pref_calc_ad(perc_nn, rep_nn)}""")<jupyter_output>By neural network model 5, with 100 nodes, the number of businesses to mail is 7,542 (35.25%). The response rate for the selected customers is predicted to be 10.12% or 381 buyers. The revenue is equal to $22,909. The expected profit is $12,274. The marginal cost is estimated to be $10,634 with a ROME of 115.42%<jupyter_text>5. MLP Neural Network<jupyter_code>from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.inspection import permutation_importance<jupyter_output><empty_output><jupyter_text>MLP Model 1: hlsize = (1,), alpha = 0.01<jupyter_code>clf_m1 = MLPClassifier(activation = "tanh",
                       solver = "lbfgs",
                       alpha = 0.01,
                       hidden_layer_sizes = (1,),
                       random_state = 1234,
                       max_iter = 1000).fit(X_train, y_train)
test_result['prob_m1'] = clf_m1.predict_proba(X_test)[:,1]
test_result['pred_m1'] = test_result['prob_m1']/2 > breakeven_rate

perc_mlp = test_result['pred_m1'].mean()
rep_mlp = functions.response_rate(test_result[test_result['pred_m1']==True]['res1'], lev='Yes')

conf_mlp = pd.crosstab(test_result['pred_m1'], test_result['res1'])
conf_mlp
print(f'''With 1-node MLP model: {pref_calc_ad(perc_mlp, rep_mlp)}''')<jupyter_output>With 1-node MLP model: the number of businesses to mail is 5,907 (27.61%). The response rate for the selected customers is predicted to be 11.75% or 347 buyers. The revenue is equal to $20,826. The expected profit is $12,496. The marginal cost is estimated to be $8,329 with a ROME of 150.03%<jupyter_text>MLP Model 2: hlsize = (2,), alpha = 0.01<jupyter_code>clf_m2 = MLPClassifier(activation = "tanh",
                       solver = "lbfgs",
                       alpha = 0.01,
                       hidden_layer_sizes = (2,),
                       random_state = 1234,
                       max_iter = 1000).fit(X_train, y_train)
test_result['prob_m2'] = clf_m2.predict_proba(X_test)[:,1]
test_result['pred_m2'] = test_result['prob_m2']/2 > breakeven_rate

perc_mlp = test_result['pred_m2'].mean()
rep_mlp = functions.response_rate(test_result[test_result['pred_m2']==True]['res1'], lev='Yes')

conf_mlp = pd.crosstab(test_result['pred_m2'], test_result['res1'])
conf_mlp
print(f'''With 2 node and 2 hidden layers' MLP model: {pref_calc_ad(perc_mlp, rep_mlp)}''')<jupyter_output>With 2 node and 2 hidden layers' MLP model: the number of businesses to mail is 6,337 (29.62%). The response rate for the selected customers is predicted to be 11.22% or 355 buyers. The revenue is equal to $21,339. The expected profit is $12,404. The marginal cost is estimated to be $8,935 with a ROME of 138.82%<jupyter_text>MLP Model3: Grid Search<jupyter_code>mlp = MLPClassifier(activation="tanh",
                       solver="lbfgs",
                       alpha=0.01,
                       hidden_layer_sizes=(1,),
                       random_state=1234,
                       max_iter=1000,
                      )#.fit(X_train, y_train)

nr_hnodes = range(1, 5)
hls = list(zip(nr_hnodes)) + list(zip(nr_hnodes, nr_hnodes))

param_grid_mlp = {"hidden_layer_sizes": hls, 
                  "alpha": [0.001, 0.01, 0.05]}

scoring = {"AUC": "roc_auc"}

clf_cv = GridSearchCV(mlp, 
                      param_grid_mlp, 
                      scoring=scoring, 
                      cv=5, 
                      n_jobs=4, 
                      refit="AUC", 
                      verbose=5)

clf_cv.fit(X_train, y_train)
def importance(clf, X, y):
    imp = permutation_importance(clf, X, y, scoring='roc_auc', n_repeats=10, random_state=1234)
    
    data = pd.DataFrame(imp.importances.T)
    data.columns = train_cols
    fig = sns.barplot(x="variable", y='value', color='slateblue', data=pd.melt(data))
    fig.set_xticklabels(fig.get_xticklabels(), rotation=75)
    fig.set(title ='Permutation Importance')
    
print("Best parameters found: ", clf_cv.best_params_)
print("Best AUC found: ", clf_cv.best_score_)

importance(clf_cv, X_train, y_train)
pred_mlp = clf_cv.predict_proba(X_test)

test_result['prob_m3'] = pred_mlp[:,1]
test_result['pred_m3'] = test_result['prob_m3']/2 > breakeven_rate

perc_mlp = test_result['pred_m3'].mean()
rep_mlp = functions.response_rate(test_result[test_result['pred_m3']==True]['res1'], lev='Yes')

conf_mlp = pd.crosstab(test_result['pred_m3'], test_result['res1'])
conf_mlp
print(f'''With 2 node and 2 hidden layers' MLP model: {pref_calc_ad(perc_mlp, rep_mlp)}''')
#test_result<jupyter_output><empty_output><jupyter_text>6. XGBoost Classifier  Untuned XGBoost<jupyter_code>import xgboost as xgb

xg_cl_0 = xgb.XGBClassifier(objective='binary:logistic',
                          n_estimators=10, seed=122)

xg_cl_0.fit(Xnon_train, y_train)
preds_0 = xg_cl_0.predict_proba(Xnon_test)
#accuracy = float(np.sum(preds==y_test))/y_test.shape[0]
#print("accuracy: %f" % (accuracy))
test_result['prob_xgb1'] = preds_0[:,1]
test_result['pred_xgb1'] = test_result['prob_xgb1']/2 > breakeven_rate

perc_nn_0 = test_result['pred_xgb1'].mean()
rep_nn_0 = functions.response_rate(test_result[test_result['pred_xgb1']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['pred_xgb1'], test_result['res1'])
conf_nn
print(f"""By untuned XGBoost model, {pref_calc_ad(perc_nn_0, rep_nn_0)}""")<jupyter_output>By untuned XGBoost model, the number of businesses to mail is 10,663 (49.84%). The response rate for the selected customers is predicted to be 7.91% or 421 buyers. The revenue is equal to $25,305. The expected profit is $10,270. The marginal cost is estimated to be $15,035 with a ROME of 68.31%<jupyter_text>Tuning using GridSearchCV<jupyter_code>from sklearn.model_selection import GridSearchCV

# Create the parameter grid: gbm_param_grid
gbm_param_grid = {
    "objective":['reg:logistic'],
    'colsample_bytree': [0.3, 0.7, 1],
    'n_estimators': [50, 100, 150],
    'max_depth': [2, 5, 7, 10],
    'eta':[0.01,0.1,0.5,0.9]
}

# Instantiate the regressor: gbm
gbm = xgb.XGBClassifier(use_label_encoder = False)

# Perform grid search: grid_mse
grid_auc = GridSearchCV(estimator=gbm, 
                        param_grid=gbm_param_grid, 
                        scoring='roc_auc', cv=4, verbose=1)


# Fit grid_mse to the data
grid_auc.fit(Xnon_train, y_train)

# Print the best parameters and lowest RMSE
print("Best parameters found: ", grid_auc.best_params_)
print("Best ROC_AUC found: ", grid_auc.best_score_)
preds_1 = grid_auc.predict_proba(Xnon_test)


test_result['prob_xgb2'] = preds_1[:,1]
test_result['pred_xgb2'] = test_result['prob_xgb2']/2 > breakeven_rate

perc_xgb = test_result['pred_xgb2'].mean()
rep_xgb = functions.response_rate(test_result[test_result['pred_xgb2']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['pred_xgb2'], test_result['res1'])
conf_nn
print(f"""By tuned XGBoost model, {pref_calc_ad(perc_xgb, rep_xgb)}""")<jupyter_output>By tuned XGBoost model, the number of businesses to mail is 5,933 (27.73%). The response rate for the selected customers is predicted to be 11.83% or 350 buyers. The revenue is equal to $21,054. The expected profit is $12,688. The marginal cost is estimated to be $8,365 with a ROME of 151.68%<jupyter_text>7. Random forest classifier<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV

n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]


from sklearn.ensemble import RandomForestClassifier

rfc_param_grid = {
    "max_features" : ['auto', 'sqrt'],
    'n_estimators': [50, 100],
    'max_depth': [2, 5, 10],
    'min_samples_split': [2, 5],
    'min_samples_leaf': [1, 2, 4],
    'bootstrap': [True, False]

}


# Instantiate the regressor: gbm
rfc = RandomForestClassifier(random_state=122)

# Perform grid search: grid_mse
grid_auc_rfc = GridSearchCV(estimator=rfc, 
                        param_grid=rfc_param_grid, 
                        scoring='roc_auc', cv=4, verbose=1)


# Fit grid_mse to the data
grid_auc_rfc.fit(Xnon_train, y_train)

# Print the best parameters and lowest RMSE
print("Best parameters found: ", grid_auc_rfc.best_params_)
print("Best ROC_AUC found: ", grid_auc_rfc.best_score_)
preds_r = grid_auc_rfc.predict_proba(Xnon_test)

test_result['prob_rfc1'] = preds_r[:,1]
test_result['pred_rfc1'] = test_result['prob_rfc1']/2 > breakeven_rate

perc_nn = test_result['pred_rfc1'].mean()
rep_nn = functions.response_rate(test_result[test_result['pred_rfc1']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['pred_rfc1'], test_result['res1'])
conf_nn
print(f"""By Random Forest model, {pref_calc_ad(perc_nn, rep_nn)}""")
#data=pd.read_csv("test_result_all.csv")
#data[['prob_xgb2', 'pred_xgb2', 'prob_rfc1', 'pred_rfc1']]

#test_result.loc[:, 'prob_xgb2'] = data['prob_xgb2'].values
#test_result.loc[:, 'pred_xgb2'] = data['pred_xgb2'].values
#test_result.loc[:, 'prob_rfc1'] = data['prob_rfc1'].values
#test_result.loc[:, 'pred_rfc1'] = data['pred_rfc1'].values
#test_result
#test_result<jupyter_output><empty_output><jupyter_text>7. Decision tree classifier<jupyter_code>from sklearn.tree import DecisionTreeClassifier
from sklearn import metrics, tree
dt = DecisionTreeClassifier(max_depth=2, 
                           random_state=1234).fit(Xnon_train, y_train)
fig = tree.plot_tree(dt, 
                     feature_names=train_cols, 
                     class_names=["yes", "no"], 
                     filled=True)
dt_clf = DecisionTreeClassifier(max_depth=3, 
                           random_state=1234).fit(Xnon_train, y_train)

test_result['prob_dt1'] = dt_clf.predict_proba(Xnon_test)[:,1]
test_result['pred_dt1'] = test_result['prob_dt1']/2 > breakeven_rate

perc_dt = test_result['pred_dt1'].mean()
rep_dt = functions.response_rate(test_result[test_result['pred_dt1']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['pred_dt1'], test_result['res1'])
conf_nn
dt_importance = pd.Series(data=dt_clf.feature_importances_, index=train_cols)
dt_importance_sorted = dt_importance.sort_values()
dt_importance_sorted.plot(kind='barh', 
                          color='lightgreen', 
                          title='Features Importances')
print(f"""By untuned decision tree model, {pref_calc_ad(perc_dt, rep_dt)}""")
dt_param_grid = {'max_depth': list(range(2, 20))}

clf_dt = DecisionTreeClassifier(max_depth=20)#.fit(X_train, y_train)

grid_auc_dt = GridSearchCV(clf_dt, 
                           dt_param_grid, 
                           scoring=scoring, 
                           cv=5, 
                           n_jobs=4, 
                           refit="AUC", 
                           verbose=5).fit(Xnon_train, y_train)

print("Tuned Parameters: {}".format(grid_auc_dt.best_params_))
print("Best score is {}".format(grid_auc_dt.best_score_))
test_result['prob_dt2'] = grid_auc_dt.predict_proba(Xnon_test)[:,1]
test_result['pred_dt2'] = test_result['prob_dt2']/2 > breakeven_rate


perc_dt = test_result['pred_dt2'].mean()
rep_dt = functions.response_rate(test_result[test_result['pred_dt2']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['pred_dt2'], test_result['res1'])
conf_nn
print(f"""By tuned decision tree model, {pref_calc_ad(perc_dt, rep_dt)}""")<jupyter_output>By tuned decision tree model, the number of businesses to mail is 6,788 (31.72%). The response rate for the selected customers is predicted to be 10.54% or 357 buyers. The revenue is equal to $21,454. The expected profit is $11,882. The marginal cost is estimated to be $9,571 with a ROME of 124.15%<jupyter_text>Comparing model performance<jupyter_code>#test_result = pd.read_csv("test_result.csv")
models = []
for col in test_result.columns:
    if col[0:4] == 'prob':
        models.append(col)
models
# summarize performance across all models in test set
test_result['res1_yes'] = (test_result['res1'] =='Yes').astype(int)
rsm.evalbin(test_result, "res1_yes", 1, models, cost=1.41, margin=30)
avg_mod = ['prob_k5', 'prob_m1', 'prob_xgb2', 'prob_rfc1', 'prob_dt2']
breakeven_rate = 1.41/60
test_result['avg_prob'] = test_result[avg_mod].sum(axis=1)/len(avg_mod)
test_result['avg_pred'] = test_result['avg_prob']/2 > breakeven_rate

perc_avg = test_result['avg_pred'].mean()
rep_avg = functions.response_rate(test_result[test_result['avg_pred']==True]['res1'], lev='Yes')

conf_nn = pd.crosstab(test_result['avg_pred'], test_result['res1'])
conf_nn
print(f"""By average all model, {pref_calc_ad(perc_avg, rep_avg)}""")
#test_result.to_csv("test_result_new.csv", index=False)
#test_result=pd.read_csv("test_result_new.csv")
#test_result['mailto_wave2'] = (test_result['avg_pred'] == True) & (test_result['res1'] == 'No')# & (test_result['avg_pred'] == True)
#mail = test_result[['id','mailto_wave2']].to_csv('mailto_wave2.csv', index = False)

#len(test_result[['id','mailto_wave2']])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13500687307717485, 'fraction_numerical': 0.04267853636185115, 'mean_word_length': 5.393596986817326, 'pattern_counts': {'":': 4, '<': 69, '<?xml version=': 0, '>': 84, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '652890', 'license': 'Apache-2.0', 'n_tokens_mistral': 11656, 'n_tokens_neox': 10735, 'n_words': 2561, 'path': 'intuit-redux.ipynb', 'repo_name': 'hollylessthan/Modeling-upsell-response-with-Deep-Learning', 'size': '30554'}