<jupyter_start><jupyter_text>Training a modelHow will we judge the success of the cancellation classifier? Let's look at a number of measures:  -The confusion matrix    True Negative  |  False Positive    False Negative |  True Positive  -Precision, True positive/(True positive + False positive).  -Recall, True positive/(True positive + False negative).  -F score, 2PR/(P+R).  -Accuracy score, percentage of our guesses correct.We can also assign a cost saved, using the previous numbers of +\$8000 for a true positive, -\$825 for a false positive.  (This will give us the cost saved compared to the case if all flights were run).<jupyter_code>def cost_saved(conf_matrix):
    cost = conf_matrix[1][1]*8000 - conf_matrix[0][1]*825
    return(cost)
import warnings
warnings.filterwarnings('ignore')
import gc

import pandas as pd

model_data = pd.read_hdf('../data/processed/model_data.hdf')

print('We have {0} samples and {1} features'.format(model_data.shape[0],model_data.shape[1]-1))<jupyter_output>We have 5819079 samples and 124 features<jupyter_text>This is too large a data set to train with my laptop's memory, so let's only look at a randomly selected subset (10% ~=600,000 rows) and a different, equally sized subset for cross validation.<jupyter_code>shuffled_data = model_data.sample(frac=0.2)
cv_data = shuffled_data.iloc[:int(len(shuffled_data)/2),:]
shuffled_data = shuffled_data.iloc[int(len(shuffled_data)/2):,:]<jupyter_output><empty_output><jupyter_text>To begin let's look only at the features generated from the original data.  Logistic regressionLet's start with the simplest algorithm first: logistic regression with the default L2 penalty, with the data split 9:1 for train:test and feature scaling applied.  As we have few positives let's add 'balanced' class scaling, which will emphasise the features which contribute most to the positives.<jupyter_code>import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.cross_validation import KFold
from sklearn.metrics import confusion_matrix, precision_recall_fscore_support, accuracy_score

alg = LogisticRegression(class_weight='balanced')
#Features from original data
predictors = list(model_data.columns[2:64])+list(model_data.columns[66:81])

#Split the data into 10 subsets
kf = KFold(shuffled_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (shuffled_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = shuffled_data['Cancelled'].iloc[train]
    alg.fit(scaler.transform(X_train), y_train)
    X_test = (shuffled_data[predictors].iloc[test,:])
    y_test = shuffled_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(shuffled_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(shuffled_data['Cancelled'].values, predictions)
print(('precision = {0:.2f}, recall = {1:.2f}, F1 = {2:.2f}, ' 
       'accuracy = {3:.2f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(shuffled_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))<jupyter_output>Confusion matrix
        0       1
0  377685  195181
1    2729    6313
precision = 0.03, recall = 0.70, F1 = 0.06, accuracy = 0.66, saved cost per flight = $-189.93<jupyter_text>Our recall is pretty high, so we are identifying most of the actual cancellations.  However the precision is very low as we are generating a lot of false positives, so with this model out of every ~33 flights not run only one would have actually been cancelled.  Because of these false positives, this model results in a loss of -$190 per flight.Now let's try regularising the regression to see if, by reducing the variance, we can get any improvement on cost.<jupyter_code>#Try these Cs
gc.collect()
C_list = [1e-7, 1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1e0, 1e1, 1e2, 1e3]

predictors = list(model_data.columns[2:64])+list(model_data.columns[66:81])

conf_matrix = {}
coef = {}
precision = []; recall = []; F1 = []; accuracy = []; saved_cost = []
for C in C_list:
    alg = LogisticRegression(class_weight='balanced', C=C)
    
    #Split the data into 10 subsets
    kf = KFold(shuffled_data.shape[0], n_folds = 10, random_state = 1)
    predictions = []

    #Train and predict
    for train, test in kf:
        X_train = (shuffled_data[predictors].iloc[train,:])
        scaler = StandardScaler().fit(X_train)
        y_train = shuffled_data['Cancelled'].iloc[train]
        alg.fit(scaler.transform(X_train), y_train)
        X_test = (shuffled_data[predictors].iloc[test,:])
        y_test = shuffled_data['Cancelled'].iloc[test]
        test_predictions = alg.predict(scaler.transform(X_test))
        predictions.append(test_predictions)
        gc.collect()

    predictions = np.concatenate(predictions,axis = 0)
    coef[C] = alg.coef_
    conf_matrix[C] = confusion_matrix(shuffled_data['Cancelled'].values, predictions)
    print('C = {0}\nConfusion matrix'.format(C))
    print(pd.DataFrame(conf_matrix[C]))
    
    scores = precision_recall_fscore_support(shuffled_data['Cancelled'].values, predictions)
    precision.append(scores[0][1]); recall.append(scores[1][1]); F1.append(scores[2][1])
    accuracy.append(accuracy_score(shuffled_data['Cancelled'].values, predictions))
    saved_cost.append(cost_saved(conf_matrix[C])/len(predictions))
    
    print(('precision = {0:.2f}, recall = {1:.2f}, F1 = {2:.2f}, ' 
       'accuracy = {3:.2f}, saved cost per flight = ${4:.2f}').format(precision[-1],recall[-1],F1[-1],
                                                                   accuracy[-1],saved_cost[-1]))
    
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib.ticker import ScalarFormatter
y_formatter = ScalarFormatter(useOffset=True)

fig = plt.figure(figsize=(13,3))
ax1 = fig.add_subplot(131)
ax1.plot(C_list,precision,'-or')
ax1.set_xlabel('C')
ax1.set_xscale('log')
ax1.set_ylabel('Precision', color='r')
ax1.set_title('Precision and recall for\n logistic regression')
for tl in ax1.get_yticklabels():
    tl.set_color('r')

ax2 = ax1.twinx()
ax2.plot(C_list, recall, '-ob')
ax2.set_ylabel('Recall', color='b')
for tl in ax2.get_yticklabels():
    tl.set_color('b')
    
ax3 = fig.add_subplot(132)
ax3.plot(C_list,F1,'-or')
ax3.set_xlabel('C')
ax3.set_xscale('log')
ax3.set_ylabel('F1', color='r')
ax3.set_title('F1 and accuracy for\n logistic regression')
for tl in ax3.get_yticklabels():
    tl.set_color('r')

ax4 = ax3.twinx()
ax4.plot(C_list, accuracy, '-ob')
ax4.set_ylabel('Accuracy', color='b')
ax4.yaxis.set_major_formatter(y_formatter)
for tl in ax4.get_yticklabels():
    tl.set_color('b')
    
ax5 = fig.add_subplot(133)
ax5.plot(C_list,saved_cost,'-or')
ax5.set_xlabel('min leaf size')
ax5.set_xscale('log')
ax5.set_ylabel('Cost ($)',color='r')
ax5.set_title('Cost saved per flight\n for logistic regression')
for tl in ax5.get_yticklabels():
    tl.set_color('r')

num_pos = [(conf_matrix[C][0][1]+conf_matrix[C][1][1])*100/sum(sum(conf_matrix[C]))
           for C in C_list]
ax6 = ax5.twinx()
ax6.plot(C_list, num_pos, '-ob')
ax6.set_ylabel('% guessed positive', color='b')
ax6.yaxis.set_major_formatter(y_formatter)
for tl in ax6.get_yticklabels():
    tl.set_color('b')

fig.tight_layout()
fig.savefig('../reports/figures/classification/log_reg_C', bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>Increasing the regularisation (decreasing C) helps the recall further by introducing more bias, but at the detriment of the precision, with a balance between the two at C=1e-5.  Decreasing the regularisation (increasing C) has little effect on either precision or recall, so the best cost is that we first found for C=1. Let's look at which features were the most important:<jupyter_code>for index in np.argsort(abs(coef[1]))[0][:-11:-1]:
    print(predictors[index]+': '+str(coef[1][0][index]))<jupyter_output>Days after winter storm: -0.661352820195
DL: -0.372698497111
MQ: 0.252823853387
AS: -0.213855856748
Distance: -0.187959239687
Week in year: 0.186754481363
HA: -0.183430034978
Days from public event: 0.161721224789
Origin NY: 0.148940697566
EV: 0.146426771238<jupyter_text>So, as might be expected, knowing when the large storms were is the most important. As New York was one of the worst hit, it is also has a positive trend for cancellations.Other temporal features are also important: there are more cancellations at the start of the week, and after public events.Perhaps more surprisingly, flight distance is also high up: longer flights are cancelled less.For the carriers, Delta Airlines (DL) cancels less while American Eagle (MQ) cancels more. This agrees with what we saw in our explanatory data analysis:<jupyter_code>carriers = pd.read_csv('../data/raw/carriersnew.csv')
carriers.dropna(how='any', inplace=True)
carriers.reset_index(inplace=True)
carriers.drop('index', axis=1, inplace=True)
carriers.set_index('Code', inplace=True)
carriers.ix['US','Description'] = 'US Airways Inc.'
carriers = carriers[~carriers['number of flights'].isnull()]
carriers.sort_values(by='number of flights', inplace=True, ascending=False)
carriers['Trend'] = [coef[1][0][np.array(predictors)==index][0] for index in carriers.index]
import colorsys

N = len(carriers)
HSV_tuples = [(x*1.0/N, 1, 1) for x in range(N)]
RGB_tuples = [colorsys.hsv_to_rgb(*x) for x in HSV_tuples]
col_dict = {}
for i in range(N):
    col_dict[carriers.index[i]] = RGB_tuples[i]

fig = plt.figure(figsize=(6,3))
#Plot the number of flights
ax1 = fig.add_subplot(111)
bars = ax1.bar(np.arange(0,N), -carriers['Trend'].values)
for i in range(N):
    bars[i].set_color(col_dict[carriers.index[i]])
ax1.set_ylabel('Logistic regression coefficient')
ax1.set_xticks(np.arange(0,N)+0.5)
ax1.set_xticklabels(carriers['Description'].values, rotation=45, ha='right')
ax1.set_title('Trend not to cancel by carrier')

fig.savefig('../reports/figures/classification/log_reg_carrier', bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>So the trends for each carrier picked out by logistic regression agree with those we saw in the exploratory data analysis.Now let's try an algorithm we'd expect to perform rather better. Now let's look at the performance on the test set.<jupyter_code>alg = LogisticRegression(class_weight='balanced', C=1)
#Features from original data
predictors = list(model_data.columns[2:64])+list(model_data.columns[66:81])

#Split the data into 10 subsets
kf = KFold(cv_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (cv_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = cv_data['Cancelled'].iloc[train]
    alg.fit(scaler.transform(X_train), y_train)
    X_test = (cv_data[predictors].iloc[test,:])
    y_test = cv_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(cv_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(cv_data['Cancelled'].values, predictions)
print(('precision = {0:.2f}, recall = {1:.2f}, F1 = {2:.2f}, ' 
       'accuracy = {3:.2f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(cv_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))<jupyter_output>Confusion matrix
        0       1
0  382825  190286
1    2683    6114
precision = 0.03, recall = 0.70, F1 = 0.06, accuracy = 0.67, saved cost per flight = $-185.72<jupyter_text>Random forest classificationLet's try a random forest classifier on the data, again doing a 9:1 train:test split on 10% of the total data.<jupyter_code>from sklearn.ensemble import RandomForestClassifier

alg = RandomForestClassifier()
predictors = list(model_data.columns[2:64])+list(model_data.columns[66:81])

#Split the data into 10 subsets
kf = KFold(shuffled_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (shuffled_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = shuffled_data['Cancelled'].iloc[train]
    alg.fit(scaler.transform(X_train), y_train)
    X_test = (shuffled_data[predictors].iloc[test,:])
    y_test = shuffled_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(shuffled_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(shuffled_data['Cancelled'].values, predictions)
print(('precision = {0:.2f}, recall = {1:.2f}, F1 = {2:.2f}, ' 
       'accuracy = {3:.2f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(shuffled_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))<jupyter_output><empty_output><jupyter_text>So for random forest our recall is far smaller but out precision is far higher: we're predicting fewer cancellations, but with much higher accuracy.  This means we can now making a saving by not running the identified flights.Now let's try and optimise the classifier.<jupyter_code>#Try these leaves
leaf_list = [1,2,3,5,7,10]

predictors = list(model_data.columns[2:64])+list(model_data.columns[66:81])

conf_matrix = {}
coef = {}
precision = []; recall = []; F1 = []; accuracy = []; saved_cost = []
for leaf in leaf_list:
    alg = RandomForestClassifier(min_samples_leaf=leaf)

    #Split the data into 10 subsets
    kf = KFold(shuffled_data.shape[0], n_folds = 10, random_state = 1)
    predictions = []

    #Train and predict
    for train, test in kf:
        X_train = (shuffled_data[predictors].iloc[train,:])
        scaler = StandardScaler().fit(X_train)
        y_train = shuffled_data['Cancelled'].iloc[train]
        alg.fit(scaler.transform(X_train), y_train)
        X_test = (shuffled_data[predictors].iloc[test,:])
        y_test = shuffled_data['Cancelled'].iloc[test]
        test_predictions = alg.predict(scaler.transform(X_test))
        predictions.append(test_predictions)
        gc.collect()

    predictions = np.concatenate(predictions,axis = 0)
    conf_matrix[leaf] = confusion_matrix(shuffled_data['Cancelled'].values, predictions)
    print('Min leaf size = {0}\nConfusion matrix'.format(leaf))
    print(pd.DataFrame(conf_matrix[leaf]))
    
    scores = precision_recall_fscore_support(shuffled_data['Cancelled'].values, predictions)
    precision.append(scores[0][1]); recall.append(scores[1][1]); F1.append(scores[2][1])
    accuracy.append(accuracy_score(shuffled_data['Cancelled'].values, predictions))
    saved_cost.append(cost_saved(conf_matrix[leaf])/len(predictions))
    
    print(('precision = {0:.2f}, recall = {1:.2f}, F1 = {2:.2f}, ' 
       'accuracy = {3:.2f}, saved cost per flight = ${4:.2f}').format(precision[-1],recall[-1],F1[-1],
                                                                   accuracy[-1],saved_cost[-1]))
fig = plt.figure(figsize=(13,3))
ax1 = fig.add_subplot(131)
ax1.plot(leaf_list,precision,'-or')
ax1.set_xlabel('min leaf size')
ax1.set_xscale('log')
ax1.set_ylabel('Precision', color='r')
ax1.set_title('Precision and recall for\n random forest classifier')
for tl in ax1.get_yticklabels():
    tl.set_color('r')

ax2 = ax1.twinx()
ax2.plot(leaf_list, recall, '-ob')
ax2.set_ylabel('Recall', color='b')
for tl in ax2.get_yticklabels():
    tl.set_color('b')
    
ax3 = fig.add_subplot(132)
ax3.plot(leaf_list,F1,'-or')
ax3.set_xlabel('min leaf size')
ax3.set_xscale('log')
ax3.set_ylabel('F1', color='r')
ax3.set_title('F1 and accuracy for\n random forest classifier')
for tl in ax3.get_yticklabels():
    tl.set_color('r')

ax4 = ax3.twinx()
ax4.plot(leaf_list, accuracy, '-ob')
ax4.set_ylabel('Accuracy', color='b')
ax4.yaxis.set_major_formatter(y_formatter)
for tl in ax4.get_yticklabels():
    tl.set_color('b')
    
ax5 = fig.add_subplot(133)
ax5.plot(leaf_list,saved_cost,'-or')
ax5.set_xlabel('min leaf size')
ax5.set_xscale('log')
ax5.set_ylabel('Cost ($)',color='r')
ax5.set_title('Cost saved per flight\n for random forest classifier')
for tl in ax5.get_yticklabels():
    tl.set_color('r')

num_pos = [(conf_matrix[leaf][0][1]+conf_matrix[leaf][1][1])*100/sum(sum(conf_matrix[leaf]))
           for leaf in leaf_list]
ax6 = ax5.twinx()
ax6.plot(leaf_list, num_pos, '-ob')
ax6.set_ylabel('% guessed positive', color='b')
ax6.yaxis.set_major_formatter(y_formatter)
for tl in ax6.get_yticklabels():
    tl.set_color('b')

fig.tight_layout()
fig.savefig('../reports/figures/classification/rf_reg_trees', bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>Increasing the minimum number of samples on a leaf decreases the number of positive predictions, increasing precision (up to a point) but decreasing recall. Therfore, as a positive is a relatively rare occurence, having no min leaf size gives the best cost.Now let's look at the number of trees, while noting how long each calculation takes.<jupyter_code>#Try these trees
tree_list = [1,3,5,10,30,50]

predictors = list(model_data.columns[2:64])+list(model_data.columns[66:81])

conf_matrix = {}
coef = {}
precision = []; recall = []; F1 = []; accuracy = []; saved_cost = []
for trees in tree_list:
    alg = RandomForestClassifier(n_estimators=trees)

    #Split the data into 10 subsets
    kf = KFold(shuffled_data.shape[0], n_folds = 10, random_state = 1)
    predictions = []

    #Train and predict
    for train, test in kf:
        X_train = (shuffled_data[predictors].iloc[train,:])
        scaler = StandardScaler().fit(X_train)
        y_train = shuffled_data['Cancelled'].iloc[train]
        %time alg.fit(scaler.transform(X_train), y_train)
        X_test = (shuffled_data[predictors].iloc[test,:])
        y_test = shuffled_data['Cancelled'].iloc[test]
        test_predictions = alg.predict(scaler.transform(X_test))
        predictions.append(test_predictions)
        gc.collect()

    predictions = np.concatenate(predictions,axis = 0)
    conf_matrix[trees] = confusion_matrix(shuffled_data['Cancelled'].values, predictions)
    print('Num trees = {0}\nConfusion matrix'.format(trees))
    print(pd.DataFrame(conf_matrix[trees]))
    
    scores = precision_recall_fscore_support(shuffled_data['Cancelled'].values, predictions)
    precision.append(scores[0][1]); recall.append(scores[1][1]); F1.append(scores[2][1])
    accuracy.append(accuracy_score(shuffled_data['Cancelled'].values, predictions))
    saved_cost.append(cost_saved(conf_matrix[trees])/len(predictions))
    
    print(('precision = {0:.2f}, recall = {1:.2f}, F1 = {2:.2f}, ' 
       'accuracy = {3:.2f}, saved cost per flight = ${4:.2f}').format(precision[-1],recall[-1],F1[-1],
                                                                   accuracy[-1],saved_cost[-1]))
fig = plt.figure(figsize=(13,3))
ax1 = fig.add_subplot(131)
ax1.plot(tree_list,precision,'-or')
ax1.set_xlabel('num trees')
ax1.set_xscale('log')
ax1.set_ylabel('Precision', color='r')
ax1.set_title('Precision and recall for\n random forest classifier')
for tl in ax1.get_yticklabels():
    tl.set_color('r')

ax2 = ax1.twinx()
ax2.plot(tree_list, recall, '-ob')
ax2.set_ylabel('Recall', color='b')
for tl in ax2.get_yticklabels():
    tl.set_color('b')
    
ax3 = fig.add_subplot(132)
ax3.plot(tree_list,F1,'-or')
ax3.set_xlabel('num trees')
ax3.set_xscale('log')
ax3.set_ylabel('F1', color='r')
ax3.set_title('F1 and accuracy for\n random forest classifier')
for tl in ax3.get_yticklabels():
    tl.set_color('r')

ax4 = ax3.twinx()
ax4.plot(tree_list, accuracy, '-ob')
ax4.set_ylabel('Accuracy', color='b')
ax4.yaxis.set_major_formatter(y_formatter)
for tl in ax4.get_yticklabels():
    tl.set_color('b')
    
ax5 = fig.add_subplot(133)
ax5.plot(tree_list,saved_cost,'-or')
ax5.set_xlabel('num trees')
ax5.set_xscale('log')
ax5.set_ylabel('Cost ($)',color='r')
ax5.set_title('Cost saved per flight\n for random forest classifier')
for tl in ax5.get_yticklabels():
    tl.set_color('r')

num_pos = [(conf_matrix[trees][0][1]+conf_matrix[trees][1][1])*100/sum(sum(conf_matrix[trees]))
           for trees in tree_list]
ax6 = ax5.twinx()
ax6.plot(tree_list, num_pos, '-ob')
ax6.set_ylabel('% guessed positive', color='b')
ax6.yaxis.set_major_formatter(y_formatter)
for tl in ax6.get_yticklabels():
    tl.set_color('b')

fig.tight_layout()
fig.savefig('../reports/figures/classification/rf_reg_trees', bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>With fewer trees we are overfitting the data, so we generate more positives meaning our precision is low but our recall is high.  Up to 10 trees the number of positives predicted decreases, so the precision increases sharply and our recall drops as we decrease the variance in the model. This leads to a decrease the cost saved per flight.  However beyond 10 trees the number of positive predictions is more or less constant, and so by adding more trees we improve the performance of the model and hence the cost.So in fact fewer (3) trees performs better.<jupyter_code>alg = RandomForestClassifier(n_estimators=3)
#Features from original data
predictors = list(model_data.columns[2:64])+list(model_data.columns[66:81])

#Split the data into 10 subsets
kf = KFold(cv_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (cv_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = cv_data['Cancelled'].iloc[train]
    alg.fit(scaler.transform(X_train), y_train)
    X_test = (cv_data[predictors].iloc[test,:])
    y_test = cv_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(cv_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(cv_data['Cancelled'].values, predictions)
print(('precision = {0:.2f}, recall = {1:.2f}, F1 = {2:.2f}, ' 
       'accuracy = {3:.4f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(cv_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))<jupyter_output>Confusion matrix
        0     1
0  570671  2440
1    7305  1492
precision = 0.38, recall = 0.17, F1 = 0.23, accuracy = 0.9833, saved cost per flight = $17.05<jupyter_text>Now let's apply this model with the new sets of features, starting with the urban data.<jupyter_code>alg = RandomForestClassifier(n_estimators=3)
#Features from original data
predictors = list(model_data.columns[2:81])

#Split the data into 10 subsets
kf = KFold(cv_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (cv_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = cv_data['Cancelled'].iloc[train]
    alg.fit(scaler.transform(X_train), y_train)
    X_test = (cv_data[predictors].iloc[test,:])
    y_test = cv_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(cv_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(cv_data['Cancelled'].values, predictions)
print(('precision = {0:.3f}, recall = {1:.3f}, F1 = {2:.3f}, ' 
       'accuracy = {3:.3f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(cv_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))<jupyter_output>Confusion matrix
        0     1
0  570721  2390
1    7284  1513
precision = 0.388, recall = 0.172, F1 = 0.238, accuracy = 0.983, saved cost per flight = $17.41<jupyter_text>And now the weather data<jupyter_code>alg = RandomForestClassifier(n_estimators=3)
#Features from original data
predictors = list(model_data.columns[2:94])

#Split the data into 10 subsets
kf = KFold(cv_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (cv_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = cv_data['Cancelled'].iloc[train]
    alg.fit(scaler.transform(X_train), y_train)
    X_test = (cv_data[predictors].iloc[test,:])
    y_test = cv_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(cv_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(cv_data['Cancelled'].values, predictions)
print(('precision = {0:.3f}, recall = {1:.3f}, F1 = {2:.3f}, ' 
       'accuracy = {3:.3f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(cv_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))<jupyter_output>Confusion matrix
        0     1
0  570478  2633
1    7212  1585
precision = 0.376, recall = 0.180, F1 = 0.244, accuracy = 0.983, saved cost per flight = $18.06<jupyter_text>And the aircraft data<jupyter_code>alg = RandomForestClassifier(n_estimators=3)
#Features from original data
predictors = list(model_data.columns[2:])

#Split the data into 10 subsets
kf = KFold(cv_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (cv_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = cv_data['Cancelled'].iloc[train]
    alg.fit(scaler.transform(X_train), y_train)
    X_test = (cv_data[predictors].iloc[test,:])
    y_test = cv_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(cv_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(cv_data['Cancelled'].values, predictions)
print(('precision = {0:.3f}, recall = {1:.3f}, F1 = {2:.3f}, ' 
       'accuracy = {3:.3f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(cv_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))
#Try these trees
tree_list = [1,3,5,10,30,50]

predictors = list(model_data.columns[2:])

conf_matrix = {}
coef = {}
precision = []; recall = []; F1 = []; accuracy = []; saved_cost = []
for trees in tree_list:
    alg = RandomForestClassifier(n_estimators=trees)

    #Split the data into 10 subsets
    kf = KFold(shuffled_data.shape[0], n_folds = 10, random_state = 1)
    predictions = []

    #Train and predict
    for train, test in kf:
        X_train = (shuffled_data[predictors].iloc[train,:])
        scaler = StandardScaler().fit(X_train)
        y_train = shuffled_data['Cancelled'].iloc[train]
        %time alg.fit(scaler.transform(X_train), y_train)
        X_test = (shuffled_data[predictors].iloc[test,:])
        y_test = shuffled_data['Cancelled'].iloc[test]
        test_predictions = alg.predict(scaler.transform(X_test))
        predictions.append(test_predictions)
        gc.collect()

    predictions = np.concatenate(predictions,axis = 0)
    conf_matrix[trees] = confusion_matrix(shuffled_data['Cancelled'].values, predictions)
    print('Num trees = {0}\nConfusion matrix'.format(trees))
    print(pd.DataFrame(conf_matrix[trees]))
    
    scores = precision_recall_fscore_support(shuffled_data['Cancelled'].values, predictions)
    precision.append(scores[0][1]); recall.append(scores[1][1]); F1.append(scores[2][1])
    accuracy.append(accuracy_score(shuffled_data['Cancelled'].values, predictions))
    saved_cost.append(cost_saved(conf_matrix[trees])/len(predictions))
    
    print(('precision = {0:.2f}, recall = {1:.2f}, F1 = {2:.2f}, ' 
       'accuracy = {3:.2f}, saved cost per flight = ${4:.2f}').format(precision[-1],recall[-1],F1[-1],
                                                                   accuracy[-1],saved_cost[-1]))
fig = plt.figure(figsize=(13,3))
ax1 = fig.add_subplot(131)
ax1.plot(tree_list,precision,'-or')
ax1.set_xlabel('num trees')
ax1.set_xscale('log')
ax1.set_ylabel('Precision', color='r')
ax1.set_title('Precision and recall for\n random forest classifier')
for tl in ax1.get_yticklabels():
    tl.set_color('r')

ax2 = ax1.twinx()
ax2.plot(tree_list, recall, '-ob')
ax2.set_ylabel('Recall', color='b')
for tl in ax2.get_yticklabels():
    tl.set_color('b')
    
ax3 = fig.add_subplot(132)
ax3.plot(tree_list,F1,'-or')
ax3.set_xlabel('num trees')
ax3.set_xscale('log')
ax3.set_ylabel('F1', color='r')
ax3.set_title('F1 and accuracy for\n random forest classifier')
for tl in ax3.get_yticklabels():
    tl.set_color('r')

ax4 = ax3.twinx()
ax4.plot(tree_list, accuracy, '-ob')
ax4.set_ylabel('Accuracy', color='b')
ax4.yaxis.set_major_formatter(y_formatter)
for tl in ax4.get_yticklabels():
    tl.set_color('b')
    
ax5 = fig.add_subplot(133)
ax5.plot(tree_list,saved_cost,'-or')
ax5.set_xlabel('num trees')
ax5.set_xscale('log')
ax5.set_ylabel('Cost ($)',color='r')
ax5.set_title('Cost saved per flight\n for random forest classifier')
for tl in ax5.get_yticklabels():
    tl.set_color('r')

num_pos = [(conf_matrix[trees][0][1]+conf_matrix[trees][1][1])*100/sum(sum(conf_matrix[trees]))
           for trees in tree_list]
ax6 = ax5.twinx()
ax6.plot(tree_list, num_pos, '-ob')
ax6.set_ylabel('% guessed positive', color='b')
ax6.yaxis.set_major_formatter(y_formatter)
for tl in ax6.get_yticklabels():
    tl.set_color('b')

fig.tight_layout()
fig.savefig('../reports/figures/classification/rf2_reg_trees', bbox_inches='tight')
#Try these leaves
leaf_list = [1,2,3,5,7,10]

predictors = list(model_data.columns[2:])

conf_matrix = {}
coef = {}
precision = []; recall = []; F1 = []; accuracy = []; saved_cost = []
for leaf in leaf_list:
    alg = RandomForestClassifier(min_samples_leaf=leaf, n_estimators=5)

    #Split the data into 10 subsets
    kf = KFold(shuffled_data.shape[0], n_folds = 10, random_state = 1)
    predictions = []

    #Train and predict
    for train, test in kf:
        X_train = (shuffled_data[predictors].iloc[train,:])
        scaler = StandardScaler().fit(X_train)
        y_train = shuffled_data['Cancelled'].iloc[train]
        alg.fit(scaler.transform(X_train), y_train)
        X_test = (shuffled_data[predictors].iloc[test,:])
        y_test = shuffled_data['Cancelled'].iloc[test]
        test_predictions = alg.predict(scaler.transform(X_test))
        predictions.append(test_predictions)
        gc.collect()

    predictions = np.concatenate(predictions,axis = 0)
    conf_matrix[leaf] = confusion_matrix(shuffled_data['Cancelled'].values, predictions)
    print('Min leaf size = {0}\nConfusion matrix'.format(leaf))
    print(pd.DataFrame(conf_matrix[leaf]))
    
    scores = precision_recall_fscore_support(shuffled_data['Cancelled'].values, predictions)
    precision.append(scores[0][1]); recall.append(scores[1][1]); F1.append(scores[2][1])
    accuracy.append(accuracy_score(shuffled_data['Cancelled'].values, predictions))
    saved_cost.append(cost_saved(conf_matrix[leaf])/len(predictions))
    
    print(('precision = {0:.2f}, recall = {1:.2f}, F1 = {2:.2f}, ' 
       'accuracy = {3:.2f}, saved cost per flight = ${4:.2f}').format(precision[-1],recall[-1],F1[-1],
                                                                   accuracy[-1],saved_cost[-1]))
fig = plt.figure(figsize=(13,3))
ax1 = fig.add_subplot(131)
ax1.plot(leaf_list,precision,'-or')
ax1.set_xlabel('min leaf size')
ax1.set_xscale('log')
ax1.set_ylabel('Precision', color='r')
ax1.set_title('Precision and recall for\n random forest classifier')
for tl in ax1.get_yticklabels():
    tl.set_color('r')

ax2 = ax1.twinx()
ax2.plot(leaf_list, recall, '-ob')
ax2.set_ylabel('Recall', color='b')
for tl in ax2.get_yticklabels():
    tl.set_color('b')
    
ax3 = fig.add_subplot(132)
ax3.plot(leaf_list,F1,'-or')
ax3.set_xlabel('min leaf size')
ax3.set_xscale('log')
ax3.set_ylabel('F1', color='r')
ax3.set_title('F1 and accuracy for\n random forest classifier')
for tl in ax3.get_yticklabels():
    tl.set_color('r')

ax4 = ax3.twinx()
ax4.plot(leaf_list, accuracy, '-ob')
ax4.set_ylabel('Accuracy', color='b')
ax4.yaxis.set_major_formatter(y_formatter)
for tl in ax4.get_yticklabels():
    tl.set_color('b')
    
ax5 = fig.add_subplot(133)
ax5.plot(leaf_list,saved_cost,'-or')
ax5.set_xlabel('min leaf size')
ax5.set_xscale('log')
ax5.set_ylabel('Cost ($)',color='r')
ax5.set_title('Cost saved per flight\n for random forest classifier')
for tl in ax5.get_yticklabels():
    tl.set_color('r')

num_pos = [(conf_matrix[leaf][0][1]+conf_matrix[leaf][1][1])*100/sum(sum(conf_matrix[leaf]))
           for leaf in leaf_list]
ax6 = ax5.twinx()
ax6.plot(leaf_list, num_pos, '-ob')
ax6.set_ylabel('% guessed positive', color='b')
ax6.yaxis.set_major_formatter(y_formatter)
for tl in ax6.get_yticklabels():
    tl.set_color('b')

fig.tight_layout()
fig.savefig('../reports/figures/classification/rf2_reg_leaves', bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>So the optimum number of trees with all features added is now 5, with no (i.e. 1) minimum leaf size still performing best.Let's see how this performs for all features:<jupyter_code>alg = RandomForestClassifier(n_estimators=5)
#Features from original data
predictors = list(model_data.columns[2:])

#Split the data into 10 subsets
kf = KFold(cv_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (cv_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = cv_data['Cancelled'].iloc[train]
    alg.fit(scaler.transform(X_train), y_train)
    X_test = (cv_data[predictors].iloc[test,:])
    y_test = cv_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(cv_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(cv_data['Cancelled'].values, predictions)
print(('precision = {0:.3f}, recall = {1:.3f}, F1 = {2:.3f}, ' 
       'accuracy = {3:.3f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(cv_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))
alg = RandomForestClassifier(n_estimators=5)
#Features from original data
predictors = list(model_data.columns[2:94])

#Split the data into 10 subsets
kf = KFold(cv_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (cv_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = cv_data['Cancelled'].iloc[train]
    alg.fit(scaler.transform(X_train), y_train)
    X_test = (cv_data[predictors].iloc[test,:])
    y_test = cv_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(cv_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(cv_data['Cancelled'].values, predictions)
print(('precision = {0:.3f}, recall = {1:.3f}, F1 = {2:.3f}, ' 
       'accuracy = {3:.3f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(cv_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))
alg = RandomForestClassifier(n_estimators=50,min_samples_leaf=3)
#Features from original data
predictors = list(model_data.columns[2:])

#Split the data into 10 subsets
kf = KFold(cv_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (cv_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = cv_data['Cancelled'].iloc[train]
    %time alg.fit(scaler.transform(X_train), y_train)
    X_test = (cv_data[predictors].iloc[test,:])
    y_test = cv_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(cv_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(cv_data['Cancelled'].values, predictions)
print(('precision = {0:.3f}, recall = {1:.3f}, F1 = {2:.3f}, ' 
       'accuracy = {3:.3f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(cv_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))<jupyter_output>Wall time: 2min 17s
Wall time: 2min 29s
Wall time: 2min 23s
Wall time: 2min 36s
Wall time: 2min 35s
Wall time: 2min 38s
Wall time: 2min 13s
Wall time: 2min 17s
Wall time: 2min 12s
Wall time: 2min 25s
Confusion matrix
        0    1
0  572874  237
1    7890  907
precision = 0.793, recall = 0.103, F1 = 0.182, accuracy = 0.986, saved cost per flight = $12.13<jupyter_text>Let's finally see how we perform without any 'future' knowledge.<jupyter_code>alg = RandomForestClassifier(n_estimators=5)
#Features from original data
predictors = list(model_data.columns[2:8])+list(model_data.columns[9:81])+list(model_data.columns[94:])

#Split the data into 10 subsets
kf = KFold(cv_data.shape[0], n_folds = 10, random_state = 1)
predictions = []

#Train and predict
for train, test in kf:
    X_train = (cv_data[predictors].iloc[train,:])
    scaler = StandardScaler().fit(X_train)
    y_train = cv_data['Cancelled'].iloc[train]
    alg.fit(scaler.transform(X_train), y_train)
    X_test = (cv_data[predictors].iloc[test,:])
    y_test = cv_data['Cancelled'].iloc[test]
    test_predictions = alg.predict(scaler.transform(X_test))
    predictions.append(test_predictions)
    gc.collect()

predictions = np.concatenate(predictions,axis = 0)
conf_matrix = confusion_matrix(cv_data['Cancelled'].values, predictions)
print('Confusion matrix')
print(pd.DataFrame(conf_matrix))
scores = precision_recall_fscore_support(cv_data['Cancelled'].values, predictions)
print(('precision = {0:.3f}, recall = {1:.3f}, F1 = {2:.3f}, ' 
       'accuracy = {3:.3f}, saved cost per flight = ${4:.2f}').format(scores[0][1],scores[1][1],scores[2][1],
                                                        accuracy_score(cv_data['Cancelled'].values, predictions),
                                                        cost_saved(conf_matrix)/len(predictions)))<jupyter_output>Confusion matrix
        0     1
0  571817  1294
1    7525  1272
precision = 0.496, recall = 0.145, F1 = 0.224, accuracy = 0.985, saved cost per flight = $15.65
{'fraction_non_alphanumeric': 0.14145535801029827, 'fraction_numerical': 0.034538035558146316, 'mean_word_length': 4.312645161290322, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 54, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '157713', 'license': 'MIT', 'n_tokens_mistral': 15057, 'n_tokens_neox': 14126, 'n_words': 2934, 'path': 'notebooks/03_03-10-16_training-model.ipynb', 'repo_name': 'sfblake/Flights_Project', 'size': '41172'}