<jupyter_start><jupyter_text>CS 109A/STAT 121A/AC 209A/CSCI E-109A: **Harvard University****Fall 2017****Instructors**: <NAME>, <NAME>, <NAME>, <NAME>  Flight DelaysThe U.S. Department of Transportation's (DOT) Bureau of Transportation Statistics tracks the on-time performance of domestic flights operated by large air carriers. Summary information on the number of on-time, delayed, canceled, and diverted flights are published in DOT's monthly Air Travel Consumer Report and in this dataset of 2015 flight delays and cancellations. DataEach entry of the flights.csv file corresponds to a flight.  More than 5,800,000 flights were recorded in 2015. These flights are described according to 31 variables. Further details of these variables can be found here, if you are interested (not needed to answer these questions).  |          Name          | Type | DESCRIPTION ||--------------------------------|---|----------------------------------------------------------------------|| DATE                     | object | The date in python datetime format || MONTH                     | int64 | The month of the year(1-12) || DAY                       | int64 | The day of the month || DAY_OF_WEEK               | int64 | The day of the week(1-7, MON-SUN) || AIRLINE                  | object | An identifier for the airline || FLIGHT_NUMBER             | int64 | The flight number || TAIL_NUMBER              | object | The tail number (aircraft) corresponding to this flight || ORIGIN_AIRPORT           | object | The code for origin airport || DESTINATION_AIRPORT      | object | The code for destination airport || SCHED_DEP                | object | The departure time in python datetime.time format || SCHED_ARR                | object | The arrival time in python datetime.time format || DEPARTURE_DELAY         |  float64| The delay incurred at the origin  (mins) || ARRIVAL_DELAY           | float64 | The delay when the flight reached the (mins) destination || DISTANCE                  | int64 | Distance in miles between origin and destination || SCHEDULED_TIME          | float64 | Scheduled time of flight (minutes) || ELAPSED_TIME            | float64 | Actual time of flight (minutes) || AIR_SYSTEM_DELAY        | float64 | What part of the delay was NASD?(mins) || SECURITY_DELAY          | float64 | What part of the delay was due to security problems? (mins) || AIRLINE_DELAY           | float64 | What part of the delay is due to the airline? (mins) || LATE_AIRCRAFT_DELAY     | float64 | What part of the delay is due to previous flight(s) being late(mins) || WEATHER_DELAY           | float64 | Delay due to extreme weather events(min) |You can read more about the various weather delays [here](https://www.rita.dot.gov/bts/help/aviation/html/understanding.html) if you are so inclined. Data/CaveatsThe data file, flights.csv, is found here (note, it is about 70MB). This data is already preprocessed, reduced, partially cleaned and therefore not identical to the original dataset.  Problem Description We will build two separate models: one model that classifies whether a flight will be delayed and a second model that predicts the length of delay given that a flight is truly delayed. Only consider models taught in class so far. ** Consider the following: **This is a large dataset; think of strategies on how to solve this problem. Create a manageable subsample of the data that you can use to train and test/validate, but eventually you should predict on all the data (excluding the training set). Questions1. (5pts) Create a new variable, `DELAY_OR_NOT`: a boolean/indicator variable which indicates any arrival delay under 15 mins as a 0, and any delay at or above 15 mins as a 1 (`ARRIVAL_DELAY >= 15`).2. (5pts) Make sure you understand the data variable descriptions before you start the analysis. Consider all the columns and determine and list which of these predictors should not be used. 3. (15pts) Perform EDA to gain intuition of the factors that affect delay and provide visuals: do delays vary across airlines, or time of departure, or airport (do, at the very least, Chicago (ORD), Boston (BOS), and your favorite another airport), or airport traffic?4. (20pts) Build a classification model that classifies delays according to `DELAY_OR_NOT`. This is an unbalanced dataset, thus consider the appropriate performance metric when reporting your results. 5. (5pts) Given your model, comment on the importance of factors as related to whether a flight is delayed.6. (5pts) Evaluate your model(s) on your test set, and finally provide a visual to show which airlines are predicted to have the most delays using all the data excluding the training and test set. 7. (15pts) Build a regression model that predicts the length of delay (on the log scale) given that a flight is truly delayed.8. (20pts) Write a report (in the last markdown cell in your notebook with your findings (without code)). Describe the main design decisions you have made with justifications. Clearly explain your methodology and results. This should not be more than 300 words. You may use up to 5 diagrams.<jupyter_code>import graphviz
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegressionCV
import sklearn.metrics as metrics
from sklearn.preprocessing import PolynomialFeatures
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
#import pydotplus
#import io
from sklearn.tree import export_graphviz
from IPython.display import Image
from IPython.display import display
%matplotlib inline
import seaborn as sns
import warnings
warnings.filterwarnings("ignore")
from sklearn import discriminant_analysis
from sklearn.tree import export_graphviz
from sklearn.model_selection import train_test_split

import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from sklearn.metrics import r2_score
import statsmodels.api as sm
from statsmodels.api import OLS
from sklearn import linear_model, datasets
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
from sklearn.linear_model import Lasso
from sklearn.linear_model import RidgeCV
from sklearn.linear_model import LassoCV
import seaborn.apionly as sns
from scipy import stats
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Question 1<jupyter_code>#Loading dataset
df = pd.read_csv('cs109a_midterm.csv')

# Creating a new variable according to ARRIVAL_DELAY
df["DELAY_OR_NOT"]=""
df["DELAY_OR_NOT"][df['ARRIVAL_DELAY']>=15]=1
df["DELAY_OR_NOT"][df['ARRIVAL_DELAY']<15]=0
print(sum(pd.isnull(df['AIR_SYSTEM_DELAY'])))
print(sum(pd.isnull(df['SECURITY_DELAY'])))
print(sum(pd.isnull(df['LATE_AIRCRAFT_DELAY'])))
print(sum(pd.isnull(df['WEATHER_DELAY'])))
print(sum(pd.isnull(df['AIRLINE_DELAY'])))
# Most of the data points in this predictors are NaN
# From Q2, dropping columns which are not suitable as predictor. Explanation in Q2 below
df=df.drop(['DATE', 'ARRIVAL_DELAY', 'SECURITY_DELAY', 'AIRLINE_DELAY','LATE_AIRCRAFT_DELAY','WEATHER_DELAY', 'AIR_SYSTEM_DELAY', 'DAY','TAIL_NUMBER', 'FLIGHT_NUMBER'], axis=1)
df.shape
df.head()
# CONVERTING CLOCK TIME TO SECONDS INTEGER (TOOK THREE HOURS) for x_train_dummy
z = df['SCHED_DEP'].str.split(pat = ':', expand=True)
x = z.values
x = z.astype(float)
x = pd.DataFrame(x)
x.iloc[:,2:3]=x.iloc[:,2:3]*(1/3600)
x.iloc[:,1:2]=x.iloc[:,1:2]*(1/60)
x['sum']=x.sum(axis=1)
df['SCHED_DEP']=x['sum']

z = df['SCHED_ARR'].str.split(pat = ':', expand=True)
x = z.values
x = z.astype(float)
x = pd.DataFrame(x)
x.iloc[:,2:3]=x.iloc[:,2:3]*(1/3600)
x.iloc[:,1:2]=x.iloc[:,1:2]*(1/60)
x['sum']=x.sum(axis=1)
df['SCHED_ARR']=x['sum']

df.head()
#splitting main dataset into a workable smaller data test and rest of the datatest which will be used for testing
msk = np.random.rand(len(df)) < 0.15
df_small = df[msk]
df_rest = df[~msk]
df_small.shape, df_rest.shape
df_x = df_small.iloc[:, 0:11]
df_y = df_small.iloc[:, 11:12]
df_x_rest = df_rest.iloc[:, 0:11]
df_y_rest = df_rest.iloc[:, 11:12]
df_x.shape, df_y.shape, df_x_rest.shape, df_y_rest.shape
# creating the smaller train/test set on which we will do the bulk of the modelling work
x_train_small, x_test_small, y_train_small, y_test_small= train_test_split(df_x, df_y, test_size = 0.33)
print("shapes:", x_train_small.shape, y_train_small.shape, x_test_small.shape, y_test_small.shape)<jupyter_output>shapes: (80980, 11) (80980, 1) (39887, 11) (39887, 1)<jupyter_text>Question 2 Which variables should not be used?- Date: this is unique number which will be never be repeated in the future (e.g. there will never be another 2015-06-25 again) thus is not practical. Drop it.- Arrival_delay: Since this has been used to create the Delay_or_not dependable variable this cannot be a predictor. Drop it.- Air system delay, Security delay, Late aircraft delay, Weather Delay: All of these variables have about 90% NaN which is too high to impute any missing values. Also these are explicit delays, very likely to cause actual delays in the arrival time thus common sense dictates that incurring any of these delay will most likely result in a delay of arrival. What could have been interesting is to parse how much a delay can be expected if any of these delays take place - however, the proportions of NaNs is too high for imputation. Drop them.- Day: 30 or 31 days in a month, when made into dummies will become 62 dummies. Besides the specific date of the month is less relevant compared to the day of the week. Most of the variance that will be in the day of the month will be captured by the day of the week predictor.- Tail number: I am assuming that with the tail number it will be possible to identify the make of plane hence some metrics about its performance, there are far too many unique tail numbers to be a useful predictor. Hence dropped it.Because I have complicated series of test-train splits I am going back up to quesiton 1 and dropping these from the main df, before I make teh train-test split in order to minimize teh drops.<jupyter_code>x_train_small.head()<jupyter_output><empty_output><jupyter_text>Question 3<jupyter_code>n = df_small['DELAY_OR_NOT'].value_counts()
n = pd.DataFrame(n).values
delay = (n[1:2]/(n[0:1]+n[1:2]))
print(delay)
print("Only 10.1% of the observations is delay = 1 data thus by classifying everything as delayed=0, we would be 90% accurate. Imbalanced data set")
a = df_small.groupby(by=['ORIGIN_AIRPORT'])['DELAY_OR_NOT'].sum()
a = pd.DataFrame(a)
b=df_small["ORIGIN_AIRPORT"].value_counts()
b = pd.DataFrame(b)
c = pd.concat ([a, b], axis=1).reset_index()
c['Average_delays']=c['DELAY_OR_NOT']/c['ORIGIN_AIRPORT']
c.sort_values(['Average_delays'], ascending=False, inplace=True)
c.shape
# BAR CHART FOR TOP TEN AIRPORTS 
plt.figure(figsize=(20, 20))
#f, (ax1, ax2) =plt.subplots(1,2, sharey=True)
airport = list(range(1,21))
labels=c.iloc[:,0:1].values
d=c.iloc[0:20, 3:4].values
plt.xticks(airport, labels, rotation=45)
plt.bar(airport,d )
plt.title ("Top 20 ORIGIN AIRPORTS with the highest % of delays")
a = df_small.groupby(by=['DESTINATION_AIRPORT'])['DELAY_OR_NOT'].sum()
a = pd.DataFrame(a)
b=df_small["DESTINATION_AIRPORT"].value_counts()
b = pd.DataFrame(b)
c = pd.concat ([a, b], axis=1).reset_index()
c['Average_delays']=c['DELAY_OR_NOT']/c['DESTINATION_AIRPORT']
c.sort_values(['Average_delays'], ascending=False, inplace=True)
c.shape
c.head()
# BAR CHART FOR TOP20 DESTINATION AIRPORTS WITH DELAYS
plt.figure(figsize=(20, 20))
airport = list(range(1,21))
labels=c.iloc[:,0:1].values
d=c.iloc[0:20, 3:4].values
plt.xticks(airport, labels, rotation=45)
plt.bar(airport,d )
plt.title ("Top 20 DESTINATION AIRPORTS with the higher % of delays")
# WHICH DAY OF THE WEEK HAS THE MOST DELAYS
a = df_small.groupby(by=['DAY_OF_WEEK'])['DELAY_OR_NOT'].sum()
a= pd.DataFrame(a)
a.sort_values(['DELAY_OR_NOT'], ascending=False, inplace=True)
print("Thursday has the most number of delayed flights")
a
# WHICH MONTH HAS THE MOST DELAYS
a = df_small.groupby(by=['MONTH'])['DELAY_OR_NOT'].sum()
a= pd.DataFrame(a)
a.sort_values(['DELAY_OR_NOT'], ascending=False, inplace=True)
print("June has the most number of delayed flights")
a
# WHICH AIRLINE HAS THE MOST DELAYS
a = df_small.groupby(by=['AIRLINE'])['DELAY_OR_NOT'].sum()
a = pd.DataFrame(a)
b=df_small["AIRLINE"].value_counts()
b = pd.DataFrame(b)
c = pd.concat ([a, b], axis=1).reset_index()
c['Average_delays']=c['DELAY_OR_NOT']/c['AIRLINE']
c.sort_values(['Average_delays'], ascending=False, inplace=True)
#print("WN has the most number of delayed flights")
a.sort_values(['DELAY_OR_NOT'], ascending=False, inplace=True)
print("WN has the largest number of delays, but NK has the largest proportion of delayed flight at 38%")
print(a), print (c)
# BAR CHART AIRLINES WITH MOST DELAYS
plt.figure(figsize=(10, 10))
airport = (range(len(c)))
labels=c.iloc[:,0:1].values
d=c.iloc[:, 3:4].values
plt.xticks(airport, labels, rotation=45)
plt.bar(airport,d )
plt.title ("Airlines and % of delays")<jupyter_output><empty_output><jupyter_text>Summary of EDA- NK is the airline with the highest proportion of delayed flight- WN is the airline with the highest number of delayed flights.- Thursday is the day with the highest number of delayed flights.- June has the highest number of delayed flights.- Neither ORL or BOS are in the top 20 percetnage of flights delayed either as origin or destination airports.  QUESTION 4<jupyter_code>x_train_small.head()
# CONVERTING TO DUMMIES
x_train_dummy = pd.get_dummies(x_train_small, columns=["MONTH", "DAY_OF_WEEK", "AIRLINE","ORIGIN_AIRPORT","DESTINATION_AIRPORT"], drop_first=True)
x_test_dummy = pd.get_dummies(x_test_small, columns=["MONTH", "DAY_OF_WEEK", "AIRLINE","ORIGIN_AIRPORT","DESTINATION_AIRPORT"], drop_first=True)
x_test_rest_dummy = pd.get_dummies(df_x_rest, columns=["MONTH", "DAY_OF_WEEK", "AIRLINE","ORIGIN_AIRPORT","DESTINATION_AIRPORT"], drop_first=True)
x_train_dummy.shape, x_test_dummy.shape, x_test_rest_dummy.shape<jupyter_output><empty_output><jupyter_text>This is a very itneresting problem that x_train_dummy and x_test_dummy dont have the same number of predictors, which means it will not be possible to predict on x_test_dummy with a model trained on x_train_dummy since the number of coefficients will not be the same. I suspect it is the Origin and Destination airports that are not exactly the same in the two data sets. so lets try creating dummies without those.<jupyter_code>x_train_dummy = pd.get_dummies(x_train_small, columns=["MONTH", "DAY_OF_WEEK", "AIRLINE"], drop_first=True)
x_test_dummy = pd.get_dummies(x_test_small, columns=["MONTH", "DAY_OF_WEEK", "AIRLINE"], drop_first=True)
x_test_rest_dummy = pd.get_dummies(df_x_rest, columns=["MONTH", "DAY_OF_WEEK", "AIRLINE"], drop_first=True)
x_train_dummy.shape, x_test_dummy.shape, x_test_rest_dummy.shape<jupyter_output><empty_output><jupyter_text>Now the number of predictors are equal in x_test_dummy and x_train_dummy thus can train the model on the test set. It is unfortunate that we had to drop the airports as predictors because there were some airports where more than 35% are delayed, but because of the data-set we are having to drop the predictors. If I had more skill I would choose the observations where both the origin and destination airports are present.<jupyter_code>x_train_dummy =x_train_dummy.drop(['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT'], axis=1)
x_test_dummy =x_test_dummy.drop(['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT'], axis=1)
x_test_rest_dummy =x_test_rest_dummy.drop(['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT'], axis=1)

x_train_dummy.shape, x_test_dummy.shape, x_test_rest_dummy.shape
x_train_dummy.head()
y_train_small=y_train_small.astype('int')
y_test_small=y_test_small.astype('int')
df_y_rest=df_y_rest.astype('int')
x_test_rest_dummy.dtypes
# LOGISTIC REGRESSION WITH REGULARIZATION AND CROSS VALIDATION
searchCV=LogisticRegressionCV(cv=5, penalty='l2')
searchCV.fit(x_train_dummy, y_train_small)
print("Accuracy of logistic model on small test set: %0.6f" % searchCV.score(x_test_dummy, y_test_small))
print("Accuracy of logistic model on large test sest: %0.6f" % searchCV.score(x_test_rest_dummy, df_y_rest))
from sklearn.metrics import confusion_matrix
y_pred=searchCV.predict(x_test_dummy)
cm = confusion_matrix(y_test_small, y_pred)
cm
#TRUE POSITIVE RATE
cm_tp=cm[1:2, 1:2]/np.sum(cm[1:2,:])

#TRUE NEGATIVE RATE
cm_tn=cm[0:1, 0:1]/np.sum(cm[0:1,:])

print("The True positive rate is %0.4f"% cm_tp)
print("The True negative rate is %0.4f"% cm_tn)<jupyter_output>The True positive rate is 0.9998
The True negative rate is 1.0000<jupyter_text>DisucssionThs is not normal because such a high accuracy rate of 100% means that I have some predictor in my predictors set which is acting as a proxy for the dependent variable. I suspect it is the variable 'Departure Delay'. So I am going to repeat by dropping that predictor to see if anything changes.<jupyter_code>#DROPPING VARIABLE 'DEPARTURE DELAY
x_train_dummy =x_train_dummy.drop(['DEPARTURE_DELAY'], axis=1)
x_test_dummy =x_test_dummy.drop(['DEPARTURE_DELAY'], axis=1)
x_test_rest_dummy =x_test_rest_dummy.drop(['DEPARTURE_DELAY'], axis=1)

x_train_dummy.shape, x_test_dummy.shape, x_test_rest_dummy.shape
# REPEATING LOGISTIC REGRESSION WITH REGULARIZATION AND CROSS VALIDATION
searchCV=LogisticRegressionCV(cv=5, penalty='l2')
searchCV.fit(x_train_dummy, y_train_small)
y_pred_log= searchCV.predict(x_test_dummy)
print("Accuracy of logistic model on small test set: %0.6f" % searchCV.score(x_test_dummy, y_test_small))
print("Accuracy of logistic model on large test sest: %0.6f" % searchCV.score(x_test_rest_dummy, df_y_rest))<jupyter_output>Accuracy of logistic model on small test set: 0.918996
Accuracy of logistic model on large test sest: 0.918902<jupyter_text>AHA!The 'Departure Delay' was hiding inside my model as a proxy for the dependent variable. The results now make more sense and it's also good to see that the results on teh smaller and larger testing sets are close enough showing there is no overfitting.<jupyter_code>y_pred=searchCV.predict(x_test_dummy)
cm = confusion_matrix(y_test_small, y_pred)
cm
#TRUE POSITIVE RATE
cm_tp=cm[1:2, 1:2]/np.sum(cm[1:2,:])

#TRUE NEGATIVE RATE
cm_tn=cm[0:1, 0:1]/np.sum(cm[0:1,:])

print("The True positive rate is %0.4f"% cm_tp)
print("The True negative rate is %0.4f"% cm_tn)
# LDA MODEL
lda= discriminant_analysis.LinearDiscriminantAnalysis()
lda_fit=lda.fit(x_train_dummy, y_train_small)
print("Accuracy with LDA on small test: %0.4f" % lda_fit.score(x_test_dummy, y_test_small))
print("Accuracy with LDA on large test: %0.4f" % lda_fit.score(x_test_rest_dummy, df_y_rest))

#QDA MODEL
qda = discriminant_analysis.QuadraticDiscriminantAnalysis()
qda_fit=qda.fit(x_train_dummy, y_train_small)
print("Accuracy with QDA with small test: %0.4f" % qda_fit.score(x_test_dummy, y_test_small))
print("Accuracy with QDA with large test: %0.4f" % qda_fit.score(x_test_rest_dummy, df_y_rest))

# CONFUSION MATRIX FOR LDA
y_pred=lda.predict(x_test_dummy)
cm = confusion_matrix(y_test_small, y_pred)

#TRUE POSITIVE RATE
cm_tp=cm[1:2, 1:2]/np.sum(cm[1:2,:])

#TRUE NEGATIVE RATE
cm_tn=cm[0:1, 0:1]/np.sum(cm[0:1,:])

print("The True positive rate for lda is %0.2f"% cm_tp)
print("The True negative rate for lda is %0.2f"% cm_tn)
# CONFUSION MATRIX FOR QDA
y_pred=qda.predict(x_test_dummy)
cm = confusion_matrix(y_test_small, y_pred)

#TRUE POSITIVE RATE
cm_tp=cm[1:2, 1:2]/np.sum(cm[1:2,:])

#TRUE NEGATIVE RATE
cm_tn=cm[0:1, 0:1]/np.sum(cm[0:1,:])

print("The True positive rate is %0.2f"% cm_tp)
print("The True negative rate is %0.2f"% cm_tn)
#3.2 FINDING THE OPTIMAL TREE DEPTH
from sklearn.model_selection import GridSearchCV
est = DecisionTreeClassifier()
parameters = {"max_depth": range(1,30)}
gs = GridSearchCV(est, param_grid=parameters, cv=5)
gs.fit(x_train_dummy.values, y_train_small.DELAY_OR_NOT.values)
gs.best_params_, gs.best_score_
# CONFUSION MATRIX FOR QDA
y_pred=gs.predict(x_test_dummy)
cm = confusion_matrix(y_test_small, y_pred)

#TRUE POSITIVE RATE
cm_tp=cm[1:2, 1:2]/np.sum(cm[1:2,:])

#TRUE NEGATIVE RATE
cm_tn=cm[0:1, 0:1]/np.sum(cm[0:1,:])

print("The True positive rate is %0.2f"% cm_tp)
print("The True negative rate is %0.2f"% cm_tn)
#2.1 ROC CURVE FOR LOGISTIC MODEL AND ALL ZEROS CLASSIFIER

from sklearn.metrics import roc_curve, auc

def make_roc(name, clf, ytest, xtest, ax=None, labe=5, proba=True, skip=0):
    initial=False
    if not ax:
        ax=plt.gca()
        initial=True
    if proba:#for stuff like logistic regression
        fpr, tpr, thresholds=roc_curve(ytest, clf.predict_proba(xtest)[:,1])
    else:#for stuff like SVM
        fpr, tpr, thresholds=roc_curve(ytest, clf.decision_function(xtest))
    roc_auc = auc(fpr, tpr)
    if skip:
        l=fpr.shape[0]
        ax.plot(fpr[0:l:skip], tpr[0:l:skip], '.-', alpha=0.3, label='ROC curve for %s (area = %0.3f)' % (name, roc_auc))
    else:
        ax.plot(fpr, tpr, '.-', alpha=0.3, label='ROC curve for %s (area = %0.3f)' % (name, roc_auc))
    label_kwargs = {}
    label_kwargs['bbox'] = dict(
        boxstyle='round,pad=0.3', alpha=0.2,
    )
    if labe!=None:
        for k in range(0, fpr.shape[0],labe):
            #from https://gist.github.com/podshumok/c1d1c9394335d86255b8
            threshold = str(np.round(thresholds[k], 5))
            ax.annotate(threshold, (fpr[k], tpr[k]), **label_kwargs)
    if initial:
        ax.plot([0, 1], [0, 1], 'k--')
        ax.set_xlim([0.0, 1.0])
        ax.set_ylim([0.0, 1.05])
        ax.set_xlabel('False Positive Rate')
        ax.set_ylabel('True Positive Rate')
        ax.set_title('ROC')
    ax.legend(loc="lower right")
    return ax
sns.set_context("poster")
ax1=make_roc("logistic",searchCV, y_test_small, x_test_dummy, labe=950, skip=100)
ax1.plot([0, 1], [0, 1], ls="--", c="r")
sns.set_context("poster")
ax1=make_roc("lda",lda, y_test_small, x_test_dummy, labe=1000, skip=10)
ax1.plot([0, 1], [0, 1], ls="--", c="r")
sns.set_context("poster")
ax1=make_roc("qda",qda, y_test_small, x_test_dummy, labe=1000, skip=10)
ax1.plot([0, 1], [0, 1], ls="--", c="r")
sns.set_context("poster")
ax1=make_roc("Decision Tree",gs, y_test_small, x_test_dummy, labe=1000, skip=10)
ax1.plot([0, 1], [0, 1], ls="--", c="r")<jupyter_output><empty_output><jupyter_text>Discussion 4The base accuracy, given the imbalance: 90% of the data is delay = 0. Thus compared to an all-zero's response, any model should do 90% or better. I whittled down the predictors to 11 and at each stage I have discussed why I have chosen to drop them. I tried four models and test them on both the small test set and then the large test set. In all cases the numbers were very close showing that the training sample was large enough and there is no overfitting.|               | Accuracy | AUC   | TPR  | TNR   | Comments                                               ||---------------|----------|-------|------|-------|--------------------------------------------------------|| Base          | 0.90     | 0.6   |      |       |                                                        || Logistic      | 0.918    | 0.851 | 0.28 | 0.998 | Similar to LDA but lower TPR                           || LDA           | 0.917    | 0.844 | 0.31 | 0.99  | As good as Logistic, similar accuracy and better TPR   || QDA           | 0.85     | 0.796 | 0.54 | 0.89  | Accuracy Performance lower than base case of 90%       || Decision tree | 0.903    | 0.788 | 0.08 | 0.99  | Max depth is 7, accuracy performance same as base case |  Question 5I will be uisng the logistic model as my best model (to be honest it's performance is same as LDA but it is easier to implement than a LDA in practical terms). SKLearn does not display logit regression information as nicely as statsmodel. It can output the predictor values but there is no way to trace what the names of those predictors are and hence make practical conclusions. For this reason I replicated the logistic with stats model with dispalys results in a nicer way<jupyter_code># LOGISTIC SKLEARN COEFFICIENTS
searchCV.coef_
# exponent of the log odd to get the direct correlation with the possibility of a delay
np.exp(searchCV.coef_)
x_test_dummy.shape, y_test_small.shape, x_train_dummy.shape
# LOGISTIC IN STATS MODEL
import statsmodels.api as sm
from sklearn.metrics import r2_score
log = sm.Logit(y_train_small, x_train_dummy)
log.fit().summary()<jupyter_output>Optimization terminated successfully.
         Current function value: 0.256206
         Iterations 8<jupyter_text>Discussion 5It can be seen that certain months (esp towards the end of the year perhaps due to snow), day_of_week, certain airline are impactful on the delay of flights while scheduled arrival and departure delay, distance, scheduled and elapsed time are less impactful. Thus to predict if a flight will be late one should look at month, day_of_week and airline.  Question 6 I already evaluated my model on the small and large test in part 4 after each model. Can scroll back up and see. The accuraries are below. They were very close to teh numbers I got when testing with teh small test set showing no overfitting:- Logistic: 0.918- LDA: 0.917- QDA: 0.86- Decision Tree: 0.903  Question 7<jupyter_code>#Loading dataset
df = pd.read_csv('cs109a_midterm.csv')

# Creating a new variable according to ARRIVAL_DELAY
df["DELAY_OR_NOT"]=""
df["DELAY_OR_NOT"][df['ARRIVAL_DELAY']>=15]=1
df["DELAY_OR_NOT"][df['ARRIVAL_DELAY']<15]=0
# CREATING A SUBSAMPLE WITH DATA WHERE DELAY=1
df_delay=df.loc[df['DELAY_OR_NOT']==1]
df_delay=df_delay.drop(['DATE', 'DEPARTURE_DELAY', 'ORIGIN_AIRPORT', 'DESTINATION_AIRPORT','SECURITY_DELAY', 'AIRLINE_DELAY','LATE_AIRCRAFT_DELAY','WEATHER_DELAY', 'AIR_SYSTEM_DELAY', 'DAY','TAIL_NUMBER', 'FLIGHT_NUMBER', 'DELAY_OR_NOT'], axis=1)
df_delay.shape
# CONVERTING CLOCK TIME TO SECONDS INTEGER (TOOK THREE HOURS) for x_train_dummy
z = df_delay['SCHED_DEP'].str.split(pat = ':', expand=True)
x = z.values
x = z.astype(float)
x = pd.DataFrame(x)
x.iloc[:,2:3]=x.iloc[:,2:3]*(1/3600)
x.iloc[:,1:2]=x.iloc[:,1:2]*(1/60)
x['sum']=x.sum(axis=1)
df_delay['SCHED_DEP']=x['sum']

z = df_delay['SCHED_ARR'].str.split(pat = ':', expand=True)
x = z.values
x = z.astype(float)
x = pd.DataFrame(x)
x.iloc[:,2:3]=x.iloc[:,2:3]*(1/3600)
x.iloc[:,1:2]=x.iloc[:,1:2]*(1/60)
x['sum']=x.sum(axis=1)
df_delay['SCHED_ARR']=x['sum']

df_delay.head()
y_df=df_delay.iloc[:,5:6]
df_delay = df_delay.drop(['ARRIVAL_DELAY'], axis=1)
df_delay.shape, y_df.shape
y_df.head()
#log (ln) tranformation of y
y_df= np.log(y_df.ARRIVAL_DELAY)
y_df=pd.DataFrame(y_df)
y_df.head()
# creating the smaller train/test set on which we will do the bulk of the modelling work
x_train_small, x_test_small, y_train_small, y_test_small= train_test_split(df_delay, y_df, test_size = 0.5)
print("shapes:", x_train_small.shape, y_train_small.shape, x_test_small.shape, y_test_small.shape)
x_train_dummy = pd.get_dummies(x_train_small, columns=["MONTH", "DAY_OF_WEEK", "AIRLINE"], drop_first=True)
x_test_dummy = pd.get_dummies(x_test_small, columns=["MONTH", "DAY_OF_WEEK", "AIRLINE"], drop_first=True)
x_train_dummy.shape, x_test_dummy.shape
x_train_dummy.head()
#SIMPLE LINEAR REGRESSION
lin = LinearRegression().fit(x_train_dummy, y_train_small)
y_pred = lin.predict(x_test_dummy)
print("R2 value simple linear regression no regularization:", r2_score(y_test_small,y_pred))
lin.coef_
# TRYING A RIDGE
from sklearn.linear_model import RidgeCV
ridgeCV = RidgeCV(alphas=(0.000001, 0.0001, 0.001, 0.01, 0.1, 1.0, 10, 100), cv=5)
ridgeCV.fit(x_train_dummy, y_train_small)
print("R2 value with RIDGE:", ridgeCV.score(x_test_dummy, y_test_small))

# TRYING A LASSO
from sklearn.linear_model import LassoCV
lassoCV = LassoCV(alphas=(0.0001, 0.001, 0.01, 0.1, 1.0, 10), cv=5)
lassoCV.fit(x_train_dummy, y_train_small)
print("R2 value with LASSO:", lassoCV.score(x_test_dummy, y_test_small))<jupyter_output>R2 value with RIDGE: 0.0380078186563
R2 value with LASSO: 0.0379587208425<jupyter_text>Question 8<jupyter_code># Reprinting the exponent of the log odds from the logistic model
np.exp(searchCV.coef_)
# COEFFICIENTS OF THE LINEAR REGRESSION MODEL
lin.coef_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11568915983462967, 'fraction_numerical': 0.026662462336206293, 'mean_word_length': 4.9390345401581355, 'pattern_counts': {'":': 1, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '370511', 'license': 'MIT', 'n_tokens_mistral': 10363, 'n_tokens_neox': 9724, 'n_words': 3050, 'path': 'Predicting_Airline_Delays_CS109A_Fall2017.ipynb', 'repo_name': 'shammiquddus/Airline-Delays', 'size': '28568'}