<jupyter_start><jupyter_text>Exploratory Data AnalysisUse this notebook to get familiar with the datasets we have. There is 10 questions we need to answer during the EDA.We shouldn't limit our EDA to these 10 questions. Let's be creative :).<jupyter_code>import pandas as pd
import numpy as np
from matplotlib import pyplot
import seaborn as sns
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
pd.set_option("display.max_columns", None)
# import csv file with the proper dtypes
dtype_dict = {
 'branded_code_share': 'string',
 'mkt_carrier': 'string',
 'mkt_carrier_fl_num': 'string',
 'op_unique_carrier': 'string',
 'tail_num': 'string',
 'op_carrier_fl_num': 'string',
 'origin_airport_id': 'string',
 'origin': 'string',
 'origin_city_name': 'string',
 'dest_airport_id': 'string',
 'dest': 'string',
 'dest_city_name': 'string',
 'crs_dep_time': 'int64',
 'dep_time': 'int64',
 'dep_delay': 'int64',
 'taxi_out': 'int64',
 'wheels_off': 'int64',
 'wheels_on': 'int64',
 'taxi_in': 'int64',
 'crs_arr_time': 'int64',
 'arr_time': 'int64',
 'arr_delay': 'int64',
 'crs_elapsed_time': 'int64',
 'actual_elapsed_time': 'int64',
 'air_time': 'int64',
 'distance': 'int64'}
df = pd.read_csv("../LHL_Midterm_Project_Predicting_Flight_Delays/data/ransmpl_clean.csv", parse_dates=[0], dtype=dtype_dict)
df.head()
# removing arr_delay outliers (more than 3 std deviations away from mean)
outliers = df[df["arr_delay"].gt(round(df["arr_delay"].mean() + df["arr_delay"].std()*3))]
df2 = df[df["arr_delay"].le(round(df["arr_delay"].mean() + df["arr_delay"].std()*3))]
outliers.shape[0]

outliers.shape[0] / df.shape[0] # percentage of outliers in dataset
df2.shape[0]

df2.shape[0] / df.shape[0]  # percentage of datapoints within 3 std deviations<jupyter_output><empty_output><jupyter_text>df - data  df2 - data with arr_delay outliers removed  **Task 1**: Test the hypothesis that the delay is from Normal distribution. and that **mean** of the delay is 0. Be careful about the outliers.<jupyter_code># stats
df["arr_delay"].mean()
df["arr_delay"].median()
df["arr_delay"].std()
df["arr_delay"].min()
df["arr_delay"].max()
# no outlier df stats
df2["arr_delay"].mean()
df2["arr_delay"].median()
df2["arr_delay"].std()
df2["arr_delay"].min()
df2["arr_delay"].max()
sns.displot(df["arr_delay"], bins=300)
pyplot.xlim(-85, 85)

pyplot.show()
sns.histplot(df["arr_delay"])
pyplot.xlim(-85,85)<jupyter_output><empty_output><jupyter_text>**Analysis**Looks like the distribution is normal but is slightly skewed.  Mean is probably around -10 minutes when ignoring outliers.  **Task 2**: Is average/median monthly delay different during the year? If yes, which are months with the biggest delays and what could be the reason?<jupyter_code>months = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
data1 = df[["fl_date", "arr_delay"]].groupby(df["fl_date"].dt.month).median()
data1.index = months
data2 = df2[["fl_date", "arr_delay"]].groupby(df["fl_date"].dt.month).median()
data2.index = months
data3 = df[["fl_date", "arr_delay"]].groupby(df["fl_date"].dt.month).mean()
data3.index = months
data4 = df2[["fl_date", "arr_delay"]].groupby(df["fl_date"].dt.month).mean()
data4.index = months
data1
figure = pyplot.figure(figsize=(15,9))

pyplot.plot(data1.index, data1["arr_delay"], label="Median - All Flights")
pyplot.plot(data2.index, data2["arr_delay"], label="Median - No Outlier Flights")
pyplot.plot(data3.index, data3["arr_delay"], label="Mean - All Flights")
pyplot.plot(data4.index, data4["arr_delay"], label="Mean - No Outlier Flights")

pyplot.ylabel("Arrival Delay in minutes")
pyplot.yticks(range(-10, 12, 2))
pyplot.grid()
pyplot.legend()
pyplot.title("Average Flight Delay by Month")

pyplot.show();<jupyter_output><empty_output><jupyter_text>**Analysis**June, July, August, and Feb have the longest delays  Sep has the least delaysSummer month delays are probably due to higher traffic (lots of people taking summer vacations)  The spike in delay length during february confuses me  **Task 3**: Does the weather affect the delay? Use the API to pull the weather information for flights. There is no need to get weather for ALL flights. We can choose the right representative sample. Let's focus on four weather types:- sunny- cloudy- rainy- snow.Test the hypothesis that these 4 delays are from the same distribution. If they are not, which ones are significantly different? I don't know how to go about this...  **Task 4**: How taxi times changing during the day? Does higher traffic lead to bigger taxi times?<jupyter_code>df2.head()
df2["dep_time"].head(15)
(df2["dep_time"] / 100).head(15)
taxi_times = df2[["dep_time", "taxi_out"]].copy()
taxi_times["n_flights"] = 1  # will use this column to count the number of flights departing at given time of day
taxi_times["dep_time"] = taxi_times["dep_time"].round(-1)
# change times like '560' to '600' etc.
taxi_times.loc[taxi_times["dep_time"].astype(str).str.endswith("60"), "dep_time"] = taxi_times.loc[taxi_times["dep_time"].astype(str).str.endswith("60"), "dep_time"].values + 40 

# converting military time to 'continuous time' (e.g. 830 = 8.5)
times = (taxi_times["dep_time"] / 100).values.astype(str)
minutes = np.array([round(int(time.split(".")[1])/6, 2) for time in list(times)])
hour = (taxi_times["dep_time"] // 100).values
hours = hour + minutes
# adding the continuous time column to to data frame
taxi_times["hours"] = hours
taxi_times
data1 = taxi_times.groupby("dep_time").agg({"taxi_out": "mean", "n_flights": "count", "hours": "mean"}).round(2)
data1.reset_index(inplace=True)
data1.loc[:50]
taxi_times = df2[["wheels_on", "taxi_in"]].copy()
taxi_times["n_flights"] = 1
taxi_times["wheels_on"] = taxi_times["wheels_on"].round(-1)
taxi_times.loc[taxi_times["wheels_on"].astype(str).str.endswith("60"), "wheels_on"] = taxi_times.loc[taxi_times["wheels_on"].astype(str).str.endswith("60"), "wheels_on"].values + 40 
data2 = taxi_times.groupby("wheels_on").agg({"taxi_in": "mean", "n_flights": "count"}).round(1)
data2.reset_index(inplace=True)
data2["hours"] = data1["hours"]
data2
fig4 = pyplot.figure(figsize=(15,9))

pyplot.plot(data1["hours"], data1["taxi_out"], label="taxi out time")
pyplot.plot(data2["hours"], data2["taxi_in"], label="taxi in time")

pyplot.title("Mean taxi time by time of day")
pyplot.xlabel("time of day")
pyplot.xticks(range(0,24,1))
pyplot.ylabel("taxi duration in minutes")

pyplot.grid()
pyplot.legend()

pyplot.show();
fig4_2 = pyplot.figure(figsize=(15,9))

pyplot.plot(data1["hours"], data1["n_flights"], label="departing flights")
pyplot.plot(data2["hours"], data2["n_flights"], label="arriving landing")

pyplot.title("number of flights by time of day")
pyplot.xlabel("time of day")
pyplot.xticks(range(0,24,1))
pyplot.ylabel("number of flights")

pyplot.grid()
pyplot.legend()

pyplot.show();<jupyter_output><empty_output><jupyter_text>**Analysis**  Taxi out times are always about 10 min longer than taxi in times  **Task 5**: What is the average percentage of delays that is already created before departure? (aka are arrival delays caused by departure delays?) Are airlines able to lower the delay during the flights?<jupyter_code>delays = df2[["dep_delay", "arr_delay"]].copy()
delays["delay_diff"] = delays["arr_delay"] - delays["dep_delay"]

delays
# late arrival & late departure / late arrival
n_delayed_both = (delays["arr_delay"].gt(15) & delays["dep_delay"].gt(15)).sum()
n_delayed_arrival = delays["arr_delay"].gt(15).sum()
n_delayed_both
n_delayed_arrival

n_delayed_both / n_delayed_arrival<jupyter_output><empty_output><jupyter_text>**analysis**  76% of arrival delays are caused by departure delays (delay defined as greater than 15 min late)<jupyter_code>for i in range(0, delays["dep_delay"].max(), 5):
    print(f"{i} min dep delay   ", round(delays["delay_diff"][delays["dep_delay"].gt(i)].mean(), 2), "minutes made up for")
delays.describe()<jupyter_output><empty_output><jupyter_text>**Task 6**: How many states cover 50% of US air traffic?<jupyter_code>df2.head()
state_counts1 = df2["origin_city_name"].copy()
state_counts2 = df2["dest_city_name"].copy()
state_counts1 = [ele.split(", ")[1] for ele in list(state_counts1.values)]
state_counts2 = [ele.split(", ")[1] for ele in list(state_counts2.values)]
state_counts1 = pd.Series(state_counts1)
state_counts2 = pd.Series(state_counts2)
state_counts1 = state_counts1.value_counts()
state_counts2 = state_counts2.value_counts()
state_counts1[state_counts1.values.cumsum() <= state_counts1.sum() / 2]
figure6 = pyplot.figure(figsize=(15,9))

pyplot.plot(state_counts1.index, state_counts1.values.cumsum() / state_counts1.sum(), label="cumulative sum of flights")
pyplot.plot(state_counts1.index, [0.5 for state in state_counts1.index], label="50% line")

pyplot.title("Cumulative sum of flights by state")
pyplot.ylabel("Proportion")
pyplot.grid()
pyplot.legend()

pyplot.show();<jupyter_output><empty_output><jupyter_text>**Analysis**  7 States cover 50% of US air traffic- California- Texas- Florida- Illinois- Georgia- New York- North Carolina  **Task 7**: Test the hypothesis whether planes fly faster when there is the departure delay?<jupyter_code># calculating average difference between arr_delay and dep_delay
# negative numbers means plane arrived earlier than expected
for i in range(0, delays["dep_delay"].max(), 5):
    print(f"{i} min dep delay   ", round(delays["delay_diff"][delays["dep_delay"].gt(i)].mean(), 2), "difference")
x = range(0, delays["dep_delay"].max(), 5)
y = [abs(round(delays["delay_diff"][delays["dep_delay"].gt(delay)].mean(), 2)) for delay in x]
x
y
fig7 = pyplot.figure(figsize=(15,9))

pyplot.plot(x, y)

pyplot.title("AVG time saved when plane delayed during departure")
pyplot.xlabel("Departure delay in minutes")
pyplot.ylabel("Time saved in minutes (dep_delay - arr_delay)")
            
pyplot.show();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16053478086047446, 'fraction_numerical': 0.025432247687977482, 'mean_word_length': 6.6471944657955415, 'pattern_counts': {'":': 5, '<': 34, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '168280', 'license': 'MIT', 'n_tokens_mistral': 3682, 'n_tokens_neox': 3472, 'n_words': 878, 'path': 'exploratory_analysis_ransmpl.ipynb', 'repo_name': 'ZacharyArgentin/LHL_Midterm_Project_Predicting_Flight_Delays', 'size': '9948'}