<jupyter_start><jupyter_text>Exploratory Data Analysis<jupyter_code># import the library
%matplotlib inline

import pandas as pd
import numpy as np
import scipy
import matplotlib.pyplot as plt
import seaborn as sns
import datetime
from scipy.stats import kurtosis
from scipy.stats import skew

sns.set_style('whitegrid')

pd.set_option('display.float_format', lambda x: '%.2f' % x)<jupyter_output><empty_output><jupyter_text>Data Perspective- One variable    - Numeric variables:        - continuous: average income        - discrete: population    - Categorical variables:        - ordinal: grade        - nominal: house, apartment, townhouse, etc.- Multiple variables:    - Numeric x Numeric    - Categorical x Numeric    - Categorical x Categorical  1. Load Cleaned Data<jupyter_code>df = pd.read_csv('../../Data/flightsmerged.csv', low_memory=False)
print(df.columns)
df.head()<jupyter_output>Index(['MONTH', 'FLIGHT_NUMBER', 'TAIL_NUMBER', 'ORIGIN_AC', 'DESTINATION_AC',
       'SCHEDULED_DEPARTURE', 'DEPARTURE_TIME', 'DEPARTURE_DELAY',
       'SCHEDULED_TIME', 'ELAPSED_TIME', 'AIR_TIME', 'DISTANCE',
       'SCHEDULED_ARRIVAL', 'ARRIVAL_TIME', 'ARRIVAL_DELAY', 'DIVERTED',
       'CANCELLED', 'CANCELLATION_REASON', 'AIR_SYSTEM_DELAY',
       'SECURITY_DELAY', 'AIRLINE_DELAY', 'LATE_AIRCRAFT_DELAY',
       'WEATHER_DELAY', 'DATE', 'AIRLINE_CODE', 'AIRLINE_NAME',
       'ORIGIN_AIRPORT', 'DESTINATION_AIRPORT'],
      dtype='object')<jupyter_text>2. One Variable  2.1. Numeric<jupyter_code>#Analyze arrival delay distribution

print('Arrival delays (minutes)')
print(df['ARRIVAL_DELAY'].describe())
plt.hist(df['ARRIVAL_DELAY'], bins=100)
plt.title('Mean Arrival Delays')
plt.savefig("Charts/arrival.png", dpi=100)
plt.show()

print("mean : ", np.mean(df['ARRIVAL_DELAY']))
print("variance  : ", np.var(df['ARRIVAL_DELAY']))
print("skewness : ",df['ARRIVAL_DELAY'].skew())
print("kurtosis : ",df['ARRIVAL_DELAY'].kurtosis())
print('')

print('Departure delays (minutes)')
print(df['DEPARTURE_DELAY'].describe())
plt.hist(df['DEPARTURE_DELAY'], bins=100)
plt.title('Mean Departure Delays')
plt.savefig("Charts/departure.png", dpi=100)
plt.show()

print("mean : ", np.mean(df['DEPARTURE_DELAY']))
print("variance  : ", np.var(df['DEPARTURE_DELAY']))
print("skewness : ",df['DEPARTURE_DELAY'].skew())
print("kurtosis : ",df['DEPARTURE_DELAY'].kurtosis())<jupyter_output>Arrival delays (minutes)
count   5593995.00
mean          0.34
std          23.54
min         -87.00
25%         -13.00
50%          -5.00
75%           6.00
max         123.00
Name: ARRIVAL_DELAY, dtype: float64<jupyter_text>Conclusion: The arrival and departure delay data is highly positively skewed with heavy outliers   2.2. Categorical<jupyter_code>print(df['CANCELLATION_REASON'].value_counts())

# Bar Chart
plt.figure(figsize=(12,6))
plt.title('# Flight Cancelled')
plt.xlabel('Cancellation Reasons')
plt.ylabel('# Flights')

labels = df['CANCELLATION_REASON'].value_counts().index
values = df['CANCELLATION_REASON'].value_counts().values

y_pos = range(len(labels))
plt.bar(y_pos, values, align='center', alpha=0.7, color=['red','purple','navy','gray'], width=0.5)
plt.xticks(y_pos, labels)

plt.savefig('Charts/cancellation_reasons.png')
plt.show()<jupyter_output>Weather                48538
Airline                25008
National Air System    15702
Security                  22
Name: CANCELLATION_REASON, dtype: int64<jupyter_text>Conclusion: 'Weather' is the main reason of flight cancellation.  3. Multiple Variables  3.1. Numeric X Numeric<jupyter_code>#Correlation

correlation_matrix = df.corr()
plt.figure(figsize=(16, 9))
sns.heatmap(correlation_matrix, square=True)
plt.show()<jupyter_output><empty_output><jupyter_text>3.2. Numeric X Categorical<jupyter_code># Linechart of arrival delay distribution by month 
f,ax=plt.subplots(1,2,figsize=(12,6))

df[['MONTH','ARRIVAL_DELAY']].groupby(['MONTH']).mean().plot(ax=ax[0])
ax[0].set_title('Average delay per month')

df[['MONTH','ARRIVAL_DELAY']].groupby(['MONTH']).sum().plot(ax=ax[1],color=['purple'])
ax[1].set_title('Delay (minutes) per month')
plt.show()<jupyter_output><empty_output><jupyter_text>Conclusion: April has the lowest average arrival delays and June has highest   4. Business Prespective  4.1. What is the average delay for each airline ?<jupyter_code>airline_arr_group = df.groupby('AIRLINE_NAME')['ARRIVAL_DELAY'].mean().round().sort_values()
airline_dep_group = df.groupby('AIRLINE_NAME')['DEPARTURE_DELAY'].mean().round().sort_values()

#Plotting
pd.concat(
    [airline_arr_group.rename('Mean Arrival Delay'), airline_dep_group.rename('Mean Departure Delay')],
    axis=1, sort=False).plot.bar(figsize=(16,8), title='Mean Delays for Airlines' )

#Another Visualization

fig = plt.figure(figsize=(12, 16))
ax1 = plt.subplot(211)

plt.title('Average Arrival Delays for Airlines')
plt.xlabel('Average delay per minutes')
plt.ylabel('Airlines Names')
sns.barplot(x=airline_arr_group.values, y=airline_arr_group.index, orient='h')

ax2 = plt.subplot(212, sharex = ax1, sharey = ax1)

plt.title('Average Departure Delays for Airlines')
plt.xlabel('Average delay per minutes')
plt.ylabel('Airlines Names')
sns.barplot(x=airline_dep_group.values, y=airline_dep_group.index, orient='h')

plt.show()<jupyter_output><empty_output><jupyter_text>4.2. What is the average arrival and departure delay times based on airport ?<jupyter_code>origin_airport_group = df.groupby(by='ORIGIN_AIRPORT')['DEPARTURE_DELAY'].mean().round().sort_values()
dest_airport_group = df.groupby(by='DESTINATION_AIRPORT')['ARRIVAL_DELAY'].mean().round().sort_values()

print("Average Departure delays:\n",origin_airport_group)
print("_______________________\n")
print("Average Arrival delays:\n",dest_airport_group)<jupyter_output>Average Departure delays:
 ORIGIN_AIRPORT
Valdez Airport                                                           -10.00
Yakutat Airport                                                           -8.00
Canyonlands Field                                                         -6.00
Pocatello Regional Airport                                                -6.00
Elko Regional Airport                                                     -6.00
Merle K. (Mudhole) Smith Airport                                          -5.00
Ford Airport                                                              -4.00
Lewiston-Nez Perce County Airport                                         -4.00
Dickinson Theodore Roosevelt Regional Airport                             -4.00
Bert Mooney Airport                                                       -4.00
Bellingham International Airport                                          -4.00
Brainerd Lakes Regional Airport                                           -3.0[...]<jupyter_text>Conclusion:1. Flights departing from Gustavus Airport have the highest average delays.2. Flights arriving to St. Cloud Regional Airport have the highest average delays.3. Flights depart early from Valdez Airport, and also arrive early to it.  4.3. What is the impact of the weather on flights?<jupyter_code>#As discussed above, weather is the most common reason for cancellation.
#For more illustration:

month_cancel_group = df.groupby('MONTH')['CANCELLED'].sum().sort_values()
print(month_cancel_group)

plt.figure(figsize=(12,7))

plt.title('Cancelled flights per month')
plt.xlabel('# Cancelled flights')
plt.ylabel('Month')
sns.barplot(x=month_cancel_group.values, y=month_cancel_group.index, orient='h')

plt.show()<jupyter_output>MONTH
9      2060
10     2434
4      4458
11     4567
7      4764
8      5015
5      5632
12     7992
6      9034
3     10950
1     11938
2     20426
Name: CANCELLED, dtype: int64
{'fraction_non_alphanumeric': 0.12780880633848551, 'fraction_numerical': 0.031172879594752564, 'mean_word_length': 3.6190761847630473, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '188775', 'license': 'MIT', 'n_tokens_mistral': 2731, 'n_tokens_neox': 2533, 'n_words': 536, 'path': 'Code/Cleaning_and_Exploration/Exploratory-Data-Analysis.ipynb', 'repo_name': 'ArwaSheraky/Airlines-Delay-Analysis', 'size': '7699'}