<jupyter_start><jupyter_text>Exploratory Data AnalysisUse this notebook to get familiar with the datasets we have. There is 10 questions we need to answer during the EDA.We shouldn't limit our EDA to these 10 questions. Let's be creative :).<jupyter_code>import pandas as pd
import numpy as np
import datetime
import matplotlib.pyplot as plt
import seaborn as sns
from datetime import datetime
flightsDF = pd.read_csv('Flights-300K.csv')
flightsDF.isna().count()
# Create DF with avg arr_delay times per day.
avg_delay = pd.read_csv('Avg_Arr_Dep_Delay.csv')

# Converting fl_date object to DateTime
avg_delay['fl_date'] = pd.to_datetime(avg_delay['fl_date'],
                              format='%Y-%m-%d')
#Groupby to compare monthly averages in delays
# NOTE: Negative values (early arrivals) ARE INCLUDED
month_arr = avg_delay.groupby(avg_delay['fl_date'].dt.strftime('%m'))['avg_arr_delay'].mean()
month_arr = month_arr.to_frame()
month_dep = avg_delay.groupby(avg_delay['fl_date'].dt.strftime('%m'))['avg_dep_delay'].mean()
month_dep = month_dep.to_frame()

#Resetting the index 
month_arr = month_arr.reset_index()
month_dep = month_dep.reset_index()
avg_delay['avg_arr_delay'].describe()<jupyter_output><empty_output><jupyter_text>**Task 1**: Test the hypothesis that the delay is from Normal distribution. and that **mean** of the delay is 0. Be careful about the outliers.<jupyter_code>#Plotting to test normal distribution Avg Arrival Delay data
sns.distplot(avg_delay['avg_arr_delay'])
#Boxplotting Arrival Delays to check for outliers
sns.boxplot(avg_delay['avg_arr_delay'])
#Plotting to test normal distribution using Avg Departure Delay data
sns.distplot(avg_delay['avg_dep_delay'])
#Boxplotting Departure Delays to check for outliers
sns.boxplot(avg_delay['avg_dep_delay'])<jupyter_output>/home/codez/anaconda3/lib/python3.7/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>**Task 2**: Is average/median monthly delay different during the year? If yes, which are months with the biggest delays and what could be the reason?<jupyter_code># Plotting Monthly Average Arrival Delays to test normal distribution using "group" data
sns.barplot(x=month_arr['fl_date'], y=month_arr['avg_arr_delay'], palette="Blues")
# Plotting Monthly Average Departure Delays to test normal distribution using "group" data
sns.barplot(x=month_dep['fl_date'], y=month_dep['avg_dep_delay'], palette='Blues')
num_daily_fl = pd.read_csv('Number_of_fl.csv')
num_daily_fl['fl_date'] = pd.to_datetime(num_daily_fl['fl_date'],
                              format='%Y-%m-%d')
month_fl = num_daily_fl.groupby(num_daily_fl['fl_date'].dt.strftime('%m'))['number_of_fl'].mean()
month_fl = month_fl.reset_index()
month_fl
sns.barplot(x=month_fl['fl_date'], y=month_fl['number_of_fl'], palette='Blues')<jupyter_output><empty_output><jupyter_text>**Task 3**: Does the weather affect the delay? Use the API to pull the weather information for flights. There is no need to get weather for ALL flights. We can choose the right representative sample. Let's focus on four weather types:- sunny- cloudy- rainy- snow.Test the hypothesis that these 4 delays are from the same distribution. If they are not, which ones are significantly different?  **Task 4**: How taxi times changing during the day? Does higher traffic lead to bigger taxi times?  **Task 5**: What is the average percentage of delays that is already created before departure? (aka are arrival delays caused by departure delays?) Are airlines able to lower the delay during the flights?  **Task 6**: How many states cover 50% of US air traffic?   **Task 7**: Test the hypothesis whether planes fly faster when there is the departure delay?   **Task 8**: When (which hour) do most 'LONG', 'SHORT', 'MEDIUM' haul flights take off?  **Task 9**: Find the top 10 the bussiest airports. Does the biggest number of flights mean that the biggest number of passengers went through the particular airport? How much traffic do these 10 airports cover?<jupyter_code>daily_ORD = pd.read_csv('Avg_Daily_ORD.csv')
daily_ATL = pd.read_csv('Avg_Daily_ATL.csv')
daily_DFW = pd.read_csv('Avg_Daily_DFW.csv')
daily_DEN = pd.read_csv('Avg_Daily_DEN.csv')
daily_CLT = pd.read_csv('Avg_Daily_CLT.csv')
daily_LAX = pd.read_csv('Avg_Daily_LAX.csv')
daily_SEA = pd.read_csv('Avg_Daily_SEA.csv')
daily_SFO = pd.read_csv('Avg_Daily_SFO.csv')
daily_PHX = pd.read_csv('Avg_Daily_PHX.csv')
daily_IAH = pd.read_csv('Avg_Daily_IAH.csv')
daily_ORD.head()
sns.displot(daily_ORD['avg_dep_delay'])
sns.boxplot(daily_ORD['avg_dep_delay'])
sns.histplot(daily_ATL['avg_dep_delay'])
sns.boxplot(daily_ATL['avg_dep_delay'])
sns.histplot(daily_DFW['avg_dep_delay'])
sns.boxplot(daily_DFW['avg_dep_delay'])
sns.histplot(daily_DEN['avg_dep_delay'])
sns.boxplot(daily_DEN['avg_dep_delay'])
sns.histplot(daily_CLT['avg_dep_delay'])
sns.boxplot(daily_CLT['avg_dep_delay'])
sns.histplot(daily_LAX['avg_dep_delay'])
sns.boxplot(daily_LAX['avg_dep_delay'])
sns.histplot(daily_SEA['avg_dep_delay'])
sns.boxplot(daily_SEA['avg_dep_delay'])
sns.histplot(daily_SFO['avg_dep_delay'])
sns.boxplot(daily_SFO['avg_dep_delay'])
sns.histplot(daily_PHX['avg_dep_delay'])
sns.boxplot(daily_PHX['avg_dep_delay'])
sns.histplot(daily_IAH['avg_dep_delay'])
sns.boxplot(daily_IAH['avg_dep_delay'])
passengersDF = pd.read_csv('Passengers-300K.csv')
avgMonthlyPassengers = passengersDF.passengers.groupby(passengersDF['month']).mean()
avgMonthlyPassengers = avgMonthlyPassengers.reset_index
avgMonthlyPassengers
sns.barplot(x=avgMonthlyPassengers['month'], y=avgMonthlyPassengers)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1141779530258872, 'fraction_numerical': 0.006000342876735814, 'mean_word_length': 6.926630434782608, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '225348', 'license': 'MIT', 'n_tokens_mistral': 2056, 'n_tokens_neox': 1930, 'n_words': 510, 'path': 'CodyDir/Data Exploration and Model Testing /.ipynb_checkpoints/exploratory_analysis_draft-checkpoint.ipynb', 'repo_name': 'CodezHere/Midterm-Project-I', 'size': '5833'}