<jupyter_start><jupyter_text>Handling null values in the Delay features, i.e when there were no Airline, Security or Weather delays the value was left blank, now filling 0's as values will not bias the data.<jupyter_code>df_flights['AIRLINE_DELAY'] = df_flights['AIRLINE_DELAY'].fillna(0)
df_flights['AIR_SYSTEM_DELAY'] = df_flights['AIR_SYSTEM_DELAY'].fillna(0)
df_flights['SECURITY_DELAY'] = df_flights['SECURITY_DELAY'].fillna(0)
df_flights['LATE_AIRCRAFT_DELAY'] = df_flights['LATE_AIRCRAFT_DELAY'].fillna(0)
df_flights['WEATHER_DELAY'] = df_flights['WEATHER_DELAY'].fillna(0)
# Counting the missing values in each variable

df_flights.isnull().mean().sort_values(ascending=False).plot.bar(figsize=(12,6))
plt.ylabel('Percentage of missing values')
plt.xlabel('Variables')
plt.title('Missing data')<jupyter_output><empty_output><jupyter_text>Examinig the CANCELLATION_REASON column to fill the NaN with letter 'NC' for our understanding that there was No Cancellation<jupyter_code>df_flights['CANCELLATION_REASON'].value_counts().sum()
df_flights['CANCELLATION_REASON'].isnull().sum()
5729195-89884
df_flights['CANCELLATION_REASON'].value_counts().plot.bar(figsize=(12,6))
plt.ylabel('Number of Reasons')
plt.xlabel('Reasons')
plt.title('Listing the Missing reasons')
# Converting NaN labels to NC
df_flights['CANCELLATION_REASON'] = df_flights['CANCELLATION_REASON'].fillna('NC')

# Verifying the change
df_flights['CANCELLATION_REASON'].value_counts()
# Plotting the missing values in each variable
df_flights.isnull().mean().sort_values(ascending=False).plot.bar(figsize=(12,6))
plt.ylabel('Percentage of missing values')
plt.xlabel('Variables')
plt.axhline(y=0.02, color='red') # highlight the 2% mark with a red line:
plt.title('Missing data')<jupyter_output><empty_output><jupyter_text>The reason for missing values in the above 3 features is due to the features DIVERTED or CANCELLED having value 1 (true), removing these null values of the 3 features will remove all the 1(true) records from DIVERTED or CANCELLED variables. We have to filter and ignore DIVERTED or CANCELLED flights while performing calculations on these columns.<jupyter_code># Visualize the variable distribution with histograms

df_flights.hist(bins=30, figsize=(12,12), density=True)
plt.show()<jupyter_output><empty_output><jupyter_text>Most of the numerical variables in the dataset are skewed.<jupyter_code># Determine the number of unique categories in each feature:
df_flights.nunique()<jupyter_output><empty_output><jupyter_text>Creating a date column by combining the 'YEAR','MONTH', 'DAY' columns<jupyter_code>df_flights['DATE'] = pd.to_datetime(df_flights[['YEAR','MONTH', 'DAY']])
df_flights["DATE"].head()<jupyter_output><empty_output><jupyter_text>Loading the Airlines dataset and merging airline name with df_flights<jupyter_code>df_airlines =pd.read_csv(r"C:\Users\91894\Desktop\iNEURON Internship\flights_Dataset\airlines.csv")
df_airlines
df_flights = df_flights.rename(columns={"AIRLINE":"IATA_CODE"})
df_merge = pd.merge(df_flights,df_airlines,on="IATA_CODE")
df_merge.head()<jupyter_output><empty_output><jupyter_text>Loading the Airports dataset and merging on origin and destination airport to get city, origin and destination state details<jupyter_code>df_airports = pd.read_csv(r"C:\Users\91894\Desktop\iNEURON Internship\flights_Dataset\airports.csv")
df_airports = df_airports.rename(columns={"IATA_CODE":"CODE"})
df_airports
# Merging the origin details
df = df_merge.merge(df_airports[['STATE','AIRPORT','CODE']], how = 'left',
                left_on = 'ORIGIN_AIRPORT', right_on = 'CODE').drop('CODE',axis=1)
df = df.rename(columns={"STATE":"ORIGIN_STATE","AIRPORT":"ORG_AIRPORT"})
df.head()
# Merging the destination details
df = df.merge(df_airports[['STATE','AIRPORT','CODE']], how = 'left',
                left_on = 'DESTINATION_AIRPORT', right_on = 'CODE').drop('CODE',axis=1)
df = df.rename(columns={"STATE":"DESTINATION_STATE","AIRPORT":"DES_AIRPORT"})
df.head()
df.head(5)
pd.set_option('display.max_columns', None)
df.dtypes<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13073338238901153, 'fraction_numerical': 0.013245033112582781, 'mean_word_length': 8.418013856812934, 'pattern_counts': {'":': 6, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '164598', 'license': 'MIT', 'n_tokens_mistral': 1477, 'n_tokens_neox': 1422, 'n_words': 292, 'path': 'flight_delays_and_cancellations_eda.ipynb', 'repo_name': 'Saravanan-Ganesh/iNEURON-Internship', 'size': '4077'}