<jupyter_start><jupyter_text>Importing Libraries<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Function for extracting data from the Flight Aware webpage<jupyter_code>def extract_data(flight_type):
    offset = [0, 20]
    data = []
    aircraft = []
    for i in offset:
        page = "https://flightaware.com/live/aircrafttype/"+str(flight_type)+"?;offset="+str(i)+";order=actualdeparturetime;sort=DESC"
        data = pd.read_html(page, attrs={"class":"prettyTable fullWidth"})
        if aircraft is None:
            aircraft = pd.DataFrame(data[0])
        else:
            df = pd.DataFrame(data[0])
            aircraft.append(df)
    return aircraft
flight_type = ["C208", "PC12", "C182"]
for i in flight_type:
    aircraft = extract_data(i)
    aircraft_table = aircraft[0].append(aircraft[1]).copy()
    aircraft_table.to_excel("Aircraft type "+i+".xlsx", sheet_name = i)<jupyter_output><empty_output><jupyter_text>ANALYSIS FOR AIRCRAFT TYPE C208  Reading the data for Aircraft Type C208 in the Jupyter Notebook<jupyter_code>c208 = pd.read_excel("/Users/apurvasalvi/Desktop/magniX/Aircraft type C208.xlsx")
c208
pd.DataFrame(c208.groupby(['Origin', 'Destination']).size())
pd.DataFrame(c208.groupby(['Origin']).size())<jupyter_output><empty_output><jupyter_text>Most airports had flights frequently going to the same location   Checking for NaN in the data<jupyter_code>c208.isnull().sum()<jupyter_output><empty_output><jupyter_text>After observing the above data, we can see that the Destination, EstimatedArrival Time, EstimatedTimeEnroute and EstimatedTimeEnroute (Min) columns have NaN values. It is safe to assume that the columns with Departure values missing could be the cancelled flights which did not depart from the location.  Tracking the cancelled flight in a different Dataframe called c208_cancelled.<jupyter_code>c208_cancelled = c208[c208['Destination'].isnull()]
c208_cancelled.Destination = c208_cancelled.Destination.fillna('Cancelled Flight')
c208_cancelled.drop(['EstimatedArrival Time', 'EstimatedTimeEnroute', 'EstimatedTimeEnroute (Min)'], axis=1)
c208_cancelled.Destination = c208_cancelled.Destination.fillna('Cancelled Flight')
c208_cancelled<jupyter_output><empty_output><jupyter_text>Visualizing the total number of flights which got cancelled from each of these airports<jupyter_code>c208_cancelled['Origin'].value_counts().plot(kind='bar')
plt.title("Number of cancelled flights")
plt.ylabel("Number of cancelled flights")
plt.xlabel("Name of the airport")<jupyter_output><empty_output><jupyter_text>The airport at Bethel has the most number of cancelled flights today  Dropping the Cancelled Flights from the original Dataframe<jupyter_code>c208.dropna(axis=0, subset=["Destination"], inplace=True)
c208<jupyter_output><empty_output><jupyter_text>There is still 1 flight which took off from Saipan Intl(PGSN) and landed in Guam Intl(PGUM) which did not record it's Departure and Estimated Arrival Time.<jupyter_code>c208['Origin'].value_counts().plot(kind='bar')
plt.title("Number of flights that took off from the airports")
plt.ylabel("Number of flights")
plt.xlabel("Name of the airport")<jupyter_output><empty_output><jupyter_text>Thus, for C208 we can conclude that maximum number of flight took off from the Baltimore/Washington Intl(KBWI) and the Airport at Bethel had the most number of cancelled flights.  ANALYSIS FOR AIRCRAFT TYPE C182  Reading the data for Aircraft Type C182 in Jupyter Notebook<jupyter_code>c182 = pd.read_excel("/Users/apurvasalvi/Desktop/magniX/Aircraft type C182.xlsx")
c182
c182.groupby(["Origin", "Destination"]).size()
c182.groupby(["Origin"]).size()<jupyter_output><empty_output><jupyter_text>Checking for NaN<jupyter_code>c182.isnull().sum()<jupyter_output><empty_output><jupyter_text>Out of the 39 flights of the type C182, we can see that 33 flights never made to their destination or got cancelled. Let's do some analysis on the cancelled flights<jupyter_code>c182_cancelled = c182[c182['Destination'].isnull()]
c182_cancelled.Destination = c182_cancelled.Destination.fillna('Cancelled Flight')
c182_cancelled.drop(['EstimatedArrival Time', 'EstimatedTimeEnroute', 'EstimatedTimeEnroute (Min)'], axis=1)<jupyter_output>/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py:5209: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self[name] = value<jupyter_text>Visualizing the number of flights that got cancelled from each airport<jupyter_code>c182_cancelled['Origin'].value_counts().plot(kind='bar')
plt.title("Number of cancelled flights")
plt.ylabel("Number of cancelled flights")
plt.xlabel("Name of the airport")<jupyter_output><empty_output><jupyter_text>The maximum number of flights that got cancelled were 2, and from the airports: Ogden-Hinckley (KOGD), Prescott Rgnl - Ernest a Love Fld (KPRC), Georgetown Muni (KGTU)  Dropping the Cancelled Flights from the original Dataframe<jupyter_code>c182.dropna(axis=0, subset=["Destination"], inplace=True)
c182<jupyter_output><empty_output><jupyter_text>Even in C182, there is still 1 flight which took off from Lancaster Rgnl(KLNC) and landed in Dallas Executive(KRBD) which did not record it's Departure and Estimated Arrival Time.<jupyter_code>c182['Origin'].value_counts().plot(kind='bar')
plt.title("Number of flights that took off from the airports")
plt.ylabel("Number of flights")
plt.xlabel("Name of the airport")<jupyter_output><empty_output><jupyter_text>ANALYSIS FOR AIRCRAFT TYPE PC12 Reading the data for aircraft type PC12 into Jupyter Notebook<jupyter_code>pc12 = pd.read_excel("/Users/apurvasalvi/Desktop/magniX/Aircraft type PC12.xlsx")
pc12
pc12.groupby(["Origin", "Destination"]).size()
pc12.groupby(["Origin"]).size()<jupyter_output><empty_output><jupyter_text>Checking for NaN<jupyter_code>pc12.isnull().sum()<jupyter_output><empty_output><jupyter_text>Out of the 33 flights of the type C182, we can see that only 5 flights never made to their destination or got cancelled. Let's do some analysis on the cancelled flights<jupyter_code>pc12_cancelled = pc12[pc12['Destination'].isnull()]
pc12_cancelled.Destination = pc12_cancelled.Destination.fillna('Cancelled Flight')
pc12_cancelled.drop(['EstimatedArrival Time', 'EstimatedTimeEnroute', 'EstimatedTimeEnroute (Min)'], axis=1)<jupyter_output><empty_output><jupyter_text>Visualizing the number of flights that got cancelled<jupyter_code>pc12_cancelled['Origin'].value_counts().plot(kind='bar')
plt.title("Number of cancelled flights")
plt.ylabel("Number of cancelled flights")
plt.xlabel("Name of the airport")<jupyter_output><empty_output><jupyter_text>Dropping the Cancelled Flights from the original Dataframe<jupyter_code>pc12.dropna(axis=0, subset=["Destination"], inplace=True)
pc12
pc12['Origin'].value_counts().plot(kind='bar')
plt.title("Number of flights that took off from the airports")
plt.ylabel("Number of flights")
plt.xlabel("Name of the airport")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10971396987702087, 'fraction_numerical': 0.028464833494541937, 'mean_word_length': 7.253135689851767, 'pattern_counts': {'":': 1, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '491507', 'license': 'MIT', 'n_tokens_mistral': 2545, 'n_tokens_neox': 2375, 'n_words': 594, 'path': 'MagniX Assignment 2.ipynb', 'repo_name': 'ApurvaSalvi/Magnix-Assessment', 'size': '7237'}