<jupyter_start><jupyter_text>Assignment 2 [Part A] Cleaning and EDA**_<NAME>,  NUID:  001834167_** Dataset Description:    The dataset I picked is a collection of the U.S airline delay information on Jan. 1st, 2015. Such 9 kinds of fields included in the raw datesets:* **AIRLINE, FLIGHT_NUMBER*** **ORIGIN_AIRPORT, DESTINATION_AIRPORT*** **SCHEDULED_DEPARTURE, DEPARTURE_TIME, DEPARTURE_DELAY*** **DISTANCE*** **SCHEDULED_ARRIVAL, ARRIVAL_TIME, ARRIVAL_DELAY*** DIVERTED* CANCELLED (Flight Cancelled (1 = cancelled))* CANCELLATION_REASON (Reason for Cancellation of flight: A - Airline/Carrier; B - Weather; C - National Air System; D - Security)* **AIR_SYSTEM_DELAY, SECURITY_DELAY, AIRLINE_DELAY, LATE_AIRCRAFT_DELAY, WEATHER_DELAY**31 columns of data included totally for above kinds of datadata tag in **bold** will be keep in the dataset, other columns of data will be discarded<jupyter_code>%matplotlib inline 
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats
import seaborn as sns

# Make plots larger
plt.rcParams['figure.figsize'] = (16, 10)
#read flights.csv in flights
flights = pd.read_csv('flights.csv', sep=',')

## show the data samples of first 10 lines
flights.head(n=10)<jupyter_output><empty_output><jupyter_text>Discard Columns* **YEAR, MONTH, DAY, DAY_OF_WEEK** are discarded because all the flights wecdoing research are in the Jan. 1st, 2015.* **TAIL_NUMBER** has no meaning to this report.* **TAXI_OUT, WHEELS_OFF, SCHEDULED_TIME, ELAPSED_TIME, AIR_TIME, WHEELS_ON, TAXI_IN** are discarded because the time are so concreate that we do not need that kind of data.<jupyter_code>flights2 = flights.drop(['YEAR','MONTH','DAY','DAY_OF_WEEK','TAIL_NUMBER','TAXI_OUT','WHEELS_OFF','SCHEDULED_TIME','ELAPSED_TIME','AIR_TIME','WHEELS_ON','TAXI_IN','CANCELLATION_REASON'],axis=1)
flights2.head(3000)<jupyter_output><empty_output><jupyter_text>Data cleaning * Are there missing values? ** Answer: There are some missing cancel or delay values in the list because some flights are on time.**<jupyter_code>flights2.isnull().sum()
flights2.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 13950 entries, 0 to 13949
Data columns (total 18 columns):
AIRLINE                13950 non-null object
FLIGHT_NUMBER          13950 non-null int64
ORIGIN_AIRPORT         13950 non-null object
DESTINATION_AIRPORT    13950 non-null object
SCHEDULED_DEPARTURE    13950 non-null int64
DEPARTURE_TIME         13490 non-null float64
DEPARTURE_DELAY        13490 non-null float64
DISTANCE               13950 non-null int64
SCHEDULED_ARRIVAL      13950 non-null int64
ARRIVAL_TIME           13480 non-null float64
ARRIVAL_DELAY          13464 non-null float64
DIVERTED               13950 non-null int64
CANCELLED              13950 non-null int64
AIR_SYSTEM_DELAY       2987 non-null float64
SECURITY_DELAY         2987 non-null float64
AIRLINE_DELAY          2987 non-null float64
LATE_AIRCRAFT_DELAY    2987 non-null float64
WEATHER_DELAY          2987 non-null float64
dtypes: float64(9), int64(6), object(3)
memory usage: 1.9+ MB<jupyter_text>* Are there inappropraite values? * Remove or impute any bad data. 1. **DIVERTED**: If the filght is diverted, the result of it is not meaningful because it does not have any information about delay.2. **CANCELLED**: If the flight is cancelled, the result of it is not meaningful because it does not have any information about delay.<jupyter_code># Check how much lines of data with flight diverted

(flights2['DIVERTED']==1).sum()

## 20 lines will be discarded
# Check how much lines of data with flight cancelled

(flights2['CANCELLED']==1).sum()

## 466 lines will be discarded
# Show all the data whose flight is not diverted and not cancelled, then store them in flights_valid.
flights3 = flights2.loc[flights2['DIVERTED']==0]
flights_valid = flights3.loc[flights3['CANCELLED']==0]
flights_valid
# Show the mean of the flight ARRIVAL DELAY time (in Minutes)
flights_valid['ARRIVAL_DELAY'].mean()<jupyter_output><empty_output><jupyter_text>3. [SCHEDULED_DEPARTURE], [DEPARTURE_TIME], [SCHEDULED_ARRIVAL], [ARRIVAL_TIME] should lower than 2400 (24:00), which represent the time.    ** The line of which columns has a value more than 2400 will be removed.**<jupyter_code>(flights_valid['SCHEDULED_DEPARTURE']>2400).sum()
(flights_valid['DEPARTURE_TIME']>2400).sum()
(flights_valid['SCHEDULED_ARRIVAL']>2400).sum()
(flights_valid['ARRIVAL_TIME']>2400).sum()<jupyter_output><empty_output><jupyter_text>Answer the following questions for the data in each column:* How is the data distributed? 1. Data of **['DEPARTURE_DELAY']** is **Normal Distribution**, because the both extreme advance departure time and extreme late departure time have the least possibility.2. Data of **['ARRIVAL_DELAY']** is **Normal Distribution**, because the both extreme advance arrival time and extreme late arrival time have the least possibility.3. Data of **['SCHEDULED_DEPARTURE'], ['DEPARTURE_TIME'], ['SCHEDULED_ARRIVAL'], ['ARRIVAL_TIME']** should be **Normal Distribution**, because the time value in the middle of the day has the greatset possibility, the time value on the earliest side and latest side has the least possibility.4. Data from **['AIR_SYSTEM_DELAY'] → ['WEATHER_DELAY']** should be **Normal Distribution**, because the value in the middle has the greatest possibility, the value on the smallest side and largest side has the least possibility. * What are the summary statistics?    1. column **['DEPARTURE_DELAY']**    2. column **['ARRIVAL_DELAY']**    3. column **['SCHEDULED_DEPARTURE'], ['DEPARTURE_TIME'], ['SCHEDULED_ARRIVAL'], ['ARRIVAL_TIME']** have the summary statistics of familiar type.    4. column **['AIR_SYSTEM_DELAY'] → ['WEATHER_DELAY']** have the summary statistics of familiar type.<jupyter_code>flights_valid.head()
flights_valid.describe()
flights_valid.info()
flights_valid.shape
# sample for the data column in Group 1

flights_valid.DEPARTURE_DELAY
# sample for the data column in Group 2

flights_valid.ARRIVAL_DELAY
# head for several representative columns of data

flights_valid[['AIRLINE','FLIGHT_NUMBER','DEPARTURE_DELAY','ARRIVAL_DELAY','AIR_SYSTEM_DELAY']].head()<jupyter_output><empty_output><jupyter_text>* Are there anomalies/outliers? Yes, there are.  Plot each colmun as appropriate for the data type:* Write a summary of what the plot tells you.<jupyter_code>plt.show(flights_valid[['SCHEDULED_DEPARTURE','DEPARTURE_TIME','SCHEDULED_ARRIVAL','ARRIVAL_TIME']].plot(kind='box'))<jupyter_output><empty_output><jupyter_text>What could be read from the boxplot above?The boxplot above for Departure and Arrival time could tell me the estimate value range, maximum value, 75th percentile, 50 percentile(mean), 25 percentile, minimum value. To the meaning of the time, * SCHEDULE_DEPARTURE and DEPARTURE_TIME could be match with the **Normal Distribution** model because the mean of the value range is in the middle.* SCHEDULE_ARRIVAL and ARRIVAL_TIME are somewhat like SCHEDULE_DEPARTURE and DEPARTURE_TIME with a larger mean of the value because the flight time.<jupyter_code>plt.show(flights_valid[['DEPARTURE_DELAY','ARRIVAL_DELAY']].plot(kind='box'))<jupyter_output><empty_output><jupyter_text>What could be read from the boxplot above?* More than 50 precent of the flights departure in delay, however less than 50 percent of the flights arrive in delay.* Some flights data are so late that they are anomalies to the boxplot.<jupyter_code>plt.show(sns.lmplot(x='DEPARTURE_DELAY',y='ARRIVAL_DELAY',data=flights_valid))<jupyter_output><empty_output><jupyter_text>What could be read from the plot above?* There are linear realtion between DEPARTURE_DELAY and ARRIVAL_DELAY<jupyter_code>flights_valid['AIR_SYSTEM_DELAY'].hist(bins=15)
flights_valid['SECURITY_DELAY'].hist(bins=15)
flights_valid['AIRLINE_DELAY'].hist(bins=15)
flights_valid['LATE_AIRCRAFT_DELAY'].hist(bins=15)
flights_valid['WEATHER_DELAY'].hist(bins=15)
flights_valid['ARRIVAL_DELAY'].hist(bins=15)<jupyter_output><empty_output><jupyter_text>What could be read from the histogram above?* It is obivious that the ARRIVAL_DELAY follows the model of Normal distribution. With most of the data distribute in an fixed middle, which is estimately around zero point.<jupyter_code>sns.distplot(flights_valid['ARRIVAL_DELAY'])<jupyter_output>/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "
{'fraction_non_alphanumeric': 0.09758656873032528, 'fraction_numerical': 0.030779993004547045, 'mean_word_length': 6.083402146985962, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '43082', 'license': 'MIT', 'n_tokens_mistral': 2943, 'n_tokens_neox': 2748, 'n_words': 877, 'path': 'Assignment 2/code/.ipynb_checkpoints/Assignment2_Bo Cao-checkpoint.ipynb', 'repo_name': 'haipiao/INFO7390', 'size': '8577'}