<jupyter_start><jupyter_text>Assignment 2 Part B - Cluster Some Data**_<NAME>,  NUID:  001834167_** Dataset Description:    The dataset I picked is a collection of the U.S airline delay information on Jan. 1st, 2015. Such 9 kinds of fields included in the raw datesets:* **AIRLINE, FLIGHT_NUMBER*** **ORIGIN_AIRPORT, DESTINATION_AIRPORT*** **SCHEDULED_DEPARTURE, DEPARTURE_TIME, DEPARTURE_DELAY*** **DISTANCE*** **SCHEDULED_ARRIVAL, ARRIVAL_TIME, ARRIVAL_DELAY*** DIVERTED* CANCELLED (Flight Cancelled (1 = cancelled))* CANCELLATION_REASON (Reason for Cancellation of flight: A - Airline/Carrier; B - Weather; C - National Air System; D - Security)* **AIR_SYSTEM_DELAY, SECURITY_DELAY, AIRLINE_DELAY, LATE_AIRCRAFT_DELAY, WEATHER_DELAY**31 columns of data included totally for above kinds of datadata tag in **bold** will be keep in the dataset, other columns of data will be discarded<jupyter_code>%matplotlib inline
from collections import Counter
import random

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from sklearn.cluster import KMeans, DBSCAN
from scipy.stats import multivariate_normal

# Make plots larger
plt.rcParams['figure.figsize'] = (16, 10)
#read flights.csv in flights
flights = pd.read_csv('flights.csv', sep=',')

## show the data samples of first 10 lines
flights.head(n=10)
# Drop inapporate data and make the dataset clean.
flights2 = flights.drop(['YEAR','MONTH','DAY','DAY_OF_WEEK','TAIL_NUMBER','ORIGIN_AIRPORT','DESTINATION_AIRPORT','TAXI_OUT','WHEELS_OFF','SCHEDULED_TIME','ELAPSED_TIME','AIR_TIME','WHEELS_ON','TAXI_IN','CANCELLATION_REASON'],axis=1)
flights2.head(3000)
# Show all the data whose flight is not diverted and not cancelled, then store them in flights_valid.
flights3 = flights2.loc[flights2['DIVERTED']==0]
flights_valid = flights3.loc[flights3['CANCELLED']==0]
flights_valid = flights_valid[flights_valid['ARRIVAL_DELAY'] <=180]
flights_valid= flights_valid.drop(['DIVERTED','CANCELLED','DEPARTURE_TIME','ARRIVAL_TIME','AIRLINE','SCHEDULED_ARRIVAL','FLIGHT_NUMBER','DEPARTURE_DELAY','AIR_SYSTEM_DELAY','SECURITY_DELAY','AIRLINE_DELAY','LATE_AIRCRAFT_DELAY','WEATHER_DELAY'],axis=1)

flights_valid

# setting color function
def set_colors(labels, colors='rgbykcmw'):
    colored_labels = []
    for label in labels:
        if (label < 0) or (label > 6):
          colored_labels.append(colors[7]) 
        else:
          colored_labels.append(colors[label])
    return colored_labels<jupyter_output><empty_output><jupyter_text>K-means method clusters<jupyter_code># Fit a k-means estimator
X = np.array(flights_valid)
estimator = KMeans(n_clusters = 6)
estimator.fit(X)

# Clusters are given in the labels_ attribute
labels = estimator.labels_
print (labels)
colors = set_colors(labels)
plt.scatter(flights_valid['DISTANCE'],flights_valid['SCHEDULED_DEPARTURE'], c=colors)
plt.title("KMeans k=5")
plt.xlabel("DISTANCE")
plt.ylabel("SCHEDULED_DEPARTURE")
plt.show()
colors = set_colors(labels)
plt.scatter(flights_valid['DISTANCE'],flights_valid['ARRIVAL_DELAY'], c=colors)
plt.title("KMeans k=5")
plt.xlabel("DISTANCE")
plt.ylabel("ARRIVAL_DELAY")
plt.show()<jupyter_output><empty_output><jupyter_text>What could be read from the diagram above?We use K-means method to get the clusters. From that, we can see that there are five clusters in DISTANCE-SCHEDULED_DEPARTURE axis plot. The dataset can be divided into short-flight, medium-flight and long-flight, and each of them has morning-noon and afternoon-evening time parts. Also we can see that short-flight and medium-flight flights are easily to get delay than the long-flight.   DBSCAN method cluster<jupyter_code># Fit a DBSCAN estimator
estimator = DBSCAN(eps=15, min_samples=10)
X = flights_valid[["DISTANCE","ARRIVAL_DELAY"]]
estimator.fit(X)
# Clusters are given in the labels_ attribute
labels = estimator.labels_

colors = set_colors(labels)
plt.scatter(flights_valid['DISTANCE'],flights_valid['ARRIVAL_DELAY'], c=colors)
plt.xlabel("DISTANCE")
plt.ylabel("ARRIVAL_DELAY")
plt.show()
# Fit a DBSCAN estimator
estimator = DBSCAN(eps=50, min_samples=10)
X = flights_valid[["DISTANCE","SCHEDULED_DEPARTURE"]]
estimator.fit(X)
# Clusters are given in the labels_ attribute
labels = estimator.labels_

colors = set_colors(labels)
plt.scatter(flights_valid['DISTANCE'],flights_valid['SCHEDULED_DEPARTURE'], c=colors)
plt.xlabel("DISTANCE")
plt.ylabel("SCHEDULED_DEPARTURE")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11665920286609942, 'fraction_numerical': 0.013210927004030452, 'mean_word_length': 7.151459854014599, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '533095', 'license': 'MIT', 'n_tokens_mistral': 1626, 'n_tokens_neox': 1535, 'n_words': 377, 'path': 'Assignment 2/code/Assignment2_Part B.ipynb', 'repo_name': 'haipiao/INFO7390', 'size': '4466'}