<jupyter_start><jupyter_text>Auto-Mpg Data<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

columns = [
    'mpg', 'cylinders', 'displacement', 'horsepower', 'weight', 'acceleration',
    'model year', 'origin', 'car name'
]
cars = pd.read_table('./resources/auto-mpg.data',delim_whitespace=True,names = columns)
print(cars.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 398 entries, 0 to 397
Data columns (total 9 columns):
mpg             398 non-null float64
cylinders       398 non-null int64
displacement    398 non-null float64
horsepower      398 non-null object
weight          398 non-null float64
acceleration    398 non-null float64
model year      398 non-null int64
origin          398 non-null int64
car name        398 non-null object
dtypes: float64(4), int64(3), object(2)
memory usage: 28.1+ KB
None<jupyter_text>Observations:* All records have 398 values.* Cols horsepower and car name has datatypes as object, but in the about sample data we can horsepower is numeric value. So lets check<jupyter_code>cars['horsepower'] = pd.to_numeric(cars['horsepower'], errors = 'coerce')
cars = cars[np.isfinite(cars['horsepower'])]<jupyter_output><empty_output><jupyter_text>Data AnalysisIn data, it's always about seeing is believing and the best way to see the data is using plots.We shall do this data analysis in two stages. In stage 1, I want to see how the data looks like for each columns with out connecting to other informations. This will helps us to understand following questions* What kind mpg/horsepower/cylinders of the car models are being released into the marked?* In second stage, we will try to see the correlations between these individual features,Does difference in mpg/horsepower/cylinders has an effect on mpg/horsepower/cylinders?<jupyter_code>tmp = plt.subplots(3, 3, figsize=(16, 16))
tmp = tmp[1].flatten().tolist()
for i, col in enumerate(['mpg', 'cylinders', 'displacement', 'horsepower', 'weight',
                         'acceleration', 'model year', 'origin', 'car name']):
    try:
        sns.distplot(cars[col], kde=False, ax=tmp[i])
    except:
        print(col);
plt.figure(figsize=(16.5, 4))
sns.distplot(cars['car name'].value_counts().values, kde=False)<jupyter_output>/Users/jkchang/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "<jupyter_text>As you can see, most the car names, we have in our raw data frame are unique.* There 241 unique car name, with only 1 record.* There 035 car names with 2 records(per model)* There 012 car names with 3 records(per model)* There 006 car names with 4 records(per model)* There 003 car names with 5 records(per model)<jupyter_code>from collections import Counter
Counter(cars['car name'].value_counts().values)
cols = ['mpg', 'cylinders', 'displacement', 'horsepower', 'weight', 'acceleration']
sns.pairplot(cars[cols])<jupyter_output><empty_output><jupyter_text>Observation* mpg seems to have negitive correlation displacement, horsepower & weight.* displacement seems to have positive correlation horsepower & weight.<jupyter_code>sns.heatmap([[0, 1], [1, 0]])
cars.mpg.corr(-1 * cars.mpg), cars.mpg.corr(-1 * cars.mpg)
cols_corr_matrix = [[cars[col].corr(cars[_]) for _ in cols] for col in cols]

cols_corr_matrix = pd.DataFrame(cols_corr_matrix, index=cols, columns=cols)
plt.figure(figsize=(9, 5))
sns.heatmap(cols_corr_matrix)
cols_corr_matrix
# Let check those correlations that can have impact.
#  I am taking a benchmark of .8 => it means something like 80% of time a change in `x` effects changes in `y` 

effective_cols_corr_matrix = cols_corr_matrix.applymap(lambda x: x if ((x < -0.7) or (x > .7)) and (x < .999) else 0)

sns.heatmap(effective_cols_corr_matrix)
effective_cols_corr_matrix<jupyter_output><empty_output><jupyter_text>Notes:black - Negitive correlation; white - Postive correlation* Accelation of a vehicle models is an independent of other.  `Example: A lower weight & low Horsepower Vehicle model can give same acceleration as high Weight, high Horsepower and vice-versa.`  * As number of Cylinder/Horsepower increase, we can positive impact/increase in Horsepower/Cylinder and weight.  `Example:I guess we can say that people/user who like to travel more, prefer vehicle models with more Horsepower/Cylinders.`  * Mileage/Weight is generally a tradeoff of Cylinder/Horsepower.  `Example: We can say that Mileage car models, generally prefered to short distances and tend to have lower weight along with low Horsepower.`<jupyter_code>import sklearn.cluster
clf = sklearn.cluster.KMeans(n_clusters=8)
clf = clf.fit(cars[cols])

cars['cluster'] = clf.predict(cars[cols])
tmp = plt.subplots(3, 3, figsize=(16, 16))
tmp = tmp[1].flatten().tolist()
for i, col in enumerate(['mpg', 'cylinders', 'displacement', 'horsepower', 'weight',
                         'acceleration',
                         'model year', 'origin',
                         'cluster'
                        ]):
    try:
        for cluter_i in range(len(clf.cluster_centers_)):
            sns.distplot(cars[cars.cluster == cluter_i][col], hist=True, ax=tmp[i])
    except:
        print(col)
clf = sklearn.cluster.KMeans(n_clusters=3)
clf = clf.fit(cars[cols])
cars['cluster'] = clf.predict(cars[cols])

tmp = plt.subplots(3, 3, figsize=(16, 16))
tmp = tmp[1].flatten().tolist()
for i, col in enumerate(['mpg', 'cylinders', 'displacement', 'horsepower', 'weight',
                         'acceleration',
                         'model year', 'origin',
                         'cluster'
                        ]):
    try:
        for cluter_i in range(len(clf.cluster_centers_)):
            sns.distplot(cars[cars.cluster == cluter_i][col], hist=True, ax=tmp[i], kde=False)
    except:
        print(col)<jupyter_output>/Users/jkchang/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "
{'fraction_non_alphanumeric': 0.1141757536882617, 'fraction_numerical': 0.022771007055805002, 'mean_word_length': 4.146039603960396, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '527036', 'license': 'Apache-2.0', 'n_tokens_mistral': 2142, 'n_tokens_neox': 1940, 'n_words': 626, 'path': 'Yudi TANG/Application/Auto-Mpg Data/Auto-Mpg Data.ipynb', 'repo_name': 'JKChang2015/Machine_Learning', 'size': '6236'}