<jupyter_start><jupyter_text>IV. Data VisualizationsThe aim of this notebook is to show some visualization of the existing data set.For this purpose, I shall do some very basic data preparation, and then proceed to explore and visualize data.Note: **Data preparation undertaken in this notebook differs from data preparation undertaken on ¨Data Preparation¨ notebook**. The reason is that data prepartion serves to different purposes, in this notebook data is prepared for visualizations. And on the other one, for Machine Learning algorithm creation<jupyter_code># import libraries
import pandas as pd
from sklearn import preprocessing
import sklearn.model_selection as ms
from sklearn import linear_model
import sklearn.metrics as sklm
import numpy as np
import numpy.random as nr
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats as ss
import math

import warnings
warnings.filterwarnings('ignore')

%matplotlib inline
%matplotlib inline<jupyter_output><empty_output><jupyter_text>1. Import and merge data sets<jupyter_code>#import first data set
autos=pd.read_csv('autos.csv',sep=',')
autos.shape
#View data
autos.head(2)
#import second data set
makes=pd.read_csv('makes.csv',sep=',')
makes.shape
# Merge the two datasets
df= pd.concat([autos,makes],axis=1)
df.shape
df.head()
#column list
df.columns<jupyter_output><empty_output><jupyter_text>To avoid problems, I changed the names replacing sign '-' by '_' in column names<jupyter_code>old_names = ['symboling', 'normalized-losses', 'make-id', 'fuel-type', 'aspiration',
       'num-of-doors', 'body-style', 'drive-wheels', 'engine-location',
       'wheel-base', 'length', 'width', 'height', 'curb-weight', 'engine-type',
       'num-of-cylinders', 'engine-size', 'fuel-system', 'bore', 'stroke',
       'compression-ratio', 'horsepower', 'peak-rpm', 'city-mpg',
       'highway-mpg', 'price', 'make-id', 'make'] 
new_names = ['symboling', 'normalized_losses', 'make_id', 'fuel_type', 'aspiration',
       'num_of_doors', 'body_style', 'drive_wheels', 'engine_location',
       'wheel_base', 'length', 'width', 'height', 'curb_weight', 'engine_type',
       'num_of_cylinders', 'engine_size', 'fuel_system', 'bore', 'stroke',
       'compression_ratio', 'horsepower', 'peak_rpm', 'city_mpg',
       'highway_mpg', 'price', 'make_id', 'make']
df.rename(columns=dict(zip(old_names, new_names)), inplace=True)
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>2. Drop duplicate rows and columns<jupyter_code>#drop duplicate rows (using subset, I drop raws where values of columns mentioned match)
df.drop_duplicates(subset=['symboling', 'normalized_losses', 'make_id', 'fuel_type', 'aspiration',
       'num_of_doors', 'body_style', 'drive_wheels', 'engine_location',
       'wheel_base', 'length', 'width', 'height', 'curb_weight', 'engine_type',
       'num_of_cylinders', 'engine_size', 'fuel_system', 'bore', 'stroke',
       'compression_ratio', 'horsepower', 'peak_rpm', 'city_mpg',
       'highway_mpg', 'price', 'make_id', 'make'], inplace = True)
#see shape after duplicate row removal
df.shape<jupyter_output><empty_output><jupyter_text>11 rows have been deleted<jupyter_code># remove duplicate columns
_, i = np.unique(df.columns, return_index=True)
df=df.iloc[:, i]
df.shape<jupyter_output><empty_output><jupyter_text>1 column has been deleted  3. Handle Missing values<jupyter_code>df.isnull().mean()
# let's no determine how many variables we have with missing information

vars_with_na = [var for var in df.columns if df[var].isnull().sum()>0]
print('Total variables that contain missing information: ', len(vars_with_na))
# Drop 'make'column since it has too many nulls
df=df.drop('make', axis=1)<jupyter_output><empty_output><jupyter_text>To keep it simple I will:    a) replace nulls from numerical variables by their mean    b) replace nulls from categorical variables by the most frequent category<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>Handle numerical:    - normalized_losses, price, stroke, horsepower, peak_rpm<jupyter_code>for var in ['bore','stroke','price','peak_rpm',
           'horsepower','normalized_losses']:
    df[var].fillna(df[var].median(), inplace=True)<jupyter_output><empty_output><jupyter_text>Handle categorical: num_of_doors<jupyter_code># find most frequent category in 'num-of-doors'
df.groupby(['num_of_doors'])['num_of_doors'].count().sort_values(ascending=False).index[0]
# let's create a variable to replace NA with the most frequent label
# both in train and test set

def impute_na(variable):
    most_frequent_category = df.groupby([variable])[variable].count().sort_values(ascending=False).index[0]
    df[variable].fillna(most_frequent_category, inplace=True)

# and let's replace the NA
for variable in ['num_of_doors',]:
    impute_na(variable)<jupyter_output><empty_output><jupyter_text>Let's see if I did well and there are no more nulls<jupyter_code># check absence of null values
for var in df.columns:
    if df[var].isnull().sum()>0:
        print(var, df[var].isnull().sum())<jupyter_output><empty_output><jupyter_text>Yeah! No more NA!  1.  General Data Exploration<jupyter_code># Let's observe the existing correlations between dataset
plt.figure(figsize=(8,8)) 

sns.heatmap(df.corr())<jupyter_output><empty_output><jupyter_text>This chart shows some very interesting correlations, such as :- price and curve_weight or engine_size- city_mpg and highway_mpg- length and wheel_base- curb_weight and width and so on.I will need to take care of this existing correlations when handling the feature selection in order to prevent overfitting.   2. Label or Target Exploration<jupyter_code>## Descriptive statistics for label (in this case it will be attractiveness)
df.price.describe()<jupyter_output><empty_output><jupyter_text>From the describe statistics, it looks like data is skewed. We'll check the Histogram to confirm<jupyter_code>plt.hist(df.price,15)<jupyter_output><empty_output><jupyter_text>Label is highly skewed. In fact, we could try to modify this column by using its log function. However, since it's the Target and not just a variable, undertaking this change could probably help me get a model to predict better the logarithm of the price, but not the price.<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>2.1 Exploring relationships between label and numerical features<jupyter_code># find categorical variables
categorical = [var for var in df.columns if df[var].dtype=='O']
print('There are {} categorical variables'.format(len(categorical)))
categorical
# Find temporal varibles
# make a list of the numerical variables first
numerical = [var for var in df.columns if df[var].dtype!='O']

# list of variables that contain year information
year_vars = [var for var in numerical if 'Yr' in var or 'Year' in var or 'Day'in var or 'Month'in var or 'Time'in var]
print('There are {} temporal variables'.format(len(year_vars)))


year_vars
# let's visualise the values of the discrete variables
discrete = []

for var in numerical:
    if len(df[var].unique())<20 and var not in year_vars:
        print(var, ' values: ', df[var].unique())
        discrete.append(var)
print()
print('There are {} discrete variables'.format(len(discrete)))
discrete
### Find Continuous
# let's remember to skip the Id variable and the target variable SalePrice, which are both also numerical

numerical = [var for var in numerical if var not in discrete and var not in ['Id', 'SalePrice'] and var not in year_vars]
print('There are {} numerical and continuous variables'.format(len(numerical)))
numerical<jupyter_output>There are 16 numerical and continuous variables<jupyter_text>From the Algorithm selection notebook, I know the most important variables are:- num_of_cylinders (cat)- curb_weight- normalized_losses- stroke- horsepower- bore- num_of_doors (cat)<jupyter_code>## Create scatter-plot matrix of most important correlations with the label


num_cols1 = ["curb_weight", "normalized_losses","stroke", "horsepower","price"] 
sns.pairplot(df[num_cols1], diag_kind="kde", size=3).map_upper(sns.kdeplot, cmap="Blues_d")<jupyter_output><empty_output><jupyter_text>* Looking at the last row, we can see com clear linear correlations with the LABEL, such as curb_weight and horsepower.* On the upper digonal, we can see some different concentration, probably showing a tendence hidden in data.* Also, we can see the presence of dots far away from the main stream of dots, which are the outliers.  2.2 Exploring relationships between label and categorical features<jupyter_code>def plot_box(df, cols, col_y='price'):
    for col in cols:
        sns.set_style("whitegrid")
        sns.boxplot(col, col_y, data=df)
        plt.xlabel(col)
        plt.ylabel(col_y)
        plt.show()
        
cat_cols = ['num_of_cylinders','num_of_doors','fuel_system','fuel_type']
plot_box(df, cat_cols)<jupyter_output><empty_output><jupyter_text>When observing the relationship between the label and the categorical variables, we can see that* Except for number_of_doors, we can see in all other categorical variables median of price depending on the category.* Biggest price difference between categories can be observed on the numeber of cylinders. Price median swifts depending between categories      2.3 Using Aesthetics to explore relationships between label and numerical/categorical features<jupyter_code>def plot_scatter_shape(df, cols, shape_col = 'fuel_type', col_y = 'price', alpha = 0.3):
    shapes = ['o', '+', 's', 'x', '^'] # pick distinctive shapes
    unique_cats = df[shape_col].unique()
    for col in cols: # loop over the columns to plot
        sns.set_style("whitegrid")
        for i, cat in enumerate(unique_cats): # loop over the unique categories
            temp = df[df[shape_col] == cat]
            sns.regplot(col, col_y, data=temp, marker = shapes[i], label = cat,
                        scatter_kws={"alpha":alpha}, fit_reg = False, color = 'red')
        plt.title('Scatter plot of ' + col_y + ' vs. ' + col) # Give the plot a main title
        plt.xlabel(col) # Set text for the x axis
        plt.ylabel(col_y)# Set text for y axis
        plt.legend()
        plt.show()
            
num_cols = ["curb_weight", "normalized_losses","stroke", "horsepower"]
plot_scatter_shape(df, num_cols) 
def plot_scatter_shape(df, cols, shape_col = 'aspiration', col_y = 'price', alpha = 0.3):
    shapes = ['o', '+', 's', 'x', '^'] # pick distinctive shapes
    unique_cats = df[shape_col].unique()
    for col in cols: # loop over the columns to plot
        sns.set_style("whitegrid")
        for i, cat in enumerate(unique_cats): # loop over the unique categories
            temp = df[df[shape_col] == cat]
            sns.regplot(col, col_y, data=temp, marker = shapes[i], label = cat,
                        scatter_kws={"alpha":alpha}, fit_reg = False, color = 'blue')
        plt.title('Scatter plot of ' + col_y + ' vs. ' + col) # Give the plot a main title
        plt.xlabel(col) # Set text for the x axis
        plt.ylabel(col_y)# Set text for y axis
        plt.legend()
        plt.show()
            
num_cols = ["curb_weight", "normalized_losses","stroke", "horsepower"]
plot_scatter_shape(df, num_cols)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11398135818908123, 'fraction_numerical': 0.003195739014647137, 'mean_word_length': 5.043991416309013, 'pattern_counts': {'":': 2, '<': 73, '<?xml version=': 0, '>': 74, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '401766', 'license': 'MIT', 'n_tokens_mistral': 3588, 'n_tokens_neox': 3472, 'n_words': 1107, 'path': '4. Data Visualizations .ipynb', 'repo_name': 'titoniubo/car_pricing', 'size': '11265'}