<jupyter_start><jupyter_text>Predicting Car Prices Based on FeaturesThe goal of this project is to utilize the K Nearest Neighbors algorithm to see if we can predict car prices using data about car features vs. known car prices.  The data comes from the UC Irvine Machine Learning Repository and can be found at the following location:__[Automobile Data Set](https://archive.ics.uci.edu/ml/datasets/automobile)__ Feature list from the data source site1. symboling: -3, -2, -1, 0, 1, 2, 3.2. normalized-losses: continuous from 65 to 256.3. make:alfa-romero, audi, bmw, chevrolet, dodge, honda,isuzu, jaguar, mazda, mercedes-benz, mercury,mitsubishi, nissan, peugot, plymouth, porsche,renault, saab, subaru, toyota, volkswagen, volvo4. fuel-type: diesel, gas.5. aspiration: std, turbo.6. num-of-doors: four, two.7. body-style: hardtop, wagon, sedan, hatchback, convertible.8. drive-wheels: 4wd, fwd, rwd.9. engine-location: front, rear.10. wheel-base: continuous from 86.6 120.9.11. length: continuous from 141.1 to 208.1.12. width: continuous from 60.3 to 72.3.13. height: continuous from 47.8 to 59.8.14. curb-weight: continuous from 1488 to 4066.15. engine-type: dohc, dohcv, l, ohc, ohcf, ohcv, rotor.16. num-of-cylinders: eight, five, four, six, three, twelve, two.17. engine-size: continuous from 61 to 326.18. fuel-system: 1bbl, 2bbl, 4bbl, idi, mfi, mpfi, spdi, spfi.19. bore: continuous from 2.54 to 3.94.20. stroke: continuous from 2.07 to 4.17.21. compression-ratio: continuous from 7 to 23.22. horsepower: continuous from 48 to 288.23. peak-rpm: continuous from 4150 to 6600.24. city-mpg: continuous from 13 to 49.25. highway-mpg: continuous from 16 to 54.26. price: continuous from 5118 to 45400.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import Imputer, normalize
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import cross_val_score, KFold, train_test_split

%matplotlib inline

# Setting some default graph styles
sns.set_style("white", {'axes.edgecolor': '#b2b2b2'})
sns.set_context("notebook", font_scale=1.25)<jupyter_output><empty_output><jupyter_text>Data Analysis and Cleaning<jupyter_code>cars_original = pd.read_csv('imports-85.data')
cars_original.head()<jupyter_output><empty_output><jupyter_text>Looks like the data has no header row.  The column names pandas chose are actually data from the first entry.  Let's create a list of the real column names.<jupyter_code>columns = ['symboling', 'normalized_losses', 'make',
           'fuel-type', 'aspiration', 'num-of-doors',
           'body-style', 'drive-wheels', 'engine-location',
           'wheel-base', 'length', 'width',
           'height', 'curb-weight', 'engine-type',
           'num-of-cylinders', 'engine-size', 'fuel-system',
           'bore', 'stroke', 'compression-ratio',
           'horsepower', 'peak-rpm', 'city-mpg',
           'highway-mpg', 'price']
cars_original = pd.read_csv('imports-85.data', names=columns, header=None)
# We want to see all of the columns for examination when we display the dataframe
pd.set_option('display.max_columns', 26)
cars_original.head()<jupyter_output><empty_output><jupyter_text>The target prediction column is: **price**The following columns appear to be both numeric (or can be converted to numeric) and ordinal.  These will be our features.- normalized_losses- wheel-base- length- width- height- curb-weight- num-of-cylinders- engine-size- bore- stroke- compression-ratio- horsepower- peak-rpm- city-mpg- highway-mpg<jupyter_code>keep_columns = ['normalized_losses',
                'wheel-base',
                'length',
                'width',
                'height',
                'curb-weight',
                'num-of-cylinders',
                'engine-size',
                'bore',
                'stroke',
                'compression-ratio',
                'horsepower',
                'peak-rpm',
                'city-mpg',
                'highway-mpg',
                'price']
# Creating a new dataframe using only the columns we want to use as features
cars = cars_original[keep_columns]
cars.head()<jupyter_output><empty_output><jupyter_text>The **normalized_losses** column has question marks instead of null values.  Let's replace question marks with NaN throughout the data frame.<jupyter_code>cars = cars.replace('?', np.nan)<jupyter_output><empty_output><jupyter_text>Now lets see where things are regarding the data types for the columns:<jupyter_code>cars.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 205 entries, 0 to 204
Data columns (total 16 columns):
normalized_losses    164 non-null object
wheel-base           205 non-null float64
length               205 non-null float64
width                205 non-null float64
height               205 non-null float64
curb-weight          205 non-null int64
num-of-cylinders     205 non-null object
engine-size          205 non-null int64
bore                 201 non-null object
stroke               201 non-null object
compression-ratio    205 non-null float64
horsepower           203 non-null object
peak-rpm             203 non-null object
city-mpg             205 non-null int64
highway-mpg          205 non-null int64
price                201 non-null object
dtypes: float64(5), int64(4), object(7)
memory usage: 25.7+ KB<jupyter_text>The **num-of-cylinders** column contains words for numbers rather than numbers.  Let's see what words are used:<jupyter_code>cars['num-of-cylinders'].value_counts()<jupyter_output><empty_output><jupyter_text>Not that many distinct values. Let's whip up a quick dictionary for substitution.<jupyter_code>num_dict = {'four': 4, 'six': 6, 'five': 5, 'eight': 8, 'two': 2, 'twelve': 12, 'three': 3}
cars['num-of-cylinders'] = cars['num-of-cylinders'].apply(lambda x: num_dict.get(x, np.nan))
cars['num-of-cylinders'].head()<jupyter_output><empty_output><jupyter_text>All other columns apper to be number-ish.  Let's try a mass conversion of the entire data frame to float.<jupyter_code>cars = cars.astype('float')
cars.head()<jupyter_output><empty_output><jupyter_text>Now let's take a look at how many missing values we have.  A little bar chart gives a nice visual representation:<jupyter_code>cars.isnull().sum().apply(lambda x: x / cars.shape[0] * 100).plot(kind='barh', figsize=(10,6));
plt.title('Percentage of Missing Values by Feature')
sns.despine()<jupyter_output><empty_output><jupyter_text>The **normalized_losses** column indicates an average yearly insurance loss for that particular car. This should not factor into the initial price of the car, and it is missing quite a bit of data.  Let's drop it.<jupyter_code>cars.drop('normalized_losses', inplace=True, axis=1)<jupyter_output><empty_output><jupyter_text>For the remaining missing data, let's use a simple mean imputation to fill in the values.<jupyter_code>imp = Imputer(strategy='mean')
cars['peak-rpm'] = imp.fit_transform(cars[['peak-rpm']])
cars['horsepower'] = imp.fit_transform(cars[['horsepower']])
cars['stroke'] = imp.fit_transform(cars[['stroke']])
cars['bore'] = imp.fit_transform(cars[['bore']])<jupyter_output><empty_output><jupyter_text>There are still 4 rows missing price data.  Let's drop those rows from the dataframe.<jupyter_code>cars.dropna(inplace=True)
cars.isnull().sum()<jupyter_output><empty_output><jupyter_text>No more empty values.Now we need to normalize the data so that one particular column doesn't skew the prediction.  **price** should be excluded as it is the target of our prediction.<jupyter_code>for column in cars.columns: 
    if column != 'price':
        cars[column] = normalize(cars[[column]], axis=0)
cars.head()<jupyter_output><empty_output><jupyter_text>Univariate KNN Prediction Let's set up a function to split the data into training and testing sets, and utilize the __[KNeighborsRegressor](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html)__ model to predict the price of the cars in the test set.  The function will return the ***Root Mean Squared Error*** as an indicator of how successful the prediction was.  In this case the **RMSE** will indicate how many dollars off we were between the actual price and the predicted price.<jupyter_code>def knn_train_test(train_cols, target_col, df, neighbors=None):
    '''
    Splits a dataframe into training and testing data sets and uses KNN 
    for prediction.
    
    '''
    
    X = df[train_cols]
    y = df[target_col]
    
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=101)
    
    if neighbors:
        knn = KNeighborsRegressor(n_neighbors=neighbors)
    else:
        knn = KNeighborsRegressor()
        
    knn.fit(X_train, y_train)
    
    prediction = knn.predict(X_test)
    
    rmse = mean_squared_error(y_test, prediction) ** .5
    
    return rmse, X_test, y_test, prediction
# Getting a list of columns to cycle through the model excluding 'price'

test_cols = list(cars.columns)
test_cols.remove('price')
# Creating an empty list to hold the RMSE values returned by our function

rmse_list = []
# Running each feature column through the model function

for col in test_cols:
    rmse, _, _, _ = knn_train_test([col], 'price', cars)
    rmse_list.append(rmse)
# For easier graphing and data manipulation, creating a DataFrame out of the results

univ_rmse = pd.DataFrame({'Feature': test_cols, 'RMSE': rmse_list})
univ_rmse.sort_values('RMSE', inplace=True)
univ_rmse
plt.figure(figsize=(10,5))
sns.barplot(x='RMSE', 
            y='Feature', 
            data=univ_rmse, 
            palette=sns.color_palette("PuBu_r", univ_rmse.shape[0])
           ).set_title('Univariate RMSE by Feature');
sns.despine()<jupyter_output><empty_output><jupyter_text>This charts shows which individual features performed best when used alone.  Next let's try some different values for the ***n_neighbors*** hyperparameter.  Trying the following values;1, 3, 5, 7, 9<jupyter_code>neighbors = list(range(1,11,2))
neighbors_test = [test_cols]
# For each individual feature, run it through KNN using the 5 values for n_neighbors

for val in neighbors:
    
    rmse_list = []
    
    for col in test_cols:
        rmse, _, _, _ = knn_train_test([col], 'price', cars, val)
        rmse_list.append(rmse)  
        
    neighbors_test.append(rmse_list)
        
#Putting the results into a dataframe for eaiser analysis

neighbors_df = pd.DataFrame({'Feature': neighbors_test[0],
                             'n1': neighbors_test[1],
                             'n3': neighbors_test[2],
                             'n5': neighbors_test[3],
                             'n7': neighbors_test[4],
                             'n9': neighbors_test[5]})

neighbors_df.head()
# Using the dataframe mean function to generate a column containing 
# the mean RMSE for each feature across all 5 n_neighbors values

neighbors_df.set_index('Feature', inplace=True)
neighbors_df['Avg. RMSE'] = neighbors_df.mean(axis=1)
neighbors_df.reset_index(inplace=True)
neighbors_df.sort_values('Avg. RMSE', inplace=True)

neighbors_df.head()
plt.figure(figsize=(10,5))
sns.barplot(x='Avg. RMSE',
            y='Feature',
            data=neighbors_df, 
            palette=sns.color_palette("YlGn_r", univ_rmse.shape[0])
           ).set_title('Univariate RMSE Averaged Across n_neighbors Values');
sns.despine()<jupyter_output><empty_output><jupyter_text>Multivariate KNN Prediction Now let's use the 5 best features in different combinations to see how it affects our prediction.  For each combination, we will try n_neighbors values of 1 - 25 as well to find the optimum k value for KNN.<jupyter_code>top_5_features = neighbors_df['Feature'][0:5]
best_multi = {'Top 2': [],
              'Top 3': [],
              'Top 4': [],
              'Top 5': []
             }
# Looping over combinations of the top 2 - top 5 features across 1-25 neighbors

for k in range(1,26):
    
    for f in range(2,6):
        
        rmse, _, _, _ = knn_train_test(top_5_features[0:f], 'price', cars, k)
        best_multi['Top ' + str(f)].append(rmse)
multi_rmse = pd.DataFrame(best_multi)
multi_rmse['k Nearest Neighbors'] = list(range(1,26))
plt.figure(figsize=(10,6))

for key, value in best_multi.items():
    
    plt.plot(multi_rmse['k Nearest Neighbors'], multi_rmse[key], label=key)
    
plt.legend()
plt.title('Multivariate RMSE Across 1-25 Neighbors')
plt.xlabel('k Nearest Neighbors')
plt.ylabel('RMSE')
sns.despine()
# Grabbing the minimum RMSE for each feature combination and its associated k value

multi_rmse_min = pd.DataFrame()

for i in range(2,6):
    
    min_index = multi_rmse['Top ' + str(i)].idxmin
    
    multi_rmse_min = multi_rmse_min.append({'Features': 'Top ' + str(i),
                                            'RMSE': multi_rmse.iloc[min_index]['Top ' + str(i)],
                                            'k Nearest Neighbors': multi_rmse.iloc[min_index]['k Nearest Neighbors']},
                                             ignore_index=True
                                          )
multi_rmse_min.sort_values('RMSE')<jupyter_output><empty_output><jupyter_text>Based on the results, it seems that the best prediction occurs when we use all 5 top Features with a k neighbors setting of 1.  To reiterate, the top 5 features were:* engine-size* horsepower* highway-mpg* city-mpg* widthWith those settings we can theoretically predict the car price within +/- $2376.24 of the actual price. Let's try K-Fold Cross-Validation to see if it improves our prediction<jupyter_code>def knn_fold_train_test(train_cols, target_col, df, folds, k_val=None):
    '''
    Uses K-Fold Cross-Validation along with KNN to predict values
    '''

    kf = KFold(folds, shuffle=True, random_state=1)
 
    if k_val:
        model = KNeighborsRegressor(n_neighbors=k_val)
    else:
        model = KNeighborsRegressor()
   
    mses = cross_val_score(model, df[train_cols], df[target_col], scoring="neg_mean_squared_error", cv=kf)
    rmses = np.sqrt(np.absolute(mses))
 
    return rmses
# Trying with 5 Folds

knn_fold_train_test(top_5_features, 'price', cars, 5, 1).mean()
# Trying with 10 Folds

knn_fold_train_test(top_5_features, 'price', cars, 10, 1).mean()<jupyter_output><empty_output><jupyter_text>Using K-Fold Cross-Validation does not seem to improve our prediction.    ResultsNow that we've tuned our parameters and feature selection,  let's take a look at some results based on the original data.  We'll compare the results from the test set vs. the known test values to get an idea of how accurate the model is.<jupyter_code># Running KNN with the features and parameters discovered above

(rmse, X_test, y_test, prediction) = knn_train_test(top_5_features, 'price', cars, 1)
# Joining the results with original data for a more human readable output

results = cars_original.iloc[X_test.index][top_5_features]
results['Actual Price'] = y_test
results['Predicted Price'] = prediction
results['Difference'] = results['Actual Price'] - results['Predicted Price']

# Taking 20 random samples from the results for viewing

results.sample(20)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10746776621506643, 'fraction_numerical': 0.023430852804502914, 'mean_word_length': 3.7160493827160495, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 77, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '496242', 'license': 'MIT', 'n_tokens_mistral': 5275, 'n_tokens_neox': 4815, 'n_words': 1448, 'path': 'car_prices.ipynb', 'repo_name': 'Adam-Klaum/car_prices', 'size': '15279'}