<jupyter_start><jupyter_text>Used Cars Database ![](https://d32c3oe4bky4k6.cloudfront.net/-/media/usdirect/images/insurance/classic-car-insurance/car_billboard-image.ashx?modified=20191106220758)  Imports<jupyter_code>import os,re,zipfile
import pandas as pd
import numpy as np
from types import SimpleNamespace
from matplotlib import pyplot as plt
plt.style.use('dark_background')
plt.style.use('seaborn')
base_size = 10
sizes=SimpleNamespace(**dict(small=(1*base_size,1*base_size),medium=(2*base_size,2*base_size),large=(3*base_size,3*base_size)))<jupyter_output><empty_output><jupyter_text>Download and Unzip<jupyter_code>%%time
zip_name = 'used-cars-database.zip'
if not os.path.exists(zip_name):
  os.environ['KAGGLE_USERNAME'] = "" # username from the json file
  os.environ['KAGGLE_KEY'] = "" # key from the json file
  !kaggle datasets download orgesleka/used-cars-database
with zipfile.ZipFile(zip_name, 'r') as zip_ref:
    zip_ref.extractall(zip_name.split('.')[0])
os.listdir(zip_name.split('.')[0])<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code>auto = pd.read_csv(zip_name.split('.')[0]+'/autos.csv',encoding='ISO-8859-1')
auto
missing_values_cols = auto.isnull().sum()
missing_values_cols = missing_values_cols[missing_values_cols > 0]
missing_values_cols = missing_values_cols.index.tolist()
missing_values_cols
# in some cases an approach like this might have worked to fill the empty values
# auto.groupby(['category', 'name'])['value'].transform(lambda x: x.fillna(x.mean()))<jupyter_output><empty_output><jupyter_text>Quick EDA<jupyter_code>fig,ax = plt.subplots(2,4,figsize=(30,15))
auto.vehicleType.value_counts().plot.barh(ax=ax.ravel()[0],title='Number of vehicles by type')
auto.fuelType.value_counts().plot.barh(ax=ax.ravel()[1],title='Number of vehicles by fuel type')
auto.yearOfRegistration.value_counts()[:20].plot.barh(ax=ax.ravel()[2],title='Number of vehicles by year of registration (first 20)')
auto.groupby('vehicleType')[['price']].mean().plot.barh(ax=ax.ravel()[3],title='Mean price of vehicles by type')
auto.groupby('fuelType')[['price']].mean().plot.barh(ax=ax.ravel()[4],title='Mean price of vehicles fuel type')
auto.groupby('vehicleType')[['kilometer']].mean().plot.barh(ax=ax.ravel()[5],title='Mean kilometer of vehicles by type')
auto.groupby('fuelType')[['kilometer']].mean().plot.barh(ax=ax.ravel()[6],title='Mean kilometer of vehicles fuel type')
auto.gearbox.value_counts().plot.barh(ax=ax.ravel()[7],title='Number of vehicles by gearbox')
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Kilometers and Prices by Brand and Model<jupyter_code>tmp = auto.groupby(['brand','model']).agg({
    'kilometer': 'mean',
    'price': 'mean'
})
t = list(set([x[0] for x in tmp.index]))
fig, ax = plt.subplots(int(len(t) / 4)+1, 4,figsize=(50,50))

for i,brand in enumerate(t):
  try:
    tmp.loc[brand].plot.barh(ax=ax.ravel()[i], title=f'Mean (kilometer, price) for {brand.replace("_"," ").capitalize()} car models')
  except:
    print(f'error for {brand}')
    continue
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>After the quick overview let's build a model to help us set the price `XGBoost` and `LabelEncoder` should do the job for us<jupyter_code>from sklearn.model_selection import train_test_split
import xgboost as xgb
from sklearn.preprocessing import LabelEncoder
from sklearn import metrics
from collections import defaultdict
d = defaultdict(LabelEncoder)
d<jupyter_output><empty_output><jupyter_text>`x_cols` are our `features` and `target` is what we look for to `predict`<jupyter_code>x_cols=['brand','model','kilometer','yearOfRegistration','monthOfRegistration','fuelType','vehicleType','notRepairedDamage','powerPS','gearbox']
target = 'price'<jupyter_output><empty_output><jupyter_text>* Copy the dataframe without the missing values* Define the columns that need encoding* Apply the encoding<jupyter_code>auto_copy = auto.dropna().copy()
enc_cols = list(set(x_cols) - set(['powerPS','yearOfRegistration','monthOfRegistration','kilometer']))
auto_copy[enc_cols] = auto_copy[enc_cols].apply(lambda x: d[x.name].fit_transform(x))<jupyter_output><empty_output><jupyter_text>After we dropped the missing values we still have 70% of the data<jupyter_code>len(auto_copy) / len(auto)<jupyter_output><empty_output><jupyter_text>Now, let's split our data into `train`/`test`<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(auto_copy[x_cols], auto_copy[target], test_size=0.33, random_state=42)<jupyter_output><empty_output><jupyter_text>Further on we can use the `XGBRegressor` with a `gamma` distribution and train it on the `train` data<jupyter_code>%%time
xg = xgb.XGBRegressor(objective ='reg:gamma', n_estimators = 1000).fit(X_train,y_train)<jupyter_output>CPU times: user 1min 23s, sys: 67.9 ms, total: 1min 23s
Wall time: 1min 23s<jupyter_text>Make a `dataframe` with `test` target vs `predicted` test target<jupyter_code>preds = pd.DataFrame({
    'true': y_test,
    'predicted': xg.predict(X_test)
})<jupyter_output><empty_output><jupyter_text>Use MAE for our metric<jupyter_code>metrics.mean_absolute_error(preds.true,preds.predicted)<jupyter_output><empty_output><jupyter_text>Let's have a look into our `Feature Importance`<jupyter_code>plt.figure(figsize=sizes.small)
xgb.plot_importance(xg)
plt.show()<jupyter_output><empty_output><jupyter_text>And here we have a copy of our `test` dataframe with `decoded` columns, but also with `original` and `predicted` price<jupyter_code>xt = X_test.copy()
xt['original_price'] = y_test
xt['predicted_price'] = xg.predict(X_test)
xt[enc_cols] = xt[enc_cols].apply(lambda x: d[x.name].inverse_transform(x))
xt[:30]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15097970608817354, 'fraction_numerical': 0.015395381385584325, 'mean_word_length': 8.560200668896321, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 61, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '225703', 'license': 'FTL', 'n_tokens_mistral': 2141, 'n_tokens_neox': 2041, 'n_words': 384, 'path': 'notebooks/2020-10-09-UsedCarsDatabase.ipynb', 'repo_name': 'cristianexer/puzzles-and-datasets', 'size': '5716'}