<jupyter_start><jupyter_text>**🚜 Predicting the sales price of bulldozers using machine learning**In this notebook, we're going to go through an example machine learning project with the goal of predicting the sale price of bulldozers.**1. Problem defition**> How well can we predict the future sale price of a bulldozer, given its characteristics and previous examples of how much similar bulldozers have been sold for?**2. Data**The data is downloaded from the Kaggle Bluebook for Bulldozers competition: https://www.kaggle.com/c/bluebook-for-bulldozers/data<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Machine learning model
from sklearn.ensemble import RandomForestRegressor

# Evaluation libraries
from sklearn.metrics import mean_squared_log_error, mean_absolute_error, r2_score

# Hyperparameter tuning
from sklearn.model_selection import RandomizedSearchCV
# Import data with parse dates
df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/data/bulldozer-dataset/TrainAndValid.csv', low_memory=False,parse_dates=['saledate'])
df.head()
df.isna().sum()
df.dtypes
fig, ax = plt.subplots()
ax.scatter(df.saledate[:1000], df.SalePrice[:1000]);<jupyter_output><empty_output><jupyter_text>**Sort DataFrame by saledate**<jupyter_code>df.sort_values(by=['saledate'], inplace=True, ascending=True)
df.saledate.head(10)
# Make a copy  of dataframe
df_temp = df.copy()<jupyter_output><empty_output><jupyter_text>Adding datetime parameters for `saledate` column<jupyter_code>df_temp['saleYear'] = df_temp.saledate.dt.year
df_temp['saleMonth'] = df_temp.saledate.dt.month
df_temp['saleDay'] = df_temp.saledate.dt.day
df_temp['saleDayOfWeek'] = df_temp.saledate.dt.dayofweek
df_temp['saleDayOfYear'] = df_temp.saledate.dt.dayofyear
df_temp.drop('saledate', axis =1, inplace=True)
df_temp.head(20).T
# finding all columns which contain Strings
for label, content in df_temp.items():
  if pd.api.types.is_string_dtype(content):
    print(label)
for label, content in df_temp.items():
  if pd.api.types.is_string_dtype(content):
    df_temp[label] = content.astype('category').cat.as_ordered()
df_temp.state.cat.categories
df_temp.state.cat.codes
df_temp.head().T<jupyter_output><empty_output><jupyter_text>Fill missing values<jupyter_code>for label, content in df_temp.items():
  if pd.api.types.is_numeric_dtype(content):
    print(label)
# Check for which numeric columns have null values
for label, content in df_temp.items():
  if pd.api.types.is_numeric_dtype(content):
    if pd.isnull(content).sum():
      print(label)
# Fill numeric rows with median
for label, content in df_temp.items():
  if pd.api.types.is_numeric_dtype(content):
    if pd.isnull(content).sum():
      df_temp[label+'_is_missing'] = pd.isnull(content)
      df_temp[label] = content.fillna(content.median())
# Check for which numeric columns have null values
for label, content in df_temp.items():
  if pd.api.types.is_numeric_dtype(content):
    if pd.isnull(content).sum():
      print(label)
# Check to see how many examples were missing
df_temp.auctioneerID_is_missing.value_counts()
# Filling and turning categorical variables into numbers
for label, content in df_temp.items():
  if not pd.api.types.is_numeric_dtype(content):
    if pd.isnull(content).sum():
      print(label)
# In place of missing data it adds -1 while converting to categories
pd.Categorical(df_temp['UsageBand']).codes
# Turn categorical variables into numbers and fill missing
for label, content in df_temp.items():
  if not pd.api.types.is_numeric_dtype(content):
    df_temp[label+'_is_missing'] = pd.isnull(content)
    df_temp[label] = pd.Categorical(content).codes + 1 
# Filling and turning categorical variables into numbers
for label, content in df_temp.items():
  if not pd.api.types.is_numeric_dtype(content):
    if pd.isnull(content).sum():
      print(label)
df_temp.isna().sum()<jupyter_output><empty_output><jupyter_text>Now that all of data is numeric as well as our dataframe has no missing values, we should be able to build a machine learning model.<jupyter_code>df_temp.saleYear.value_counts()
 # Split data into training and validation
 df_val = df_temp[df_temp.saleYear == 2012]
 df_train = df_temp[df_temp.saleYear != 2012]

 len(df_val), len(df_train)
# Split data into X & y
X_train, y_train = df_train.drop('SalePrice', axis=1), df_train.SalePrice
X_valid, y_valid = df_val.drop('SalePrice', axis=1), df_val.SalePrice

X_train.shape, y_train.shape, X_valid.shape, y_valid.shape
# Create evaluation function (the competition uses RMSLE)

def rmsle(y_test, y_preds):
  return np.sqrt(mean_squared_log_error(y_test, y_preds))

# Create function to evaluate model on a few different levels
def show_score(model):
  train_preds = model.predict(X_train)
  valid_preds = model.predict(X_valid)
  score = {'Training MAE': mean_absolute_error(y_train, train_preds),
           'Valid MAE': mean_absolute_error(y_valid, valid_preds),
           'Training RMSLE': rmsle(y_train, train_preds),
           'Valid RMSLE': rmsle(y_valid, valid_preds),
           'Training R2': r2_score(y_train, train_preds),
           'Valid R2': r2_score(y_valid, y_valid)}
  return score
%%time
model = RandomForestRegressor(n_jobs=-1, 
                              random_state=42, 
                              max_samples= 10000)

model.fit(X_train, y_train)

show_score(model)<jupyter_output>CPU times: user 46.7 s, sys: 272 ms, total: 47 s
Wall time: 32.2 s<jupyter_text>**Hyerparameter tuning with RandomizedSearchCV**<jupyter_code>%%time

rf_grid = {
    'n_estimators': np.arange(10, 100, 10),
    'max_depth': [None, 3, 5, 10],
    'min_samples_split': np.arange(2,20,2),
    'min_samples_leaf': np.arange(1,20,2),
    'max_features': [0.5, 1, 'sqrt', 'auto'],
    'max_samples': [10000],
}

# Instantiate Model
rs_model = RandomizedSearchCV(RandomForestRegressor(n_jobs=-1, random_state=42),
                              param_distributions=rf_grid,
                              n_iter = 2,
                              cv=5,
                              verbose=True)

# Fit the model
rs_model.fit(X_train, y_train)
show_score(rs_model)<jupyter_output><empty_output><jupyter_text>**Make predictions on test data**<jupyter_code>df_test = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/data/bulldozer-dataset/Test.csv', 
                      low_memory=False, 
                      parse_dates=['saledate'])

df_test.head()<jupyter_output><empty_output><jupyter_text>**Preprocessing the data (getting the test dataset in the same format as our training dataset)**<jupyter_code>def preprocess_data(df):

  df["saleYear"] = df.saledate.dt.year
  df["saleMonth"] = df.saledate.dt.month
  df["saleDay"] = df.saledate.dt.day
  df["saleDayOfWeek"] = df.saledate.dt.dayofweek
  df["saleDayOfYear"] = df.saledate.dt.dayofyear
    
  df.drop("saledate", axis=1, inplace=True)

  # Fill the numeric rows with median
  for label, content in df.items():
      if pd.api.types.is_numeric_dtype(content):
          if pd.isnull(content).sum():
              # Add a binary column which tells us if the data was missing or not
              df[label+"_is_missing"] = pd.isnull(content)
              # Fill missing numeric values with median
              df[label] = content.fillna(content.median())
      # Filled categorical missing data and turn categories into numbers
      if not pd.api.types.is_numeric_dtype(content):
          df[label+"_is_missing"] = pd.isnull(content)
          # We add +1 to the category code because pandas encodes missing categories as -1
          df[label] = pd.Categorical(content).codes+1

  return df
df_tests = preprocess_data(df_test)
df_tests.head()
# We don't have `auctioneerID_is_missing` column because there were no missing value in that column so we need to manually create and update value wit true
df_tests['auctioneerID_is_missing'] = False
df_tests.shape
# Make predictions on the test data
test_preds = rs_model.predict(df_test)<jupyter_output>/usr/local/lib/python3.7/dist-packages/sklearn/base.py:493: FutureWarning: The feature names should match those that were passed during fit. Starting version 1.2, an error will be raised.
Feature names must be in the same order as they were in fit.

  warnings.warn(message, FutureWarning)<jupyter_text>We've made some predictions but they're not in the same format Kaggle is asking for: https://www.kaggle.com/c/bluebook-for-bulldozers/overview/evaluation<jupyter_code># Format predictions into the same format Kaggle is after
df_preds = pd.DataFrame()
df_preds['SalesID'] = df_tests['SalesID']
df_preds['SalesPrice'] = test_preds
df_preds.head()
# Export for submission
df_preds.to_csv('/content/drive/MyDrive/Colab Notebooks/data/bulldozer-dataset/test_predictions.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10893665158371041, 'fraction_numerical': 0.010633484162895928, 'mean_word_length': 4.508411214953271, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '527305', 'license': 'MIT', 'n_tokens_mistral': 3049, 'n_tokens_neox': 2910, 'n_words': 723, 'path': 'bulldozer_price_detector.ipynb', 'repo_name': 'anmolmishra99/Bulldozer-Price-Detector', 'size': '8840'}