<jupyter_start><jupyter_text>Blue Book for Bulldozers  Problem definition- Predict the auction sale price for a piece of heavy equipment to create a "blue book" for bulldozers.. DataThe data is downloaded from the Kaggle Bluebook for Bulldozers competition: https://www.kaggle.com/c/bluebook-for-bulldozers/dataThere are 3 main datasets:* Train.csv is the training set, which contains data through the end of 2011.* Valid.csv is the validation set, which contains data from January 1, 2012 - April 30, 2012 You make predictions on this set throughout the majority of the competition. Your score on this set is used to create the public leaderboard.* Test.csv is the test set, which won't be released until the last week of the competition. It contains data from May 1, 2012 - November 2012. Your score on the test set determines your final rank for the competition. EvaluationThe evaluation metric for this competition is the RMSLE (root mean squared log error) between the actual and predicted auction prices.For more on the evaluation of this project check: https://www.kaggle.com/c/bluebook-for-bulldozers/overview/evaluationNote: The goal for most regression evaluation metrics is to minimize the error. For example, our goal for this project will be to build a machine learning model which minimises RMSLE. FeaturesKaggle provides a data dictionary detailing all of the features of the dataset. You can view this data dictionary on Google Sheets: https://docs.google.com/spreadsheets/d/18ly-bLR8sbDJLITkWG7ozKm8l3RyieQ2Fpgix-beSYI/edit?usp=sharing Loading Data<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import sklearn
df = pd.read_csv("TrainAndValid.csv",
                 low_memory=False)
df.info()
df.isna().sum()
df.columns
fig, ax = plt.subplots()
ax.scatter(df["saledate"][:1000], df["SalePrice"][:1000])
df.saledate[:1000]
df.saledate.dtype
df.SalePrice.plot.hist()<jupyter_output><empty_output><jupyter_text>Parsing dates- When we work with time series data, we want to enrich the time & date component as much as possible.- We can do that by telling pandas which of our columns has dates in it using the parse_dates parameter.<jupyter_code># Import data again but this time parse dates
df = pd.read_csv("TrainAndValid.csv",
                 low_memory=False,
                 parse_dates=["saledate"])
df.saledate.dtype
df.saledate[:1000]
fig, ax = plt.subplots()
ax.scatter(df["saledate"][:1000], df["SalePrice"][:1000])
df.head()
df.head().T
df.saledate.head(20)<jupyter_output><empty_output><jupyter_text>Sort DataFrame by saledate- When working with time series data, it's a good idea to sort it by date.<jupyter_code>df.sort_values(by=["saledate"], inplace=True, ascending=True)
df.saledate.head(20)<jupyter_output><empty_output><jupyter_text>Make a copy of the original DataFrame- We make a copy of the original dataframe so when we manipulate the copy, we've still got our original data.<jupyter_code>df_tmp = df.copy()<jupyter_output><empty_output><jupyter_text>Add datetime parameters for saledate column<jupyter_code>df_tmp["saleYear"] = df_tmp.saledate.dt.year
df_tmp["saleMonth"] = df_tmp.saledate.dt.month
df_tmp["saleDay"] = df_tmp.saledate.dt.day
df_tmp["saleDayOfWeek"] = df_tmp.saledate.dt.dayofweek
df_tmp["saleDayOfYear"] = df_tmp.saledate.dt.dayofyear
df_tmp.head().T
df_tmp.drop("saledate", axis=1, inplace=True)
df_tmp.state.value_counts()
df_tmp.head()
len(df_tmp)<jupyter_output><empty_output><jupyter_text>ModellingNow start to do some model-driven EDA.<jupyter_code>from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(n_jobs=-1,
                              random_state=42)
model.fit(df_tmp.drop("SalePrice", axis=1), df_tmp["SalePrice"])
df_tmp.info()
df_tmp["UsageBand"].dtype
df_tmp.isna().sum()<jupyter_output><empty_output><jupyter_text>Convert string to categories- One way we can turn all of our data into numbers is by converting them into pandas catgories.- We can check the different datatypes compatible with pandas here: https://pandas.pydata.org/pandas-docs/stable/reference/general_utility_functions.htmldata-types-related-functionality<jupyter_code>df_tmp.head().T
pd.api.types.is_string_dtype(df_tmp["UsageBand"])
for label, content in df_tmp.items():
    if pd.api.types.is_string_dtype(content):
        print(label)
random_dict = {"key1": "hello",
               "key2": "world!"}

for key, value in random_dict.items():
    print(f"this is a key: {key}",
          f"this is a value: {value}")
for label, content in df_tmp.items():
    if pd.api.types.is_string_dtype(content):
        df_tmp[label] = content.astype("category").cat.as_ordered()
df_tmp.info()
df_tmp.state.cat.categories
df_tmp.state.cat.codes
df_tmp.isnull().sum()/len(df_tmp)<jupyter_output><empty_output><jupyter_text>Save preprocessed data<jupyter_code>df_tmp.to_csv("train_tmp.csv",
              index=False)
df_tmp = pd.read_csv("train_tmp.csv",
                     low_memory=False)
df_tmp.head().T
df_tmp.isna().sum()<jupyter_output><empty_output><jupyter_text>Fill missing valuesFill numerical missing values first<jupyter_code>for label, content in df_tmp.items():
    if pd.api.types.is_numeric_dtype(content):
        print(label)
df_tmp.ModelID
for label, content in df_tmp.items():
    if pd.api.types.is_numeric_dtype(content):
        if pd.isnull(content).sum():
            print(label)
for label, content in df_tmp.items():
    if pd.api.types.is_numeric_dtype(content):
        if pd.isnull(content).sum():
            # Add a binary column which tells us if the data was missing or not
            df_tmp[label+"_is_missing"] = pd.isnull(content)
            # Fill missing numeric values with median
            df_tmp[label] = content.fillna(content.median())
hundreds = np.full((1000,), 100)
hundreds_billion = np.append(hundreds, 1000000000)
np.mean(hundreds), np.mean(hundreds_billion), np.median(hundreds), np.median(hundreds_billion)
for label, content in df_tmp.items():
    if pd.api.types.is_numeric_dtype(content):
        if pd.isnull(content).sum():
            print(label)
df_tmp.auctioneerID_is_missing.value_counts()
df_tmp.isna().sum()<jupyter_output><empty_output><jupyter_text>Filling and turning categorical variables into numbers<jupyter_code>for label, content in df_tmp.items():
    if not pd.api.types.is_numeric_dtype(content):
        print(label)
for label, content in df_tmp.items():
    if not pd.api.types.is_numeric_dtype(content):
        df_tmp[label+"_is_missing"] = pd.isnull(content)
        df_tmp[label] = pd.Categorical(content).codes+1
pd.Categorical(df_tmp["state"]).codes+1
df_tmp.info()
df_tmp.head().T
df_tmp.isna().sum()
df_tmp.head()
len(df_tmp)
%%time

model = RandomForestRegressor(n_jobs=-1,
                              random_state=42)

# Fitting the model
model.fit(df_tmp.drop("SalePrice", axis=1), df_tmp["SalePrice"])
model.score(df_tmp.drop("SalePrice", axis=1), df_tmp["SalePrice"])<jupyter_output><empty_output><jupyter_text>Splitting data into train/validation sets<jupyter_code>df_tmp.saleYear
df_tmp.saleYear.value_counts()
df_val = df_tmp[df_tmp.saleYear == 2012]
df_train = df_tmp[df_tmp.saleYear != 2012]
len(df_val), len(df_train)
X_train, y_train = df_train.drop("SalePrice", axis=1), df_train.SalePrice
X_valid, y_valid = df_val.drop("SalePrice", axis=1), df_val.SalePrice
X_train.shape, y_train.shape, X_valid.shape, y_valid.shape
y_train<jupyter_output><empty_output><jupyter_text>Building an evaluation function<jupyter_code>from sklearn.metrics import mean_squared_log_error, mean_absolute_error, r2_score
def rmsle(y_test, y_preds):
    return np.sqrt(mean_squared_log_error(y_test, y_preds))
def show_scores(model):
    train_preds = model.predict(X_train)
    val_preds = model.predict(X_valid)
    scores = {"Training MAE": mean_absolute_error(y_train, train_preds),
              "Valid MAE": mean_absolute_error(y_valid, val_preds),
              "Training RMSLE": rmsle(y_train, train_preds),
              "Valid RMSLE": rmsle(y_valid, val_preds),
              "Training R^2": r2_score(y_train, train_preds),
              "Valid R^2": r2_score(y_valid, val_preds)}
    return scores<jupyter_output><empty_output><jupyter_text>Testing our model on a subset<jupyter_code>len(X_train)
%%time

model.fit(X_train, y_train)
(X_train.shape[0] * 100) / 1000000
10000 * 100
show_scores(model)<jupyter_output><empty_output><jupyter_text>Hyerparameter tuning with RandomizedSearchCV<jupyter_code>%%time
from sklearn.model_selection import RandomizedSearchCV
rf_grid = {"n_estimators": np.arange(10, 100, 10),
           "max_depth": [None, 3, 5, 10],
           "min_samples_split": np.arange(2, 20, 2),
           "min_samples_leaf": np.arange(1, 20, 2),
           "max_features": [0.5, 1, "sqrt", "auto"],
           "max_samples": [10000]}
rs_model = RandomizedSearchCV(RandomForestRegressor(n_jobs=-1,
                                                    random_state=42),
                              param_distributions=rf_grid,
                              n_iter=2,
                              cv=5,
                              verbose=True)<jupyter_output><empty_output><jupyter_text>Make predictions on test data<jupyter_code>df_test = pd.read_csv("Test.csv",
                      low_memory=False,
                      parse_dates=["saledate"])

df_test.head()<jupyter_output><empty_output><jupyter_text>Preprocessing the data (To get the test dataset in the same format as training dataset)<jupyter_code>def preprocess_data(df):
    df["saleYear"] = df.saledate.dt.year
    df["saleMonth"] = df.saledate.dt.month
    df["saleDay"] = df.saledate.dt.day
    df["saleDayOfWeek"] = df.saledate.dt.dayofweek
    df["saleDayOfYear"] = df.saledate.dt.dayofyear
    df.drop("saledate", axis=1, inplace=True)
    for label, content in df.items():
        if pd.api.types.is_numeric_dtype(content):
            if pd.isnull(content).sum():
                df[label+"_is_missing"] = pd.isnull(content)
                df[label] = content.fillna(content.median())
        if not pd.api.types.is_numeric_dtype(content):
            df[label+"_is_missing"] = pd.isnull(content)
            df[label] = pd.Categorical(content).codes+1
    return df
df_test = preprocess_data(df_test)
df_test.head()
X_train.head()
set(X_train.columns) - set(df_test.columns)
df_test["auctioneerID_is_missing"] = False
df_test.head()<jupyter_output><empty_output><jupyter_text>- Finally test dataframe has the same features as our training dataframe. Feature ImportanceFeature importance seeks to figure out which different attributes of the data were most importance when it comes to predicting the target variable (SalePrice).<jupyter_code>def plot_features(columns, importances, n=20):
    df = (pd.DataFrame({"features": columns,
                        "feature_importances": importances})
          .sort_values("feature_importances", ascending=False)
          .reset_index(drop=True))
    
    # Plot the dataframe
    fig, ax = plt.subplots()
    ax.barh(df["features"][:n], df["feature_importances"][:20])
    ax.set_ylabel("Features")
    ax.set_xlabel("Feature importance")
    ax.invert_yaxis()
df["Enclosure"].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11484444444444444, 'fraction_numerical': 0.01448888888888889, 'mean_word_length': 4.2920978363123234, 'pattern_counts': {'":': 16, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '436330', 'license': 'MIT', 'n_tokens_mistral': 3912, 'n_tokens_neox': 3721, 'n_words': 851, 'path': 'sale-price-of-bulldozers/Blue Book for Bulldozers.ipynb', 'repo_name': 'abhisngh/Data-Science', 'size': '11250'}