<jupyter_start><jupyter_text>Step 1: import necessary packages and data<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import plotly.express as px
from scipy.stats import shapiro
from sklearn.feature_selection import RFE
from sklearn.model_selection import KFold, StratifiedKFold, train_test_split
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.dummy import DummyRegressor
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score, cross_validate, GridSearchCV
from sklearn.metrics import explained_variance_score, r2_score, mean_absolute_error, mean_squared_error
import xgboost
from xgboost import XGBRegressor, plot_importance

from collections import Counter


from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler<jupyter_output><empty_output><jupyter_text>Step 2: get high-level insights into data<jupyter_code>Data = pd.read_csv('Mileage.csv')
Data.columns=['mpg','cylinders','displacement','horsepower','weight','acceleration','model year','origin','car name']

Data.replace({'?':np.nan},inplace=True)

Data['horsepower'] = pd.to_numeric(Data['horsepower'])
Data.dtypes ## Confirmed that horsepower is object, so change to numeric
Data.describe()<jupyter_output><empty_output><jupyter_text>Take-away from above: there are 398 cars in this dataset, with set numbers of cylinders and set model years, set origin. mpg is target variable and is continuous, as are displacement, weight, acceleration Car names are strings, so not on here. Regression appears the best approach, with some features potentially needing encoding.  Step 2a) Seaborn heatmap to understand how different variables impact final predicted mpg<jupyter_code>plt.figure(figsize=(15,10))
sns.heatmap(Data.corr(),annot=True,annot_kws={"size":12})<jupyter_output><empty_output><jupyter_text>Conclusions from above: (1) Car weight and displacement have the strongest correlation: the heavier the weight and higher displacement, the lower mpg (2) Car horsepower and cylider number are also strongly correlated with mpg: more HP and more cylinders, less mpg (3) Less impactful: car origin, model year, acceleration.  Rule of thumb: threshold correlation coefficient for inclusion of feature is 0.6, sometimes as high as 0.7. Suggest dropping acceleration at minimum, and potentially origin, model year too  Step 2b) Using barplots to better understand distribution of categorical features<jupyter_code>plt.hist(Data['model year'],histtype='bar')
plt.hist(Data['origin'],histtype='bar')
plt.hist(Data['mpg'],histtype='bar')<jupyter_output><empty_output><jupyter_text>Conclusions from above:  - mpg is skewed right, suggesting potential need for log transform - categorical numerical variables increase with mpg, so simple encoding should suffice - not ordinal  Step 2c) Using boxplots to search for possible outliers<jupyter_code>Data.boxplot(column=['horsepower'])
Data.boxplot(column=['acceleration'])<jupyter_output><empty_output><jupyter_text>Step 3) Prepare dataset before training model on data by encoding and removing outliers  First create function to remove outliers, then apply:<jupyter_code># Outlier detection - courtesy of Yassine's https://www.kaggle.com/yassineghouzam/titanic-top-4-with-ensemble-modeling

def detect_outliers(df,n,features):
    """
    Takes a dataframe df of features and returns a list of the indices
    corresponding to the observations containing more than n outliers according
    to the Tukey method.
    """
    outlier_indices = []
    
    # iterate over features(columns)
    for col in features:
        # 1st quartile (25%)
        Q1 = np.percentile(df[col], 25)
        # 3rd quartile (75%)
        Q3 = np.percentile(df[col],75)
        # Interquartile range (IQR)
        IQR = Q3 - Q1
        
        # outlier step
        outlier_step = 2 * IQR
        
        # Determine a list of indices of outliers for feature col
        outlier_list_col = df[(df[col] < Q1 - outlier_step) | (df[col] > Q3 + outlier_step )].index
        
        # append the found outlier indices for col to the list of outlier indices 
        outlier_indices.extend(outlier_list_col)
        
    # select observations containing more than 2 outliers
    outlier_indices = Counter(outlier_indices)        
    multiple_outliers = list( k for k, v in outlier_indices.items() if v > n )
    
    return multiple_outliers  
drop = detect_outliers(Data,0,['mpg','cylinders','displacement','horsepower','weight','acceleration'])
Data = Data.drop(drop, axis = 0).reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Encoding categorical variables and imputing missing values in horsepower<jupyter_code># cols=['cylinders','model year','origin']
# Data[cols] = Data[cols].astype('str')

Data['car make'] = Data['car name']
Data['car make'] = Data['car name'].apply(lambda x: x.split()[0]) 

Data.drop(columns=['car name'],inplace=True)



Data = pd.get_dummies(Data,columns=['car make'])

Data['mpg'] = np.log(1 + 100*Data['mpg'])

x_no_log = Data.drop(columns=['mpg'])
y_no_log = Data['mpg']

imp = SimpleImputer(missing_values=np.nan,strategy='median')
x_no_log['horsepower'] = imp.fit(x_no_log['horsepower'].values.reshape(-1, 1)).transform(x_no_log['horsepower'].values.reshape(-1, 1))<jupyter_output><empty_output><jupyter_text>Create testing and training datasets<jupyter_code>xtrain,xtest,ytrain,ytest = train_test_split(x_no_log,y_no_log, test_size=0.3, random_state=42)<jupyter_output><empty_output><jupyter_text>Step 4) Creating instances of RandomForest and XGBoost, then setting up the hyperparameter grid for use in GridSearchCV<jupyter_code>rf = RandomForestRegressor()
xgbr = XGBRegressor()
rf_param_grid = {"max_depth": [None],
              "max_features": [5, 10, 20],
              "min_samples_split": [2, 3, 10],
              "min_samples_leaf": [1, 3, 10],
              "bootstrap": [False],
              "n_estimators" :[100,300]}
kfold = KFold(n_splits=10)
gsRFC_all = GridSearchCV(rf,param_grid = rf_param_grid, cv=7, scoring="neg_mean_squared_error", n_jobs= 4, verbose = 1)
gsRFC_all.fit(xtrain,ytrain)

RandForest_best = gsRFC_all.best_estimator_

# Best score
gsRFC_all.best_score_
xgb_params = {'nthread':[4], #when use hyperthread, xgboost may become slower
              'learning_rate': [.03, 0.05, .07], 
              'max_depth': [5, 6, 7],
              'min_child_weight': [4],
              'subsample': [0.7],
              'colsample_bytree': [0.7],
              'n_estimators': [500,1000]}

gsXGB = GridSearchCV(xgbr, xgb_params, cv = 7, scoring='neg_mean_squared_error', 
                     refit=True, n_jobs = 5, verbose=True)
gsXGB.fit(xtrain,ytrain)

XGB_best = gsXGB.best_estimator_

gsXGB.best_score_

ypred = XGB_best.predict(xtest)
explained_variance_score(ytest,ypred)
mean_absolute_error(ytest,ypred)
mean_squared_error(ytest,ypred,squared=True)
plot_importance(XGB_best)
ypred_alt = RandForest_best.predict(xtest)
explained_variance_score(ypred_alt,ytest)
mean_absolute_error(ytest,ypred_alt)
mean_squared_error(ytest,ypred_alt,squared=True)<jupyter_output><empty_output><jupyter_text>Step 5) Implementing PyTorch neural network to train on mpg dataset for predicting<jupyter_code>import torch
from torch import nn
# Define network dimensions
n_input_dim = xtrain.shape[1]
# Layer size
n_hidden = 4 # Number of hidden nodes
n_output = 1 # Number of output nodes for predicted mpg

# Build mdel
torch_model = torch.nn.Sequential(
    torch.nn.Linear(n_input_dim, n_hidden),
    torch.nn.ELU(),
    torch.nn.Linear(n_hidden, n_output)
)
    
print(torch_model)
loss_func = torch.nn.MSELoss() #Choosing mean square error as loss metric
learning_rate = 0.01
optimizer = torch.optim.Adam(torch_model.parameters(), lr=learning_rate)
train_error = []
iters = 600

Y_train_t = torch.FloatTensor(ytrain.values).reshape(-1,1) #Converting numpy array to torch tensor

for i in range(iters):
    X_train_t = torch.FloatTensor(xtrain.values)  #Converting numpy array to torch tensor
    y_hat = torch_model(X_train_t)
    loss = loss_func(y_hat, Y_train_t)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

    train_error.append(loss.item())
    
fig, ax = plt.subplots(2, 1, figsize=(12,8))
ax[0].plot(train_error)
ax[0].set_ylabel('Loss')
ax[0].set_title('Training Loss')
X_test_t = torch.FloatTensor(xtest.values)
ypredict = torch_model(X_test_t)
mean_squared_error(ytest,ypredict.detach().numpy(),squared=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10574473013666898, 'fraction_numerical': 0.014709288858003242, 'mean_word_length': 5.225666906993511, 'pattern_counts': {'":': 6, '<': 42, '<?xml version=': 0, '>': 43, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '455957', 'license': 'MIT', 'n_tokens_mistral': 3003, 'n_tokens_neox': 2792, 'n_words': 711, 'path': 'FINAL.ipynb', 'repo_name': 'Ranga2904/MPGPredictor_XG_PyTorch', 'size': '8634'}