<jupyter_start><jupyter_text>Advanced Models Data Analysis  Project 2 Used Vehicle Price Prediction  Members:* <NAME> - 201113637* <NAME> - 201111642* <NAME> - 201924139   1. Objectives* Build efficient car price prediction model based on Machine Learning.* Select the best prediction Model based on their Performance.* Expose the model as an API on a cloud service.  2. MethodologyIn order to select an effective car price prediction model, the Analytics team selected a DataBase from TrueCar.com that contains car data by the following aggregation:* Year.* Mileage* State.* Make.* Model.Three models based on Regression Trees will be used for evaluation:* Bagged Trees.* Random Forest.* XG Boosting.The current prediction seeks to find the most accurate price for a car considering its principal features, in which the performance indicator to select the model will be the RMSE (Square Root of Mean Square Error).  2.1. Data Preliminar Analysis- 1.2 Million listings scraped from TrueCar.com - Price, Mileage, Make, Model dataset from Kaggle: [data](https://www.kaggle.com/jpayne/852k-used-car-listings)- Each observation represents the price of an used car<jupyter_code>#Library Import 

%matplotlib inline
import pandas as pd
import numpy as np
import scipy as sp
import category_encoders as ce
import matplotlib.pyplot as plt

from sklearn import model_selection
from sklearn.model_selection import KFold
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score
from scipy.sparse import csr_matrix
from sklearn.metrics import mean_squared_error
from statistics import mean 
from random import randrange
data = pd.read_csv('../datasets/dataTrain_carListings.zip')
data.head()
data.shape
data.dtypes
data.Price.describe()
data.plot(kind='scatter', y='Price', x='Year')<jupyter_output><empty_output><jupyter_text>There is a correlation between the Mileage and Car Price. The year affect the price of the car, If the year of the car is recent, the more the Car Price will be.<jupyter_code>data.plot(kind='scatter', y='Price', x='Mileage')<jupyter_output><empty_output><jupyter_text>There is a correlation between the Mileage and Car Price. In this case the less Mileage, the less the Car Price will be.<jupyter_code>data.columns<jupyter_output><empty_output><jupyter_text>A random sample (50000 Observations)  is taken from computational purposes.<jupyter_code>data=data.sample(n=50000, random_state=1) # Se toma una muestra aleatoria de la base para hacer los entrenamientos y calibración de parametros<jupyter_output><empty_output><jupyter_text>2.2. Data ProcessingCar Listings brings information of the price of a car due to its key information.The type of every parameter is considered as it follows:* Year: Numeric.* Mileage: Numeric.* State: Categorical.* Make: Categorical.* Model: Categorical.<jupyter_code># Predictor Set
X_cat = data[["State", "Make", "Model"]]
X_num = data[["Year","Mileage"]]
encoder = ce.BinaryEncoder().fit(X_cat,axis=1)
X_cat = encoder.transform(X_cat)
X = pd.concat([X_num,X_cat],  axis=1, sort = False)
# Showing Predictor Data Frame Header

X.head()<jupyter_output><empty_output><jupyter_text>The Predictor Dataset will be hace 1000 observations and 594 variables<jupyter_code># Dependent Variable
y = data['Price']<jupyter_output><empty_output><jupyter_text>3. Model SelectionThe Machine Learning Models that will be evaluated are:* Bagged Trees.* Random Forest.* XGBoosting.In order to test the effectiveness of the models, they will be evaluated using a cross validation algorithm.  3.1. Cross Validation Algorithm Cross validation is one of the techniques used to test the effectiveness of machine learning models, it is also a re-sampling procedure used to evaluate a model if we have limited data.Since we have a large amount of data (500k rowsx546 colums), cross validation could be a good tool to reduce the computational load on the machine and get the best results with a limited sample of data.We have created our own Cross Validation Algorithm. As it follows:<jupyter_code># train/test split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=.3, random_state=1)
# function that creates the k folds splits 

from random import randrange

def cross_validation_split(dataset, folds):
        dataset_split = []
        dataset_split_index = []
        df_copy = dataset
        fold_size = int(df_copy.shape[0] / folds)
        
        # for loop to save each fold
        for i in range(folds):
            fold = []
            fold_index=[]
            # while loop to add elements to the folds
            while len(fold_index) < fold_size:
                # select a random element
                r = randrange(df_copy.shape[0])
                # determine the index of this element 
                index = df_copy.index[r]
                # save the randomly selected line 
                #fold.append(df_copy.loc[index].values.tolist())
                fold_index.append(df_copy.index[r].tolist())
                # delete the randomly selected line from
                # dataframe not to select again
                df_copy = df_copy.drop(index)
            # save the fold     
            #dataset_split.append(np.asarray(fold)) 
            dataset_split_index.append(np.asarray(fold_index))
            
        return dataset_split_index 
# function that defines the test and train sets for each iteration

def k_cv(dataset, f):
    data=cross_validation_split(dataset,f)
    test_index=[]
    train_index=[]
    
    #se defininen los tamaños de los sets 
    #para el set de training el tamaño es igual (k-1)*x[0].shape[0]
    #para el test set el tamaño es igual a x[0].shape[0]
    # determine training and test sets 
    
    for i in range(f):
        lista=data.copy()
        test_index.append(np.asarray(data[i]))
        lista.pop(i)
        x=np.concatenate(lista, axis=None )
        train_index.append(np.asarray(x))
       
    return [train_index, test_index]
# Kfold cross validation function definition

def cross_validation(X_train, y_train, model,k=10):
    
    scores = []
    
    cv = k_cv(X_train,k)

    train=cv[0]
    test=cv[1]
    
    
    for train_index in range(k): 
        
        for test_index in range(k):
            
                X_train_rf, X_test_rf, y_train_rf, y_test_rf =X_train.loc[train[train_index]], X_train.loc[test[test_index]], y_train.loc[train[train_index]], y_train.loc[test[test_index]]
    
    model.fit(X_train_rf, y_train_rf)
    y_pr=model.predict(X_test_rf)
    scores.append(mean_squared_error(y_test_rf, y_pr)**0.5)
    
    return mean(scores)<jupyter_output><empty_output><jupyter_text>3.2. Bagged TreesDecision trees are sensitive to the specific data on which they are trained. Bootstrap Aggregation is a general procedure that can be used to reduce the variance for those algorithm that have high variance.A Bagging Classifier meta-estimator can typically be used as a way to reduce the variance of a black-box estimator (in this case, will be used for a decision tree), by introducing randomization into its construction procedure and then making an ensemble out of it.An algorithm that has high variance are decision trees, like classification and regression trees (CART). For this prediction, we will use Regression trees.<jupyter_code>from sklearn.ensemble import BaggingRegressor
from sklearn.tree import DecisionTreeRegressor
# train/test split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=.3, random_state=1)
# Instruct BaggingRegressor to use DecisionTreeRegressor as the "base estimator"
bagreg = BaggingRegressor(DecisionTreeRegressor(), n_estimators=500, 
                          bootstrap=True, oob_score=True, random_state=42)
# fit and predict
bagreg.fit(X_train, y_train)
y_pred = bagreg.predict(X_test)
np.sqrt(mean_squared_error(y_test, y_pred))<jupyter_output><empty_output><jupyter_text>For Bagged Trees we obtained $5.462 of Mean Square Error.  Parameter TuningIn order to obtain a more accurate prediction for car price through Bagged Trees, we calibrate every parameter to evaluate the Resulting MSE.One of the principal parameters in Bagged Trees is n_estimators, which represents the base number of trees in the ensemble. This value will be evaluated between 10 to 1000 jumping from 100 to 100.<jupyter_code>#Calibrating the parameter n_stimators

# finding best value for learning_rate
n_estimators_range = range(10, 250, 50)

# list to store the average RMSE
accuracy_scores = []

for n in n_estimators_range:
    bagreg = BaggingRegressor(DecisionTreeRegressor(), n_estimators=n, 
                          bootstrap=True, oob_score=True, random_state=42)
    bagreg.fit(X_train, y_train)
    y_pred = bagreg.predict(X_test)
    accuracy_scores.append(mean_squared_error(y_test, y_pred)**0.5)
plt.plot(n_estimators_range, accuracy_scores)
plt.xlabel('n_estimators')
plt.ylabel('MSE')
results = {'values':n_estimators_range, 'MSE': accuracy_scores}
res2 = pd.DataFrame(data=results)
res2<jupyter_output><empty_output><jupyter_text>After the n_estimators tunning, we found that the MSE can be reduced Bagging the trees with 210 estimators, for which the MSE resultant will be $5.475 for a car price prediction.In order to obtain a more accurate prediction, we search if there is a better machine learning algorithm.  3.3. Random ForestIt is a bagging-based algorithm with a key difference wherein only a subset of features is selected at random.A random forest regressor is a meta estimator that fits a number of classifying decision trees on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting.<jupyter_code># Random forest 
clf = RandomForestRegressor(random_state=1, n_jobs=-1)
clf.fit(X_train, y_train)
y_pr=clf.predict(X_test)
mean_squared_error(y_test, y_pr)**0.5<jupyter_output><empty_output><jupyter_text>Applying Random Forest we obtained $5.501 of Mean Square Error predincting the price of a car given some requested features.  Parameter tunningIn order to obtain a better MSE in our Random Forest Prediction, we calibrate our parameters and observe the results.  In this case we will change: n_estimators: represents the number of trees in the forest. Usually the higher the number of trees the better to learn the data. However, adding a lot of trees can slow down the training process considerably,therefore we do a parameter search to find the sweet spot. This Tuning is done in order to reduce overfitting.<jupyter_code># findind the best n_estimators: 

# list of values to try for n_estimators
estimator_range = range(10, 1000, 100)

# list to store the average Accuracy for each value of n_estimators
accuracy_scores = []

# use 5-fold cross-validation with each value of n_estimators (WARNING: SLOW!)
for estimator in estimator_range:
    clf = RandomForestRegressor(n_estimators=estimator, random_state=1, n_jobs=-1)
    accuracy_scores.append(cross_validation(X_train, y_train, clf,k=10))
    
plt.plot(estimator_range, accuracy_scores)
plt.xlabel('n_estimators')
plt.ylabel('MSE')
results1 = {'values':estimator_range, 'MSE': accuracy_scores}
res1 = pd.DataFrame(data=results1)
res1
X.columns
feature_cols = X.columns<jupyter_output><empty_output><jupyter_text>max_features is the size of the random subsets of features to consider when splitting a node.<jupyter_code># list of values to try for max_features: 
feature_range = range(1, len(feature_cols)+1)

# list to store the average Accuracy for each value of max_features
accuracy_scores = []

# use 10-fold cross-validation with each value of max_features (WARNING: SLOW!)
for feature in feature_range:
    clf = RandomForestRegressor(n_estimators=110, max_features=feature, random_state=1, n_jobs=-1)
    accuracy_scores.append(cross_validation(X_train, y_train, clf,k=10))
    
plt.plot(feature_range, accuracy_scores)
plt.xlabel('max_features')
plt.ylabel('MSE')
results1 = {'values':feature_range, 'MSE': accuracy_scores}
res1 = pd.DataFrame(data=results1)
res1<jupyter_output><empty_output><jupyter_text>max_depth represents the depth of each tree in the forest. The deeper the tree, the more splits it has and it captures more information about the data.<jupyter_code># list of values to try for max_depth: 
max_depth_range = range(10, 100,10)

# list to store the average RMSE for each value of max_depth
accuracy_scores = []

for depth in max_depth_range:
    clf =  RandomForestRegressor(max_depth=depth, n_estimators=110,max_features=12, random_state=1,n_jobs=-1)
    accuracy_scores.append(cross_validation(X_train, y_train, clf,k=10))
    
    
plt.plot(max_depth_range, accuracy_scores)
plt.xlabel('max_depth')
plt.ylabel('MSE')
results1 = {'values':max_depth_range, 'MSE': accuracy_scores}
res1 = pd.DataFrame(data=results1)
res1
# model with the optimized parameters 
clf_optimizado =  RandomForestRegressor(max_depth=60, n_estimators=110, max_features=12, random_state=1,n_jobs=-1)
clf_optimizado.fit(X_train, y_train)
y_pr=clf_optimizado.predict(X_test)
mean_squared_error(y_test, y_pr)**0.5<jupyter_output><empty_output><jupyter_text>Applying Random Forest Machine Learning with con kfold =10 reduces RMSE from 5431.5 to  5.366  3.4. XGBoostXGBoost is a decision-tree-based ensemble Machine Learning algorithm that uses a gradient boosting framework.<jupyter_code># train/test split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=.2, random_state=1)
#XGBoost
from xgboost import XGBRegressor
from sklearn import metrics
xgb = XGBRegressor()
xgb
xgb.fit(X_train, y_train)
y_pred = xgb.predict(X_test)
metrics.mean_squared_error(y_pred, y_test.values)**0.5<jupyter_output><empty_output><jupyter_text>n_estimators : represents the number of trees in the forest. Usually the higher the number of trees the better to learn the data. learning_rate: A technique to slow down the learning in the gradient boosting model is to apply a weighting factor for the corrections by new trees when added to the model.This weighting is called the shrinkage factor or the learning rate, depending on the literature or the tool. Gamma: A node is split only when the resulting split gives a positive reduction in the loss function. Gamma specifies the minimum loss reduction required to make a split.Makes the algorithm conservative. The values can vary depending on the loss function and should be tuned.<jupyter_code># XGB  

xg =  XGBRegressor( random_state=1,n_jobs=-1)
xg.fit(X_train, y_train)
y_pred = xg.predict(X_test)
metrics.mean_squared_error(y_pred, y_test.values)**0.5<jupyter_output><empty_output><jupyter_text>Applying the XGBoost, the RMSE is 5.647  *Parameter Tuning*  Parameters:* Learning Rate: Step size shrinkage used in update to prevents overfitting.* Max_Depth: Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit.* Col_Sample (By tree): Is the subsample ratio of columns when constructing each tree. Subsampling occurs once for every tree constructed.<jupyter_code># findind best value for learning_rate
learning_rate_range = np.arange(0, 1, 0.1)

# list to store the average RMSE
accuracy_scores = []

for lr in learning_rate_range:
    xg = XGBRegressor(objective="reg:linear",eta=lr, random_state=42)
    xg.fit(X_train, y_train)
    y_pred = xg.predict(X_test)
    accuracy_scores.append(mean_squared_error(y_test, y_pred)**0.5) 
plt.plot(learning_rate_range, accuracy_scores)
plt.xlabel('learning_rate')
plt.ylabel('MSE')
results = {'values':learning_rate_range, 'accuracy': accuracy_scores}
res = pd.DataFrame(data=results)
res
res.min()
# defining colsample_bytree
col_range = np.arange(0, 1, 0.1)

# list to store the average RMSE
accuracy_scores = []

for col in col_range:
    xgb_model = XGBRegressor(objective="reg:linear",colsample_bytree=col ,eta=0.2,random_state=42)
    xgb_model.fit(X_train, y_train)
    y_pred = xgb_model.predict(X_test)
    accuracy_scores.append(mean_squared_error(y_pred, y_test.values)**0.5) 
plt.plot(col_range, accuracy_scores)
plt.xlabel('colsample_bytree')
plt.ylabel('MSE')
results1 = {'values':col_range, 'MSE': accuracy_scores}
res1 = pd.DataFrame(data=results1)
res1
res1.min()
# findind best value for max_depth
max_depth_range = range(0, 20, 2)

# list to store the average RMSE
accuracy_scores = []

for md in max_depth_range :
    xg = XGBRegressor(objective="reg:linear",eta=0.2,colsample_bytree=0.8 ,max_depth=md, random_state=42)
    xg.fit(X_train, y_train)
    y_pred = xg.predict(X_test)
    accuracy_scores.append(mean_squared_error(y_pred, y_test.values)**0.5) 
plt.plot(max_depth_range, accuracy_scores)
plt.xlabel('max_depth')
plt.ylabel('MSE')
results1 = {'values':max_depth_range, 'MSE': accuracy_scores}
res1 = pd.DataFrame(data=results1)
res1
res1.min()
# XGB optimizado 

xg_op= XGBRegressor(objective="reg:linear",eta=0.2,colsample_bytree=0.8 ,max_depth=8, random_state=42)
xg_op.fit(X_train, y_train)
y_pred = xg_op.predict(X_test)
mean_squared_error(y_pred, y_test.values)**0.5<jupyter_output>[23:44:48] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.0.0/src/objective/regression_obj.cu:167: reg:linear is now deprecated in favor of reg:squarederror.<jupyter_text>The MSE for the XG Boosting is $5.103 with the Optimized parameters.|  Parameters      | Value   || ---------------- |:-------:|| Learning Rate    | 0.2     || Max_Depth        | 8       |   | Col_Sample       | 0.8     |  Results and Conclusions |  Model           | MSE     || ---------------- |:-------:|| Bagged Trees     | 5.475   || Random Forest    | 5366.10 || XG Boosting      | 5.103   |According to the results, we can see that XGBoost is the model with the best performance (lower RMSE), with a RMSE of 5.103This model will be used in the API to predict the price of a car.  API SET UP  1. Save model<jupyter_code>import joblib
joblib.dump(clf_optimizado, 'model_deployment_proyecto/car_price_prediction.aaaa', compress=3)
joblib.dump(encoder, 'model_deployment_proyecto/encoder.aaaa', compress=3)<jupyter_output><empty_output><jupyter_text>2. Model in batch<jupyter_code>from model_deployment_proyecto.m09_model_deployment import predict_price
predict_price(Year=2015,Mileage=54593,State='MS',Make='Toyota',Model='CamrySE')<jupyter_output><empty_output><jupyter_text>3.API<jupyter_code>from flask import Flask
from flask_restx import Api, Resource, fields
import joblib
from model_deployment_proyecto.m09_model_deployment import predict_price

app = Flask(__name__)

api = Api(
    app, 
    version='1.0', 
    title='Car Prices Prediction API',
    description='Car Prices Prediction API')

ns = api.namespace('predict', 
     description='Price prediction')
   
parser = api.parser()

parser.add_argument(
    'Year', 
    type=int, 
    required=True, 
    help='Year to be analyzed', 
    location='args')

parser.add_argument(
    'Mileage', 
    type=int, 
    required=True, 
    help='Mileage to be analyzed', 
    location='args')

parser.add_argument(
    'State', 
    type=str, 
    required=True, 
    help='State to be analyzed', 
    location='args')

parser.add_argument(
    'Make', 
    type=str, 
    required=True, 
    help='Make to be analyzed', 
    location='args')


parser.add_argument(
    'Model', 
    type=str, 
    required=True, 
    help='Model to be analyzed', 
    location='args')

resource_fields = api.model('Resource', {
    'result': fields.String,
})

@ns.route('/')
class PhishingApi(Resource):

    @api.doc(parser=parser)
    @api.marshal_with(resource_fields)
    def get(self):
        args = parser.parse_args()
        
        return {
        "result": predict_price(args['Year'],args['Mileage'],args['State'],args['Make'],args['Model'])
        }, 200
    
    
if __name__ == '__main__':
    app.run(debug=True, use_reloader=False, host='0.0.0.0', port=5000)<jupyter_output>* Serving Flask app "__main__" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
{'fraction_non_alphanumeric': 0.0887914901999409, 'fraction_numerical': 0.017137791785679108, 'mean_word_length': 4.645537948290242, 'pattern_counts': {'":': 1, '<': 79, '<?xml version=': 0, '>': 78, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '215724', 'license': 'MIT', 'n_tokens_mistral': 6733, 'n_tokens_neox': 6327, 'n_words': 1986, 'path': 'Exercises/P2-UsedVehiclePricePredictionCL.ipynb', 'repo_name': 'catalinalago/Proyecto2_ModelosAvanzadosMIIA', 'size': '20361'}