<jupyter_start><jupyter_text>Real-time Ad Bidding Price Predictor_Author: <NAME>_ The task in this dataset is to predict the base price an advertiser would need to pay to win an auction for their ad to be served on a publisher's website. We're asked to predict next month's reserve base prices. The publisher's objective is to determine a reserve price for their inventory to maximize revenue. The minimum (reserve) price will then be set by the Publisher in their programmatic advertising platform.This is a supervised, regression machine learning problem.  About the Dataset- This data was uploaded to Kaggle on 2020-06-02 by <NAME>.- Each row in the data represents the actual revenue generated by a publisher, and not at the bid level.- Data dictionary below| Key | Value Type | Definition || :- | :- | :- || date | string | The date the ad was served on || advertiser_id | integer |  each id denotes a different bidder in the auction || order_id | integer | each id denotes an order generated at the time of creation. Each order ID is useful to ensure you and others are referring to the same Advertiser campaign. || line_item_type_id | integer | line items contain information about how specific ads are served on a website along with pricing and other delivery details. For example: Sponsorship, Standard, Network, Bulk, Price Priority. Line item types also have different priorities, or how a line item competes with other line items. Not all line items have the same importance. Some line items may be contractually guaranteeded to serve or may promise more revenue than others. These can be priotiezed higher than other line items. || monetization_channel_id | integer |  it describes the mode through which demand partner integrates with a particular publisher - it can be header bidding (running via prebid.js), dynamic allocation, exchange bidding, direct etc || site_id | integer | each id denotes a different website || ad_type_id | integer | each id denotes a different ad_type. These can be display ads , video ads, text ads etc || ad_unit_id | integer | each id denotes a different ad unit (one page can have more than one ad units) | | geo_id | integer |  each id denotes a different country. our maximum traffic is from english speaking countries || device_category_id | integer | each id denoted a different device_category like desktop , mobile, tablet || os_id | integer | each id denotes a different operating system for mobile device category only (android , ios etc) . for all other device categories, osid will correspond to not_mobile || integration_type_id | integer |  it describes how the demand partner is setup within a publisher's ecosystem - can be adserver (running through the publisher adserver) or hardcoded || total_impressions | float | measurement column measuring the impressions for the particular set of dimensions || viewable_impressions | float | Number of impressions on the site that were viewable out of all measurable impressions. A display ad is counted as viewable if at least 50% of its area was displayed on screen for at least one second || measureable_impressions | float | Impressions that were measurable by Active View out of the total number of eligible impressions. This value should generally be close to 100%. For example, an impression that is rendering in a cross-domain iframe may not be measurable. Active View is a technology on YouTube and certain Display Network websites and mobile apps that allows Google Ads to determine if your ad is viewable by potential customers. || total_revenue | float | measurement column measuring the revenue for the particular set of dimensions || revenue_share_percentage | float | not every advertiser gives all the revenue to the publisher. They charge a certain share for the services they provide. This captures the fraction of revenue that will actually reach the publishers pocket || cpm | float | CPM: cost-per-thousand impressions. A measure that calculates the Adverrtiser's cost for 1000 impressions  Install Required Libraries<jupyter_code>!pip install pydot --quiet<jupyter_output><empty_output><jupyter_text>Import Libraries<jupyter_code># Data analysis and wrangling
import numpy as np
import pandas as pd

# Data visualization
import matplotlib.pyplot as plt
import seaborn as sns

# show plots in Jupyter Notebook browser
%matplotlib inline 

# Jupyter enviornment
from IPython.core.display import display, HTML

# Datetime
import datetime

# Machine Learning
from sklearn.tree import DecisionTreeRegressor, export_graphviz
from sklearn.ensemble import RandomForestRegressor

# ML helper functions
from sklearn.model_selection import train_test_split, cross_val_score, cross_validate, KFold, GridSearchCV
from sklearn.metrics import mean_squared_error<jupyter_output><empty_output><jupyter_text>Configure Jupyter Enviornment<jupyter_code># Set pandas display configurations with more rows, columns, and greater column width
pd.set_option('display.max_rows', 999)
pd.set_option('display.max_columns', 999)
pd.set_option('display.max_colwidth', 255)

# Expand the jupyter container to have more working space
display(HTML("<style>.container {width:80% !important} </style>"))

# Configure Plotting Style Guide
plt.style.use('ggplot')<jupyter_output><empty_output><jupyter_text>Load the Data<jupyter_code># Hook up google drive to Colab
from google.colab import drive
drive.mount('/content/drive')
# Read in the file
relative_path = '/Users/etran/datr-1116/final_project_2/data/rtb-ad-dataset.csv'
ads_df = pd.read_csv(relative_path)<jupyter_output><empty_output><jupyter_text>Examine the Data<jupyter_code># Display the first 5 rows
ads_df.head()<jupyter_output><empty_output><jupyter_text>Shape of Data<jupyter_code># Display the number of rows and columns
ads_df.shape<jupyter_output><empty_output><jupyter_text>Check for Missing Data<jupyter_code>ads_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Feature EngineeringLets convert the date from string to datetime data type<jupyter_code># Replace existing date column with datetime
ads_df['date'] = pd.to_datetime(ads_df['date'])<jupyter_output><empty_output><jupyter_text>Now lets create a function that will extract the Year, Month, Day, Weekday, and Day of Week<jupyter_code># Function to extract the Year, Month, Day, Weekday, and Day Of Week from date column in dataframe
def get_date_attributes(df):
    
    # Retrieve year of date
    ads_df['year'] = ads_df['date'].apply(lambda x: x.year)
    
    # Retrieve month of date
    ads_df['month'] = ads_df['date'].apply(lambda x: x.month)
    
    # Retrieve day of date
    ads_df['day'] = ads_df['date'].apply(lambda x: x.day)
    
    # Retrieve month of date
    ads_df['weekday'] = ads_df['date'].apply(lambda x: x.weekday())    

    # Retrieve month of date
    ads_df['day_of_week'] = ads_df['date'].apply(lambda x: x.strftime("%A"))
    
# Retrieve date attributes
get_date_attributes(ads_df)<jupyter_output><empty_output><jupyter_text>Next, compute the CPM- _CPM = (Cost to the Advertiser / No. of Impressions) x 1000_<jupyter_code># Calculate the CPM. CPM = (total_revenue / measurable_impressions) x revenue_share_percent x 1000. 
def revenue_per_impression(r, i):
    return r / i if i else 0

ads_df['revenue_per_imp'] = ads_df.apply(lambda x: revenue_per_impression(((x['total_revenue'])*x['revenue_share_percent']),x['measurable_impressions']) , axis=1)
ads_df['CPM'] = ads_df['revenue_per_imp'].apply(lambda x: x*1000)<jupyter_output><empty_output><jupyter_text>Data PreparationHere we look at the dimensions of the data and identify any anomalies, missing data, outliers that will affect our model.<jupyter_code>ads_df.describe().round(2).T<jupyter_output><empty_output><jupyter_text>Lets filter out all observations with negative CPM values and remove outliers from our dataset<jupyter_code># Save a copy of the original dataframe in new variable
ads_df_original = ads_df.copy()

# Filter out all oberservations with negative CPM
ads_df = ads_df[ads_df['CPM'] >= 0]

# Filter out the outliers from the dataset 
ads_df = ads_df[ads_df['CPM'] < ads_df['CPM'].quantile(0.95)]

print('***** DATASET *****')
print("Original: {}".format(ads_df_original.shape))
print("Filtered: {}".format(ads_df.shape))
print('')
print("Remaining: {:.2f}%".format((ads_df.shape[0] /ads_df_original.shape[0])*100))<jupyter_output>***** DATASET *****
Original: (567291, 24)
Filtered: (538924, 24)

Remaining: 95.00%<jupyter_text>Feature Selection<jupyter_code>ads_df.columns<jupyter_output><empty_output><jupyter_text>**Criteria**- We will drop 'total_impressions', 'viewable_impressions', 'measureable_impressions', 'unfilled_impressions', and 'total_revenue' as we used these features to calculate CPM- We will drop 'integration_type_id' and 'revenue_share_percent' because these features only contain one value- We will drop 'date', 'year', 'month', 'day', 'weekday', 'day_of_week' because these are mainly used for EDA- We will drop 'revenue_per_imp' because is just a proxy to 'CPM'<jupyter_code>feature_list = ['advertiser_id', 
                'order_id', 
                'monetization_channel_id', 
                'line_item_type_id', 
                'site_id', 
                'ad_unit_id', 
                'ad_type_id', 
                'geo_id', 
                'device_category_id', 
                'os_id']

target_col = 'CPM'

X = ads_df[feature_list].copy()
y = ads_df[target_col].copy()
print(X.shape)
print(y.shape)<jupyter_output>(538924, 10)
(538924,)<jupyter_text>Train/Test Split Split the dataset so 70% is used for training and 30% for testing<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)
print('***** TRAIN/TEST SPLIT *****')
print('X       : ', X.shape)
print('y       : ', y.shape)
print('X_train : ', X_train.shape)
print('y_train : ', y_train.shape)
print('X_test  : ', X_test.shape)
print('y_test  : ', y_test.shape)
print('')
print('Training: {:0.2f}'.format(X_train.shape[0]/X.shape[0]))<jupyter_output>***** TRAIN/TEST SPLIT *****
X       :  (538924, 10)
y       :  (538924,)
X_train :  (377246, 10)
y_train :  (377246,)
X_test  :  (161678, 10)
y_test  :  (161678,)

Training: 0.70<jupyter_text>ML Models  Baseline ModelFirst, we need to establish a baseline that we will use as a starting point to compare our models. We can establish a baseline by using the average CPM for all days in June. In other words, our baseline is the error between the overall average CPM minus the acutual CPM.<jupyter_code>ads_df_preds = pd.concat([X,y], axis=1)
ads_df_preds['predicted_CPM'] = ads_df_preds['CPM'].mean()
ads_df_preds.sample(5)
# Calculate the error between predicted vs actuals
errors = abs(ads_df_preds['predicted_CPM'] - ads_df_preds['CPM'])

# Calculate the average accuracy
mape = 100 * (errors / ads_df_preds['CPM'])
mape.replace([np.inf, -np.inf],0,inplace=True)

accuracy = 100 - np.mean(mape)
print('Accuracy: {:.2f}%'.format(accuracy))
print('MSE Score: {:.3f}'.format(mean_squared_error(ads_df_preds['CPM'], ads_df_preds['predicted_CPM'])))
print('RMSE Score: {:.3f}'.format(np.sqrt(mean_squared_error(ads_df_preds['CPM'], ads_df_preds['predicted_CPM']))))<jupyter_output>MSE Score: 1.044
RMSE Score: 1.022<jupyter_text>**Interpretation:** Our baseline model predicts the correct CPM only 9.46% of the time. On average, our predicted value varies from our actual CPM price by $1.02. One way to look at this is if we ploted this on a scatter plot, we would draw a line straight through it using the average of all CPM values and calculate the variance between the line versus the actual values. The goal of our ML models is to produce an error lower than our baseline.  Decision Tree RegressorDecision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. Decision trees are effective for learning non-linear functions, and are particular adept at finding threshold effects.<jupyter_code># Instantiate model with defaults
tree_reg = DecisionTreeRegressor(random_state=42)

# Print default model parameters
tree_reg.get_params()
# Train the model on training data
tree_reg.fit(X_train, y_train)
# Make predictions on the test set
y_train_pred = tree_reg.predict(X_train)
y_test_pred = tree_reg.predict(X_test)

# Display performance metrics
print("R Squared: {:.3f}".format(tree_reg.score(X_test, y_test)))
print('')
print("TRAIN MSE  : {:.3f}".format(mean_squared_error(y_train, y_train_pred)))
print("TEST MSE   : {:.3f}".format(mean_squared_error(y_test, y_test_pred)))
print("TEST RMSE  : {:.3f}".format(np.sqrt(mean_squared_error(y_test, y_test_pred))))<jupyter_output>R Squared: 0.700

TRAIN MSE  : 0.220
TEST MSE   : 0.313
TEST RMSE  : 0.559<jupyter_text>**Interpretation:** Out of the box, our decision tree model has learned how to predict the maximum CPM for July with a 70% accuracy.<jupyter_code># Compute feature importance
feature_importance = pd.DataFrame({'feature':feature_list, 'importance':tree_reg.feature_importances_}).sort_values(by='importance', ascending=False)
feature_importance
# Plot the feature importance
feature_importance = pd.DataFrame({'feature':feature_list, 'importance':tree_reg.feature_importances_}).sort_values(by='importance')
ax = feature_importance.plot(x='feature', y='importance', kind='barh', figsize=(12,6))
ax.set_title("Feature Importance", fontsize=13, pad=10)
ax.legend(loc='center right');<jupyter_output><empty_output><jupyter_text>Improve Model w/ Hyperparamater TuningMake a bunch of different models with different settings, evaluate them all on the same validation set, and see which one does the best. We'll try to tune two hyperparameters `max_depth, and max_features`. (Warning: This can be computationally expensive!) Tune `max_depth`<jupyter_code># List of values to try for max_depth:
max_depth_range = list(range(1, 40))

# List to store the average RMSE for each value of max_depth:
RMSE_scores = []

# Use 5-fold cross-validation with each value of max_depth
for depth in max_depth_range:
    tree_reg = DecisionTreeRegressor(max_depth=depth, random_state=42)
    MSE_scores = cross_val_score(tree_reg, X_train, y_train, cv=5, scoring='neg_mean_squared_error')
    RMSE_scores.append(np.mean(np.sqrt(-MSE_scores)))
# Plot max_depth (x-axis) versus RMSE (y-axis).
plt.plot(max_depth_range, RMSE_scores);
plt.xlabel('max_depth');
plt.ylabel('RMSE (lower is better)');<jupyter_output><empty_output><jupyter_text>Tune `max_features`<jupyter_code># List of values to try for max_features:
max_features_range = list(range(1, 10))

# List to store the average RMSE for each value of max_depth:
RMSE_scores = []

# Use 5-fold cross-validation with each value of max_depth
for feature in max_features_range:
    tree_reg = DecisionTreeRegressor(max_features=feature, random_state=42)
    MSE_scores = cross_val_score(tree_reg, X_train, y_train, cv=5, scoring='neg_mean_squared_error')
    RMSE_scores.append(np.mean(np.sqrt(-MSE_scores)))
# Plot max_features (x-axis) versus RMSE (y-axis).
plt.plot(max_features_range, RMSE_scores);
plt.xlabel('max_features');
plt.ylabel('RMSE (lower is better)');
# Re-train the Decision Tree model with the new hyper parameters

# Instantiate model with new hyper parameters
tree_reg_hp = DecisionTreeRegressor(max_depth=20, max_features=7)

# Train the model on training data
tree_reg_hp.fit(X_train, y_train)

# Make predictions on the test set
y_test_pred_hp = tree_reg_hp.predict(X_test)

# Display performance metrics
print("R Squared: {:.3f}".format(tree_reg_hp.score(X_test, y_test)))
print('')
print("TEST MSE   : {:.3f}".format(mean_squared_error(y_test, y_test_pred_hp)))
print("TEST RMSE  : {:.3f}".format(np.sqrt(mean_squared_error(y_test, y_test_pred_hp))))<jupyter_output>R Squared: 0.700

TEST MSE   : 0.313
TEST RMSE  : 0.559<jupyter_text>**Summary:** Our decision tree model performs about the same out of the box versus after tuning the `max_depth` and `max_features` hyperparmeters. Lets try another model to see if we can get better predictions.  Random Forest RegressorRandom forests are an ensemble method where hundreds (or thousands) of individual decision trees are fit to boostrap re-samples of the original dataset, with each tree being allowed to use a random selection of N variables, where N is the major configurable parameter of this algorithm. Ensembling many re-sampled decision trees serves to reduce their variance, producing more stable estimators that generalize well out-of-sample. Random forests are extrememly hard to over-fit, are very accurate, generalize well, and require little tuning.<jupyter_code># Instantiate Random Forest Regressor
rf_reg = RandomForestRegressor(n_estimators=200, random_state=42)

# Get Default Paramaters
rf_reg.get_params()
# Train the model on training data
rf_reg.fit(X_train,y_train)

# Predict using test set
y_pred = rf_reg.predict(X_test)
# Display performance metrics
print("R Squared  : {:.3f}".format(rf_reg.score(X_test, y_test))) #Return the coefficient of determination of the prediction.
print('MSE Score  : {:.3f}'.format(mean_squared_error(y_test, y_pred)))
print('RMSE Score : {:.3f}'.format((np.sqrt(mean_squared_error(y_test, y_pred)))))<jupyter_output>R Squared  : 0.714
MSE Score  : 0.299
RMSE Score : 0.546<jupyter_text>**Interpretation:** Out of the box, our random forest model has learned how to predict the CPM with a 71% accuracy. We saw a 0.013 or 4.2% improvement in MSE (0.298 vs 0.311) with our Random Forest model compared to our Decision Tree model. Not bad, could we do better by tuning our hyper parameters?  Improve Model w/ Hyperparameter TuningWe will attempt to improve the model by adjusting the settings for these three hyperparameters: `n_estimators, max_features, and min_samples_split`. (Warning: This can be computationally expensive!)<jupyter_code># List of values to try for n_estimators:
estimator_range = list(range(10, 200, 10))

# List to store the average RMSE for each value of n_estimators:
mse_scores = []

# Use five-fold cross-validation with each value of n_estimators (Warning: Slow!).
for estimator in estimator_range:
    rf_reg = RandomForestRegressor(n_estimators=estimator, random_state=42)
    MSE_score = cross_val_score(rf_reg, X, y, cv=5, scoring='neg_mean_squared_error')
    mse_scores.append(np.mean((-MSE_score)))

# Find the average RMSE
print('Average RMSE: {:.3f}'.format(np.mean(np.sqrt(mse_scores))))<jupyter_output><empty_output><jupyter_text>**Summary:** At this point, our model performs pretty decent at 71% accuracy. But this is pretty much a black box approach. We basically feed the random forest model some data to train on, asked it to make a prediction, then we evaluate the predictions and ask ourselves if they seem reasonable, but we really don't know how the model arrived at these predictions.There are a couple of ways we can figure out how the "sausage" is made. 1. Look at the feature importances of our explantory variables.2. Look at a single tree in the forest and see how the model is making each split  Intepret Model Results<jupyter_code># Compute feature importance
feature_importance = pd.DataFrame({'feature': feature_list, 'importance': rf_reg.feature_importances_}).sort_values('importance', ascending=False)
feature_importance
# Plot the feature importance
feature_importance = pd.DataFrame({'feature': feature_list, 'importance': rf_reg.feature_importances_}).sort_values('importance')
ax = feature_importance.plot(x='feature', y='importance', kind='barh', figsize=(12,6))
ax.set_title("Feature Importance", fontsize=13, pad=10)
ax.legend(loc='center right');
# Plot the cumulative feature importance


# Sort Feature Importance
sorted_feature_importances = feature_importance.sort_values('importance', ascending=False).reset_index(drop=True)
sorted_feature_importances = sorted_feature_importances['importance']

# Sort Features
sorted_features = feature_importance.sort_values('importance', ascending=False).reset_index(drop=True)
sorted_features = sorted_features['feature']

# Cumulative importances
cumulative_importances = np.cumsum(sorted_feature_importances)

# Make a line graph
plt.figure(figsize=(12,6))
plt.plot(sorted_features, cumulative_importances, 'g-')

# Draw line at 95% of importance attained
plt.hlines(y = 0.95, xmin=0, xmax=len(sorted_feature_importances), color = 'black', linestyles = 'dashed')
plt.xticks(sorted_features, sorted_features, rotation=45)
plt.xlabel('Variable'); plt.ylabel('Cumulative Importance'); 
plt.title('Cumulative Importances', fontsize=15);
# Display an example of a small tree and backtest it!

# Limit depth of tree to 3 levels
rf_small = RandomForestRegressor(n_estimators=10, max_depth=3)
rf_small.fit(X_train, y_train)

# Extract the small tree
tree_small = rf_small.estimators_[5]

# Save the tree as a png image
export_graphviz(tree_small, out_file = 'small_tree.dot', feature_names=feature_list, rounded=True, precision=1)
(graph, ) = pydot.graph_from_dot_file('small_tree.dot')
graph.write_png('small_tree.png');<jupyter_output><empty_output><jupyter_text>![small_tree.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABegAAAHxCAIAAACZMsuKAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd0CT1/4/8JMQCCNsARkOliCi4kC24hVZRREVERUEax2t1oXSWip6URFBWqnX0duKA6uMC1QttVoQy5AgyqgKdVWKpewRLASMye+P53fzzQVERiQo79dfPOc5OeedGIJ8OM95aAKBgAAAAAAAAAAAwNBDl3QAAAAAAAAAAADoHgo3AAAAAAAAAABDFAo3AAAAAAAAAABDF<KEY>//<KEY>)<jupyter_code># Validate a data point from the dataset using the top 5 feature, which makes up about 90% 
columns = ['date', 'line_item_type_id', 'advertiser_id', 'ad_type_id', 'geo_id',  'ad_unit_id', 'CPM']
pd.DataFrame(ads_df[columns].loc[110350])<jupyter_output><empty_output><jupyter_text>**Interpretation:** Lets look at an example where we're making a prediction for July using a random day from June. The actual variables are line_item_type_id, advertiser_id, geo_id, ad_type_id. If we follow the tree down we will see we predicted a value of 1.9. 1. Start a root node evaluate if `line_item_type_id <= 15`, its `False` so we move to the right2. Next evaluate if `geo_id <= 186`, its `False` so we move to the right again3. Lastly, evaluate if `ad_type_id <= 13.5`, its `True` so we move left and end at 1.9 as our predicted CPM value.The root node only has 238k/538k data points. This is because in a random forest, each tree is trained on a random subset of the data points with replacement. Random sample of data points, combined with random sampling of a subset of features at each node of the tree is why the model is called a "random" forest.  Save Predictions / Model<jupyter_code># Save a copy of processed dataset
ads_df.to_csv('rtb-ad-dataset-processed.csv', index=False)
import pickle
# context manager: opens file > write file > close out file
# .pkl extension, wb = write byte, assign alias
# Warning: only pickle files you know of or wrote. They can be embedded code that has virus or malicious code.
with open('rtb-predictor-rf_reg-model.pkl', 'wb') as pickle_file:
    pickle.dump(rf_reg, pickle_file)<jupyter_output><empty_output><jupyter_text>Next Steps & Ideas to Improve Model Performance- All observations with line_item_type = 3, 9, 11 all receive impressions but 0 CPM value. We can try to exclude these observations from our dataset to improve the accuracy of our model.- Try excluding unimportant features for future model iterations- Try different train/test splits (90/10, 80/20, 70/30, time-series-split) to see if our model's performance improves.- Try different hyperparameters (settings). However, need to weigh the tradeoff between decrease in accuracy versus the extra time required to train the models (performance vs cost from computation resources).- Try different ml algorithms such as XGB, LGB, and CAT- Add more data! As of now, we are confident our model can predict the reserve CPM with 71% accuracy from one month of data. - Create mapping function to apply "category labels" to explantatory variables (numerical IDs) to gain more context and insights into ID values.- Deploy model into a production enviornment and track prediction CPM vs actual CPM accuracy over time (Daily, Weekly cadence).- Build an app using streamlit so end-users can toggle different configurations to get reserve CPMs  Closing RemarksOverall, this publisher who owns multiple website makes about 40k/month or 480k/year on ad revenue. Programmatic advertising uses your reserve CPM rates to select the highest bidder. When your reserve CPM are outdated, your ad networks would be competing with incorrect rates, preventing you from maximizing ad revenue. With this CPM predictor, you could increase your revenue by updating your CPM rates at least once per month. Then you can run optimization tests to determine the sweet spot between maximizing ad utilization and revenue.  Model Selection (Continued)  Random Search with Cross ValidationWe'll use a random search cross validation model to identify the best hyperparameters we should use. The RandomizedSearchCV method from the sklearn library returns a grid of hyperparameters randomly sample for each K-fold cross validation.<jupyter_code>from sklearn.model_selection import RandomizedSearchCV
# Create the random grid
random_grid = {'bootstrap': [True, False], # method of sampling data points
               'max_depth': [10, 25, 50, 100, 150, 200, None], # max number of levels in each decision tree
               'max_features': ['auto'], # max number of features considered for splitting a node
               'min_samples_leaf': [1, 5], # min number of data points allowed in a leaf node
               'min_samples_split': [2, 5, 10], # min number of data points placed in a node before the node is split
               'n_estimators': [10, 25, 50, 100, 150, 200] # number of trees in the forest
               }

random_grid
# First create the base model to tune
rf = RandomForestRegressor(random_state = 42)

# Random search of parameters, using 3 fold cross validation, search across 100 different combinations, and use all available cores
rf_random = RandomizedSearchCV(estimator=rf, param_distributions=random_grid,
                              n_ite=100, scoring='neg_mean_absolute_error', 
                              cv=3, verbose=2, random_state=42, n_jobs=-1,
                              return_train_score=True)

# Fit the random search model
rf_random.fit(X_train, y_train);
rf_random.best_params_
cv_results = pd.DataFrame.from_dict(rf_random.cv_results_)
print(cv_results.shape)
cv_results.sort_values('rank_test_score', ascending=False).head(2).T<jupyter_output><empty_output><jupyter_text>Grid Search Cross Validation<jupyter_code>from sklearn.model_selection import GridSearchCV
# Create the parameter grid based on the results of random search 
param_grid = {'bootstrap': [True],
              'max_depth': [None],
              'max_features': [0.4],
              'min_samples_leaf': [5],
              'min_samples_split': [8, 10, 12],
              'n_estimators': [150, 200, 250]
              }

# Create a based model
rf_reg = RandomForestRegressor()

# Instantiate the grid search model
grid_search = GridSearchCV(estimator = rf_reg, param_grid=param_grid, cv=3, n_jobs =-1, verbose=2, return_train_score=True)
# Fit the grid search to the data
grid_search.fit(X_train, y_train)
grid_search.best_params_
gs_cv_results = pd.DataFrame.from_dict(grid_search.cv_results_)
print(gs_cv_results.shape)
gs_cv_results.sort_values('rank_test_score', ascending=False).head(2)<jupyter_output>(9, 22)<jupyter_text>Random Forest Regressor w/ Only Important Features (5)<jupyter_code># Instantiate another Random Forest Model
rf_most_important = RandomForestRegressor(n_estimators=200, random_state=42)
# Select the 4 most important features
important_features =  ['line_item_type_id', 'order_id', 'ad_type_id', 'geo_id',  'ad_unit_id']

X_train_important = X_train[important_features]
X_test_important = X_test[important_features]
# Train the Random Forest
rf_most_important.fit(X_train_important, y_train)
# Make predictions using test set
predictions = rf_most_important.predict(X_test_important)
# Calculate the errors
errors = abs(predictions - y_test)
mse = mean_squared_error(y_test, predictions)
rmse = np.sqrt(mse)
accuracy = rf_most_important.score(X_test_important, y_test)


# Display performance metrics
print('*** Performance Metrics ***')
print('R Squared: {:.3f}'.format(accuracy))
print('MAE Score  :', round(np.mean(errors), 2))
print('MSE Score  : {:.3f}'.format(mse))
print('RMSE Score : {:.3f}'.format(rmse))<jupyter_output>*** Performance Metrics ***
R Squared: 0.680
MAE Score  : 0.29
MSE Score  : 0.334
RMSE Score : 0.578<jupyter_text>**Interpretation:** This tells us that our model perfom worse when we have fewer features to train on. By adding more features, our model does a better job at finding the relationships between our features and labels to accurately predict CPM.  Model Evaluation Function<jupyter_code>def weird_division(n, d):
    return n / d if d else 0

def evaluate(model, X_test, y_test):
    predictions = model.predict(X_test)
    errors = abs(predictions - y_test)
    accuracy = model.score(X_test, y_test)
    print('Model Performance')
    print('Average Error: {:0.4f}'.format(np.mean(errors)))
    print('R Squared: = {:0.2f}'.format(accuracy))
    
    return accuracy
# Base Model
base_model = RandomForestRegressor(n_estimators=10, random_state=42)
base_model.fit(X_train, y_train)
base_accuracy = evaluate(base_model, X_test, y_test)
# Random Search Best Model
best_random = rf_random.best_estimator_
random_accuracy = evaluate(best_random, X_test, y_test)
# Grid Search Best Model
best_grid = grid_search.best_estimator_
grid_accuracy = evaluate(best_grid, X_test, y_test)<jupyter_output><empty_output><jupyter_text>Model Comparison  Parking Lot  Alternative Models<jupyter_code>!pip install catboost -- quiet
!pip install gboost -- quiet
!pip install lightgbm -- quiet
from catboost import CatBoostRegressor
from xgboost import XGBRFRegressor
from lightgbm import LGBMRegressor
from sklearn.ensemble import RandomForestRegressor
models = {'LGBM': LGBMRegressor(objective='mse', n_jobs=-1),
          'CatBoost': CatBoostRegressor(objective='RMSE', verbose=0),
          'XGB': XGBRFRegressor(objective='reg:squarederror'),
          'RandomForest': RandomForestRegressor(criterion='mse', n_jobs=-1)
         }
for name, model in models.items():
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print("Model: {}, Score: {:.3f}".format(name, mean_squared_error(y_test, y_pred)))
for name, model in models.items():
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print("Model: {}, Score: {:.3f}".format(name, np.sqrt(mean_squared_error(y_test, y_pred))))<jupyter_output>Model: LGBM, Score: 0.571
Model: CatBoost, Score: 0.548
Model: XGB, Score: 0.826
Model: RandomForest, Score: 0.547<jupyter_text>Alternative Train/Test Split<jupyter_code>train_date = datetime.datetime(2019, 6, 22)
X_train = ads_df[ads_df['date'] < train_date][feature_list]
y_train = ads_df.loc[ads_df['date'] < train_date][target_col]
X_test = ads_df.loc[ads_df['date'] >= train_date][feature_list]
y_test = ads_df.loc[ads_df['date'] >= train_date][target_col]

print('***** TRAIN/TEST SPLIT *****')
print('TOTAL :', X.shape, y.shape)
print('TRAIN :', X_train.shape, y_train.shape)
print('TEST  :', X_test.shape, y_test.shape)
print('')
print('Ratio : {}'.format(round(X_train.shape[0]/(X_train.shape[0]+X_test.shape[0]),3)))<jupyter_output>***** TRAIN/TEST SPLIT *****
TOTAL : (538924, 10) (538924,)
TRAIN : (376973, 10) (376973,)
TEST  : (161951, 10) (161951,)

Ratio : 0.699<jupyter_text>Cross Validation<jupyter_code>cv = cross_validate(tree_reg, X, y, cv=5)

for i in range(len(cv['test_score'])):
  print("CV {} Score: {:.3f}".format(i, cv['test_score'][i]))

print('')
print("Mean Score: {:.3f}".format(cv['test_score'].mean()))<jupyter_output><empty_output><jupyter_text>Streamlit<jupyter_code>[ads_df['advertiser_id'].unique()]
ads_df.nunique()
ads_df['ad_unit_id'].sort_values(ascending=True).unique()
ads_df.info()
ads_df['order_id'].sort_values(ascending=True).unique()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09764996907854051, 'fraction_numerical': 0.018305504019789733, 'mean_word_length': 5.566700507614213, 'pattern_counts': {'":': 0, '<': 134, '<?xml version=': 0, '>': 133, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '363824', 'license': 'MIT', 'n_tokens_mistral': 10734, 'n_tokens_neox': 9979, 'n_words': 3301, 'path': 'ipynb/Real_time_Ad_Bidding_Price_Predictor_Models_V2_Inclass.ipynb', 'repo_name': 'ericttran3/rtb-price-predictor-app', 'size': '123200'}