<jupyter_start><jupyter_text>![](https://ga-dash.s3.amazonaws.com/production/assets/logo-9f88ae6c9c3871690e33280fcf557f33.png) Capstone Project---  [Capstone Project, Part 1: Proposal](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part1-proposal.ipynb) [Capstone Project, Part 2: Brief](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part2-brief.ipynb)- [Writing data to MongoDB](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part2-brief.ipynbmongo_db)- [Data Dictionary](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part2-brief.ipynbdata_dictionary)- [Map of races around the world](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part2-brief.ipynbworld-map) [Capstone Project, Part 3: Technical Notebook](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part3-technical-notebook.ipynb)- [Feature Engineering](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part3-technical-notebook.ipynbfeature_eng)- [Regression Approaches](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part3-technical-notebook.ipynbregression_approaches)- [Classification Approaches](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part3-technical-notebook.ipynbclassification_approaches)- [Feature Importance](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part3-technical-notebook.ipynbfeature_importance)- [Feature Selection](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part3-technical-notebook.ipynbfeature_selection)- [Models Comparison](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part3-technical-notebook.ipynbmodels_comparison) [Capstone Project, Part 4: Presentation](https://61c08c5e1627a3416b0c37b4--pensive-nobel-d54f9f.netlify.app/) [Capstone Project, Part 5: Appendix](https://nbviewer.org/github/jaeyow/f1-predictor/blob/main/final-project-part5-appendix.ipynb) [Capstone Project, Part 6: MLOps](https://github.com/jaeyow/f1-predictor/blob/main/.github/workflows/f1-mlops.yml)Using GitHub Actions as a cheap (and free) MLOps tool alternative: - invoke MLOps workflow on-demand (or with a cron schedule)- get latest source- setup Python build/MLOps environment- data retrieval and preparation- feature engineering- preparation for model training (including dummify categorical features)- feature selection- model building and scoring- setup serverless (lambda) API in AWS- deploy model to serverless API- profit!![](./images/f1-mclaren-car.png)<jupyter_code># install modules first
!pip install pandas
!pip install matplotlib
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from datetime import date
from datetime import datetime
from timeit import default_timer as timer
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import precision_score, explained_variance_score
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report
from sklearn.base import BaseEstimator
from sklearn.ensemble import AdaBoostRegressor
from sklearn.ensemble import BaggingRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import RidgeCV
from sklearn.linear_model import ARDRegression
from sklearn.svm import LinearSVR
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import StackingRegressor
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPRegressor
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.ensemble import StackingClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import LinearSVC
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import GridSearchCV
import pickle

import warnings
warnings.filterwarnings('ignore')

results_df = pd.read_csv('results_from_mongo.csv')
results_df.drop(columns=['Unnamed: 0'],inplace=True)

# only keep 2021 for the api
api_df = results_df[results_df['Season']==2021]
api_df.reset_index(drop=True, inplace=True)
api_df.to_csv(f'2021_races_drivers.csv')<jupyter_output><empty_output><jupyter_text>Feature EngineeringAt this stage, we will be creating the following features from exiting data:- Driver experience- Constructor experience- Driver Recent Wins- Driver Age- Driver Recent Form- Driver recent DNFs- Home Circuit advantage- Dummify the following categorical parameters: Season?, Race Name?, Number of Laps, Weather  Driver experienceDriver's experience in Formula 1, where a more experienced F1 driver typically places better than a rookie.<jupyter_code>results_df['DriverExperience'] = 0
drivers = results_df['Driver'].unique()
for driver in drivers:
    df_driver = pd.DataFrame(results_df[results_df['Driver']==driver]).tail(60) # Arbitrary number, just look at the last x races
    df_driver.loc[:,'DriverExperience'] = 1
    
    results_df.loc[results_df['Driver']==driver, "DriverExperience"] = df_driver['DriverExperience'].cumsum()
    results_df['DriverExperience'].fillna(value=0,inplace=True)<jupyter_output><empty_output><jupyter_text>Constructor ExperienceConstructor's experience in Formula 1, where a more experienced F1 constructor typically places better than a rookie.<jupyter_code>results_df['ConstructorExperience'] = 0
constructors = results_df['Constructor'].unique()
for constructor in constructors:
    
    df_constructor = pd.DataFrame(results_df[results_df['Constructor']==constructor]).tail(60) # Arbitrary number, just look at the last x races per driver
    df_constructor.loc[:,'ConstructorExperience'] = 1
    
    results_df.loc[results_df['Constructor']==constructor, "ConstructorExperience"] = df_constructor['ConstructorExperience'].cumsum()
    results_df['ConstructorExperience'].fillna(value=0,inplace=True)<jupyter_output><empty_output><jupyter_text>Driver Recent WinsA new feature is added to represent the dirver's most recent past wins. Excluding the result of the current race ensures that there is no possibility of data leakage that might affect the results.<jupyter_code>results_df['DriverRecentWins'] = 0
drivers = results_df['Driver'].unique()

results_df.loc[results_df['Position']==1, "DriverRecentWins"] = 1
for driver in drivers:
    mask_first_place_drivers = (results_df['Driver']==driver) & (results_df['Position']==1)
    df_driver = results_df[mask_first_place_drivers]
    results_df.loc[results_df['Driver']==driver, "DriverRecentWins"] = results_df[results_df['Driver']==driver]['DriverRecentWins'].rolling(60).sum() # 60 races, about 3 years rolling
    results_df.loc[mask_first_place_drivers, "DriverRecentWins"] = results_df[mask_first_place_drivers]['DriverRecentWins'] - 1  # but don't count this race's win
    results_df['DriverRecentWins'].fillna(value=0,inplace=True)<jupyter_output><empty_output><jupyter_text>Driver Recent DNFsA new feature has also been added to represent a driver's recent DNFs (Did Not Finish), whatever/whoever's fault it is. We also have to take care and avoid data leakage into this new feature, by not counting the current race.<jupyter_code>results_df['DriverRecentDNFs'] = 0
drivers = results_df['Driver'].unique()

results_df.loc[(~results_df['Status'].str.contains('Finished|\+')), "DriverRecentDNFs"] = 1
for driver in drivers:
    mask_not_finish_place_drivers = (results_df['Driver']==driver) & (~results_df['Status'].str.contains('Finished|\+'))
    df_driver = results_df[mask_not_finish_place_drivers]
    results_df.loc[results_df['Driver']==driver, "DriverRecentDNFs"] = results_df[results_df['Driver']==driver]['DriverRecentDNFs'].rolling(60).sum() # 60 races, about 3 years rolling
    results_df.loc[mask_not_finish_place_drivers, "DriverRecentDNFs"] = results_df[mask_not_finish_place_drivers]['DriverRecentDNFs'] - 1  # but don't count this race
    results_df['DriverRecentDNFs'].fillna(value=0,inplace=True)

# results_df[results_df['Driver']=='<NAME>'].tail(60)<jupyter_output><empty_output><jupyter_text>Fix issues with Recent form valuesIn Formula 1, only the top 10 finishers score points, so even if a driver finished 11th, they will not score anything which will not help our calculation. So in this part, we give all finishers a score. The 1st place top points, and lower places get lower points and so on. We can then use this column as a variable (instead of F1's official points) to calclulate for the the Driver's recent form.<jupyter_code># Add new RFPoints column - ALL finishers score points - max points First place and one less for each lesser place (using LogSpace)
seasons = results_df['Season'].unique()
results_df['RFPoints'] = 0
for season in seasons:
    rounds = results_df[results_df['Season']==season]['Round'].unique()
    for round in rounds:
        mask = (results_df['Season']==season) & (results_df['Round']==round)
        finisher_mask = ((results_df['Status'].str.contains('Finished|\+'))) # Count only if finished the race
        finished_count = results_df.loc[(mask) & finisher_mask, "RFPoints"].count()
        point_list = np.round(np.logspace(1,4,40, base=4),4) # use list of LogSpaced numbers
        point_list[::-1].sort()
        
        results_df.loc[(mask) & finisher_mask, "RFPoints"] = point_list[:finished_count].tolist()<jupyter_output><empty_output><jupyter_text>Driver Recent FormNow that we've got our adjusted points system "RFPoints", we can now calculate for a more accurate Driver Recent Form. We also have to take care and avoid data leakage into this new feature.<jupyter_code>results_df['DriverRecentForm'] = 0
# for all drivers, calculate the rolling X DriverRecentForm and add to a new column in 
# original data frame, this represents the 'recent form', then for NA's just impute to zero
drivers = results_df['Driver'].unique()
for driver in drivers:
    df_driver = results_df[results_df['Driver']==driver]
    results_df.loc[results_df['Driver']==driver, "DriverRecentForm"] = df_driver['RFPoints'].rolling(30).sum() - df_driver['RFPoints'] # calcluate recent form points but don't include this race's points
    results_df['DriverRecentForm'].fillna(value=0,inplace=True)<jupyter_output><empty_output><jupyter_text>Constructor Recent FormNow that we've got our adjusted points system "RFPoints", we can now also calculate for a more accurate Constructor Recent Form. We also have to take care and avoid data leakage into this new feature.<jupyter_code>results_df['ConstructorRecentForm'] = 0
# for all constructors, calculate the rolling X RFPoints and add to a new column in 
# original data frame, this represents the 'recent form', then for NA's just impute to zero
constructors = results_df['Constructor'].unique()
for constructor in constructors:
    df_constructor = results_df[results_df['Constructor']==constructor]
    results_df.loc[results_df['Constructor']==constructor, "ConstructorRecentForm"] = df_constructor['RFPoints'].rolling(30).sum() - df_constructor['RFPoints'] # calcluate recent form points but don't include this race's points
    results_df['ConstructorRecentForm'].fillna(value=0,inplace=True)<jupyter_output><empty_output><jupyter_text>Driver AgeSurely a driver's age has some effect and may have some influence to the outcome of the race.<jupyter_code>def calculate_age(born, race):
    date_born = datetime.strptime(born,'%Y-%m-%d')
    date_race = datetime.strptime(race,'%Y-%m-%d')
    return date_race.year - date_born.year - ((date_race.month, date_race.day) < (date_born.month, date_born.day))

results_df['Age'] = results_df.apply(lambda x: calculate_age(x['DOB'], x['Race Date']), axis=1)<jupyter_output><empty_output><jupyter_text>Home CircuitIs there such a thing as Homecourt Advantage in Formula 1 racing? It doesn't look like it does, based on the preliminary EDA, however, I've got a feeling that it might have some. In the following cell, I have created a mapping between driver nationality vs race country, and this is used when we want to convey the Homecourt advantage concept in this model.<jupyter_code>def is_race_in_home_country(driver_nationality, race_country):
    nationality_country_map = {
        'American': ['USA'],
        'American-Italian': ['USA','Italy'],
        'Argentine': ['Argentina'],
        'Argentine-Italian': ['Argentina','Italy'],
        'Australian': ['Australia'],
        'Austrian': ['Austria'],
        'Belgian': ['Belgium'],
        'Brazilian': ['Brazil'],
        'British': ['UK'],
        'Canadian': ['Canada'],
        'Chilean': ['Brazil'],
        'Colombian': ['Brazil'],
        'Czech': ['Austria','Germany'],
        'Danish': ['Germany'],
        'Dutch': ['Netherlands'],
        'East German': ['Germany'],
        'Finnish': ['Germany','Austria'],
        'French': ['France'],
        'German': ['Germany'],
        'Hungarian': ['Hungary'],
        'Indian': ['India'],
        'Indonesian': ['Singapore','Malaysia'],
        'Irish': ['UK'],
        'Italian': ['Italy'],
        'Japanese': ['Japan','Korea'],
        'Liechtensteiner': ['Switzerland','Austria'],
        'Malaysian': ['Malaysia','Singapore'],
        'Mexican': ['Mexico'],
        'Monegasque': ['Monaco'],
        'New Zealander': ['Australia'],
        'Polish': ['Germany'],
        'Portuguese': ['Portugal'],
        'Rhodesian': ['South Africa'],
        'Russian': ['Russia'],
        'South African': ['South Africa'],
        'Spanish': ['Spain','Morocco'],
        'Swedish': ['Sweden'],
        'Swiss': ['Switzerland'],
        'Thai': ['Malaysia'],
        'Uruguayan': ['Argentina'],
        'Venezuelan': ['Brazil']
    }
    
    countries = ['None']
    
    try:
      countries = nationality_country_map[driver_nationality]
    except:
      print("An exception occurred, This driver has no race held in his home country.")
    return race_country in countries

results_df['IsHomeCountry'] = results_df.apply(lambda x: is_race_in_home_country(x['Nationality'], x['Country']), axis=1)<jupyter_output><empty_output><jupyter_text>Handle all categorical variablesDummify applicable categorical variables and ensure that the variables for the model are all numeric.- Weather- Race name (circuit)- Driver nationality  Dummify FTWDummify the following parameters, and just drop irrelevant columns<jupyter_code># Weather
results_df = pd.get_dummies(results_df, columns = ['Weather', 'Nationality', 'Race Name'],drop_first=True)

for col in results_df.columns:
    if 'Nationality' in col and results_df[col].sum() < 300:
        results_df.drop(col, axis = 1, inplace = True)
        
    elif 'Race Name' in col and results_df[col].sum() < 130:
        results_df.drop(col, axis = 1, inplace = True)
        
    else:
        pass
results_df.columns.tolist()
#### Drop Columns which are not needed/required for modelling
results_df.drop(['Race Date', 'Race Time', 'Status', 'DOB', 'Constructor', 'Constructor Nat', 'Circuit Name',
                 'Race Url', 'Lat', 'Long', 'Locality', 'Country','Laps','Points',
                 'RFPoints'], axis=1, inplace=True)
results_df.shape

results_df['Season'] = pd.to_numeric(results_df['Season'])
results_df.columns
scoring_raw ={'model':[], 'params': [], 'score': [], 'train_time': [], 'test_time': []}
from sklearn.preprocessing import MinMaxScaler
# Prepare train set
np.set_printoptions(precision=4)
model_df = results_df.copy()
model_df['Position'] = model_df['Position'].map(lambda x: 1 if x == 1 else 0)

train = model_df[(model_df['Season'] >= 1950) & (model_df['Season'] < 2021)]
X_train = train.drop(['Position','Driver'], axis = 1)
y_train = train['Position']

scaler = MinMaxScaler()
X_train = pd.DataFrame(scaler.fit_transform(X_train, y_train), columns = X_train.columns)

model_df.columns<jupyter_output><empty_output><jupyter_text>Regression Function<jupyter_code>def regression_test_score(model, print_output=False):
    # --- Test ---
    score = 0
    races = model_df[(model_df['Season'] == 2021)]['Round'].unique()
    for race in races:
        test = model_df[(model_df['Season'] == 2021) & (model_df['Round'] == race)]
        X_test = test.drop(['Position','Driver'], axis = 1)
        y_test = test['Position']
        X_test = pd.DataFrame(scaler.transform(X_test), columns = X_test.columns)
        X_test.to_csv(f'{2021}_{race}.csv')

        # make predictions
        prediction_df = pd.DataFrame(model.predict(X_test), columns = ['prediction'])
        merged_df = pd.concat([prediction_df, test[['Driver','Position']].reset_index(drop=True)], axis=1)
        merged_df.rename(columns = {'Position': 'actual_pos'}, inplace = True)
        
        # shuffle data to remove original order that will influence selection
        # of race winner when there are drivers with identical win probablilities
        merged_df = merged_df.sample(frac=1).reset_index(drop=True) 
        merged_df.sort_values(by='prediction', ascending=False, inplace=True)
        merged_df['predicted_pos'] = merged_df['prediction'].map(lambda x: 0)
        merged_df.iloc[0, merged_df.columns.get_loc('predicted_pos')] = 1
        merged_df.reset_index(drop=True, inplace=True)
        if (print_output == True):
            print(merged_df)

        # --- Score --- 
        score += precision_score(merged_df['actual_pos'], merged_df['predicted_pos'], zero_division=0)
        
    return score / len(races)<jupyter_output><empty_output><jupyter_text>Dumb ClassifierI have created a a dumb classifier and all it does is create a list of numbers between 0.001 and 0.99 which represents the order of finishing, and shuffle for good measure<jupyter_code>class F1OracleDumbClassifier(BaseEstimator):
    def fit(self, X, y=None):
        pass
    def predict(self, X):
        # numbers between 0.001 to 0.99 and shuffle - jose's awesome dumb classifier!
        numbers = np.round(np.logspace(np.log10(0.001),np.log10(0.999),len(X)),4)
        np.random.shuffle(numbers)
        return numbers
def joses_dumb_classifier(X_train, y_train):
    start = timer()
    model = F1OracleDumbClassifier()
    model.fit(X_train, y_train)
    end = timer()
    scoring_raw['train_time'].append(end - start)

    start = timer()
    model_score = regression_test_score(model)
    end = timer()

    scoring_raw['model'].append("Jose's Dumb Classifier")
    scoring_raw['score'].append(np.round(model_score*100, 3))
    scoring_raw['params'].append('none')
    scoring_raw['test_time'].append(end - start)
        
joses_dumb_classifier(X_train, y_train)
running_score = pd.DataFrame(scoring_raw)
running_score<jupyter_output><empty_output><jupyter_text>Feature Importance using Linear Regression CoefficientsUsing Linear Regression, we can identify the most relevant features, so that we may be able to remove the features that don't add any value, but bloat.<jupyter_code>def feature_importance_using_linear_regression(X_train, y_train):
    model = LinearRegression()
    model.fit(X_train, y_train)

    # Assess the importance of features using Linear Regression coefficients
    importance = model.coef_
    importance_df = pd.DataFrame(importance, columns = ['importance'])
    features_df = pd.DataFrame(results_df.columns.tolist(), columns = ['feature_name'])
    features_df.drop(features_df[(features_df['feature_name'] == 'Driver') | (features_df['feature_name'] == 'Position')].index, inplace=True)
    merged_features_df = pd.concat([importance_df, features_df.reset_index(drop=True)], axis=1)
    merged_features_df.sort_values(by='importance', ascending=True, inplace=True)
    merged_features_df.set_index('feature_name', inplace=True)
    selected_features_df = merged_features_df[(merged_features_df['importance'] > 0)][['importance']]

    # plot feature importance 
    axis = selected_features_df.plot(kind='barh', title="F1 Prediction Feature Importance", figsize=(16, 10), color='#00CC99')
    y_label = axis.yaxis.get_label()
    y_label.set_visible(False)

feature_importance_using_linear_regression(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Feature importance using Random Forest<jupyter_code>def feature_importance_using_random_forest(X_train, y_train):
    sel = SelectFromModel(RandomForestClassifier(n_estimators = 100))
    sel.fit(X_train, y_train)
    features_to_keep = sel.get_support()
    feature_importance_df = pd.DataFrame(
        {'feature_name': X_train.columns.tolist(),
         'imp_mask': sel.get_support(),
         'importance': sel.estimator_.feature_importances_
        })
    return feature_importance_df, features_to_keep
    
feature_importance_df, features_to_keep = feature_importance_using_random_forest(X_train, y_train)
# feature_importance_df = feature_importance_df[feature_importance_df['imp_mask']==True]
feature_importance_df.sort_values(by='importance', inplace=True)
feature_importance_df.reset_index(drop=True, inplace=True)
feature_importance_df.set_index('feature_name', inplace=True)

# plot feature importance 
axis = feature_importance_df.plot(kind='barh', title="F1 Prediction Feature Importance", figsize=(16, 20), color='#00CC99')
y_label = axis.yaxis.get_label()
y_label.set_visible(False)
feature_importance_df = feature_importance_df.nlargest(50, 'importance')
feature_importance_df<jupyter_output><empty_output><jupyter_text>Feature SelectionBased on the feature importance discovered in the previous section, the identified insignificant features can be safely ommitted as it contributes to model bloat. I've decided to use the features identified by **Random Forest**, since I did not trust the ones from **Linear Regression**.<jupyter_code>def keep_significant_features():
    # keep significant features (Position and Driver will be dropped later, we still need them)
    return results_df[feature_importance_df.index.tolist() + ['Position','Driver']]
results_df = keep_significant_features()
results_df.columns
results_df.shape
from sklearn.preprocessing import MinMaxScaler
# Prepare train set
np.set_printoptions(precision=4)
model_df = results_df.copy()
model_df['Position'] = model_df['Position'].map(lambda x: 1 if x == 1 else 0)

train = model_df[(model_df['Season'] >= 1950) & (model_df['Season'] < 2021)]
X_train = train.drop(['Position','Driver'], axis = 1)
y_train = train['Position']

scaler = MinMaxScaler()
X_train = pd.DataFrame(scaler.fit_transform(X_train, y_train), columns = X_train.columns)
X_train.shape<jupyter_output><empty_output><jupyter_text>Regression ApproachesThe problem of predicting the winner of a race can be considered a regression problem. We do this by submitting the independent variables (20 drivers) to the chosen machine learning algorithm, and allow the algorithm to make a winner prediction. In fact, the estimator does not only pick a winner among the 20 drivers. It responds with a "prediction" column, where it is sorted descending order, and can be used as the predicted finishing order of the race. The driver that ends up at the top of this list is the winner of this race. The following Regression techniques are applied to help solve our problem:- Linear Regression- Ada Boost Regressor- Bagging Regressor- Extra Trees Regressor- Gradient Boosting Regressor- Random Forest Regressor- Stacking Regressor- Neural Network (MLP Regressor)  Linear Regression<jupyter_code>def linear_regression(X_train, y_train):
    params={'fit_intercept': ['True', 'False']}

    for fit_intercept in params['fit_intercept']:
        start = timer()
        model_params = (fit_intercept)
        model = LinearRegression(fit_intercept = fit_intercept)
        model.fit(X_train, y_train)
        end = timer()
        scoring_raw['train_time'].append(end - start)

        start = timer()
        model_score = regression_test_score(model)
        end = timer()

        scoring_raw['model'].append('Linear Regression')
        scoring_raw['score'].append(np.round(model_score*100, 3))
        scoring_raw['params'].append(model_params)
        scoring_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>AdaBoost Regressor<jupyter_code>def adaboost_regressor(X_train, y_train):
    params={'n_estimators': [100,200,300],
            'learning_rate': [0.001,0.01,0.1,1],
            'loss': ['linear','square','exponential']}

    for n_estimators in params['n_estimators']:
        for learning_rate in params['learning_rate']:
            for loss in params['loss']:
                start = timer()
                model_params = (n_estimators, learning_rate, loss)
                model = AdaBoostRegressor(random_state=0, n_estimators=n_estimators, learning_rate=learning_rate, loss=loss)
                model.fit(X_train, y_train)
                end = timer()
                scoring_raw['train_time'].append(end - start)

                start = timer()
                model_score = regression_test_score(model)
                end = timer()

                scoring_raw['model'].append('AdaBoost Regressor')
                scoring_raw['score'].append(np.round(model_score*100, 3))
                scoring_raw['params'].append(model_params)
                scoring_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>Bagging Regressor<jupyter_code>def bagging_regressor(X_train, y_train):
    params={'n_estimators': [100,200,300],
            'max_samples': [10,20,30],
            'max_features': [20,40,50],
            'bootstrap': [True,False],
            'bootstrap_features': [True,False]}

    for n_estimators in params['n_estimators']:
        for max_samples in params['max_samples']:
            for max_features in params['max_features']:
                for bootstrap in params['bootstrap']:
                    for bootstrap_features in params['bootstrap_features']:
                        start = timer()
                        model_params = (n_estimators, max_samples, max_features, bootstrap, bootstrap_features)
                        model = BaggingRegressor(random_state=0, base_estimator=DecisionTreeRegressor(),
                             n_estimators=n_estimators, max_samples=max_samples, max_features=max_features, bootstrap=bootstrap, bootstrap_features=bootstrap_features)
                        model.fit(X_train, y_train)
                        end = timer()
                        scoring_raw['train_time'].append(end - start)

                        start = timer()
                        model_score = regression_test_score(model)
                        end = timer()

                        scoring_raw['model'].append('Bagging Regressor (DT)')
                        scoring_raw['score'].append(np.round(model_score*100, 3))
                        scoring_raw['params'].append(model_params)
                        scoring_raw['test_time'].append(end - start)
        
bagging_regressor(X_train, y_train)
running_score = pd.DataFrame(scoring_raw)
running_score.score.value_counts()
running_score.head(50)<jupyter_output><empty_output><jupyter_text>ExtraTrees Regressor<jupyter_code>def extratrees_regressor(X_train, y_train):
    params={'n_estimators': [100,200,300],
            'max_depth': [10],
            'min_samples_split': [2,4,6],
            'min_samples_leaf': [1,3,5],
            'max_features': ['auto','sqrt','log2']}

    for n_estimators in params['n_estimators']:
            for max_depth in params['max_depth']:
                for min_samples_split in params['min_samples_split']:
                    for min_samples_leaf in params['min_samples_leaf']:
                        for max_features in params['max_features']:
                            start = timer()
                            model_params = (n_estimators, max_depth, min_samples_split, min_samples_leaf, max_features)
                            model = ExtraTreesRegressor(random_state=0, n_estimators=n_estimators,
                                                        max_depth=max_depth, min_samples_split=min_samples_split,
                                                        min_samples_leaf=min_samples_leaf, max_features=max_features)
                            model.fit(X_train, y_train)
                            end = timer()
                            scoring_raw['train_time'].append(end - start)

                            start = timer()
                            model_score = regression_test_score(model)
                            end = timer()

                            scoring_raw['model'].append('Extra Trees Regressor')
                            scoring_raw['score'].append(np.round(model_score*100, 3))
                            scoring_raw['params'].append(model_params)
                            scoring_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>Gradient Boosting Regressor<jupyter_code>def gradientboosting_regressor(X_train, y_train):
    params={'n_estimators': [100,200,300],
            'learning_rate': [0.001,0.01,0.1,1],
            'subsample': [0.001,0.1,1],
            'max_depth': [5,10,20]}

    for n_estimators in params['n_estimators']:
            for learning_rate in params['learning_rate']:
                for subsample in params['subsample']:
                        for max_depth in params['max_depth']:
                            start = timer()
                            model_params = (n_estimators, learning_rate, subsample, max_depth)
                            model = GradientBoostingRegressor(random_state=0, n_estimators=n_estimators,
                                                        learning_rate=learning_rate, subsample=subsample,
                                                         max_depth=max_depth)
                            model.fit(X_train, y_train)
                            end = timer()
                            scoring_raw['train_time'].append(end - start)

                            start = timer()
                            model_score = regression_test_score(model)
                            end = timer()

                            scoring_raw['model'].append('Gradient Boosting Regressor')
                            scoring_raw['score'].append(np.round(model_score*100, 3))
                            scoring_raw['params'].append(model_params)
                            scoring_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>Random Forest Regressor<jupyter_code>def random_forest_regressor(X_train, y_train):
    params={'n_estimators': [100,200,300],
            'max_depth': [10],
            'min_samples_split': [2,4,6],
            'min_samples_leaf': [1,3,5],
            'max_features': ['auto','sqrt','log2']}

    for n_estimators in params['n_estimators']:
        for max_depth in params['max_depth']:
            for min_samples_split in params['min_samples_split']:
                for min_samples_leaf in params['min_samples_leaf']:
                    for max_features in params['max_features']:
                        start = timer()
                        model_params = (n_estimators, max_depth, min_samples_split, min_samples_leaf, max_features)
                        model = RandomForestRegressor(random_state=0, n_estimators=n_estimators, max_depth=max_depth,
                                                     min_samples_split=min_samples_split, min_samples_leaf=min_samples_leaf,
                                                     max_features=max_features)
                        model.fit(X_train, y_train)
                        end = timer()
                        scoring_raw['train_time'].append(end - start)

                        start = timer()
                        model_score = regression_test_score(model)
                        end = timer()

                        scoring_raw['model'].append('Random Forest Regressor')
                        scoring_raw['score'].append(np.round(model_score*100, 3))
                        scoring_raw['params'].append(model_params)
                        scoring_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>Stacking Regressor<jupyter_code>def stacking_regressor(X_train, y_train):
    params={'n_estimators': [100,200,300],
            'max_depth': [10]}

    for n_estimators in params['n_estimators']:
        for max_depth in params['max_depth']:
            start = timer()
            model_params = (n_estimators, max_depth)
            estimators = [('lr', RidgeCV()),('adr', ARDRegression())]
            model = StackingRegressor(estimators=estimators,
                                      final_estimator=RandomForestRegressor(random_state=0, n_estimators=n_estimators, max_depth=max_depth))
            model.fit(X_train, y_train)
            end = timer()
            scoring_raw['train_time'].append(end - start)

            start = timer()
            model_score = regression_test_score(model)
            end = timer()

            scoring_raw['model'].append('Stacking Regressor (RF)')
            scoring_raw['score'].append(np.round(model_score*100, 3))
            scoring_raw['params'].append(model_params)
            scoring_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>Neural Network - MLP Regressor<jupyter_code>def mlp_regressor(X_train, y_train):
    params={
        'hidden_layer_sizes': [(80,20,40,5), (75,30,50,10,3)], 
        'activation': ['identity', 'relu','logistic',], 
        'solver': ['lbfgs','sgd', 'adam'], 
        'alpha': np.logspace(-4,1,10)}

    for hidden_layer_sizes in params['hidden_layer_sizes']:
        for activation in params['activation']:
            for solver in params['solver']:
                for alpha in params['alpha']:
                    start = timer()
                    model_params = (hidden_layer_sizes, activation, solver, alpha)
                    model = MLPRegressor(random_state=1, max_iter=500, hidden_layer_sizes=hidden_layer_sizes,
                                        activation=activation, solver=solver, alpha=alpha)
                    model.fit(X_train, y_train)
                    end = timer()
                    scoring_raw['train_time'].append(end - start)

                    start = timer()
                    model_score = regression_test_score(model)
                    end = timer()

                    scoring_raw['model'].append('Neural Network - MLP Regressor')
                    scoring_raw['score'].append(np.round(model_score*100, 3))
                    scoring_raw['params'].append(model_params)
                    scoring_raw['test_time'].append(end - start)
        

scoring_raw ={'model':[], 'params': [], 'score': [], 'train_time': [], 'test_time': []}
# run all Regressors + the Dumb Classifier
regressors = [('Dumb Classifier', joses_dumb_classifier),
              ('Linear Regression', linear_regression),
              ('AdaBoost Regressor', adaboost_regressor),
              ('Bagging Regressor', bagging_regressor),
              ('Gradient Boosting Regressor', gradientboosting_regressor),
              ('Random Forest Regressor', random_forest_regressor),
              ('Stacking Regressor (RF)', stacking_regressor),
              ('Neural Network (MLP Regressor)', mlp_regressor)]

start = timer()
for regressor in regressors:
    start_reg = timer()
    print(f'Running {regressor[0]}')
    regressor[1](X_train, y_train)
    end_reg = timer()
    print(f'{regressor[0]} done => {np.round(end_reg - start_reg, 4)}s.')
end = timer()

print(f'Running all regressors: {np.round(end - start, 4)}s')
running_score = pd.DataFrame(scoring_raw)
running_score

running_score.to_csv('regressions_scores.csv')<jupyter_output>Running Dumb Classifier
Dumb Classifier done => 0.1877s.
Running Linear Regression
Linear Regression done => 0.4183s.
Running AdaBoost Regressor
AdaBoost Regressor done => 332.6744s.
Running Bagging Regressor
Bagging Regressor done => 134.8787s.
Running Gradient Boosting Regressor
Gradient Boosting Regressor done => 1111.0537s.
Running Random Forest Regressor
Random Forest Regressor done => 585.9226s.
Running Stacking Regressor (RF)
Stacking Regressor (RF) done => 16.3893s.
Running Neural Network (MLP Regressor)
Neural Network (MLP Regressor) done => 1744.2518s.
Running all regressors: 3925.7784s<jupyter_text>Classification ApproachesThis problem can also be approached as a classification problem since we want to predict 2 categories - the race winner, and then everybody else. Like the Regression problem, we will need to submit the details for the 20 competitors, wherein the classification algorithm will predict if the driver WILL WIN, or NOT WIN. Because this produces 2 columns of probabilities for all competitors, we can then simply sort this, and the one with the highest probability of winning will be our winner. The following classification models were used:- Logistic Regression- Random Forest Classifier- SVM Classifier- Ada Boost Classifier- Extra Trees  Classifier- Gradient Boosting- Stacking Classifier- Neural Networks (MLP Classifier)  Classification Function<jupyter_code>from sklearn.preprocessing import MinMaxScaler

def classification_test_score(model):
    # --- Test ---
    score = 0
    races = model_df[(model_df['Season'] == 2021)]['Round'].unique()
    for race in races:
        test = model_df[(model_df['Season'] == 2021) & (model_df['Round'] == race)]
        X_test = test.drop(['Position','Driver'], axis = 1)
        y_test = test['Position']
        X_test = pd.DataFrame(scaler.transform(X_test), columns = X_test.columns)

        # make predictions
        prediction_df = pd.DataFrame(model.predict_proba(X_test), columns = ['proba_not_win', 'proba_win'])
        merged_df = pd.concat([prediction_df, test[['Driver','Position']].reset_index(drop=True)], axis=1)
        merged_df.rename(columns = {'Position': 'actual_pos'}, inplace = True)
        
        # shuffle data to remove original order that will influence selection
        # of race winner when there are drivers with identical win probablilities
        merged_df = merged_df.sample(frac=1).reset_index(drop=True)
        merged_df.sort_values(by='proba_win', ascending=False, inplace=True)
        merged_df['predicted_pos'] = 0
        merged_df.iloc[0, merged_df.columns.get_loc('predicted_pos')] = 1
        merged_df.reset_index(drop=True, inplace=True)
        
        # --- Score ---
        score += precision_score(merged_df['actual_pos'], merged_df['predicted_pos'], zero_division=0)

    return score / len(races)

scoring_clf_raw ={'model':[], 'params': [], 'score': [], 'train_time': [], 'test_time': []}<jupyter_output><empty_output><jupyter_text>Logistic RegressionI have tried out using [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html?highlight=gridsearchcvsklearn.model_selection.GridSearchCV) however, the precision score was a bit lower than the manual search, so I reverted to the manual method.<jupyter_code>def logistic_regression(X_train, y_train):
    params={'penalty': ['l1', 'l2'],
            'solver': ['saga', 'liblinear'],
            'C': np.logspace(-3,1,20)}

    for penalty in params['penalty']:
        for solver in params['solver']:
            for C in params['C']:
                start = timer()
                model_params = (penalty, solver, C)
                model = LogisticRegression(penalty=penalty, solver=solver, C=C, max_iter = 10000)
                model.fit(X_train, y_train)
                end = timer()
                scoring_clf_raw['train_time'].append(end - start)

                start = timer()
                model_score = classification_test_score(model)
                end = timer()

                scoring_clf_raw['model'].append('Logistic Regression')
                scoring_clf_raw['score'].append(np.round(model_score*100, 3))
                scoring_clf_raw['params'].append(model_params)
                scoring_clf_raw['test_time'].append(end - start)

def logistic_regression_grid(X_train, y_train):
    params={'penalty': ['l1', 'l2'],
            'solver': ['saga', 'liblinear'],
            'C': np.logspace(-3,1,20)}
    
    start = timer()
    model = GridSearchCV(LogisticRegression(max_iter = 10000), params, cv=2)
    model.fit(X_train, y_train)
    end = timer()
    scoring_clf_raw['train_time'].append(end - start)
        
    start = timer()
    model_score = classification_test_score(model)
    end = timer()
    scoring_clf_raw['model'].append('Logistic Regression')
    scoring_clf_raw['score'].append(np.round(model_score*100, 3))
    scoring_clf_raw['params'].append(model.best_params_)
    scoring_clf_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>Random Forest ClassifierLike in Logistic Regression, I have tried out using [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html?highlight=gridsearchcvsklearn.model_selection.GridSearchCV) however, the precision score was a bit lower than the manual search, so I reverted to the manual method.<jupyter_code>def random_forest_classifier(X_train, y_train):
    params={'n_estimators': [100,200,300],
            'max_depth': [10],
            'min_samples_split': [2,4,6],
            'min_samples_leaf': [1,3,5],
            'max_features': ['auto','sqrt','log2']}

    for n_estimators in params['n_estimators']:
        for max_depth in params['max_depth']:
            for min_samples_split in params['min_samples_split']:
                for min_samples_leaf in params['min_samples_leaf']:
                    for max_features in params['max_features']:
                        start = timer()
                        model_params = (n_estimators, max_depth, min_samples_split, min_samples_leaf, max_features)
                        model = RandomForestClassifier(random_state=0, n_estimators=n_estimators, max_depth=max_depth,
                                                     min_samples_split=min_samples_split, min_samples_leaf=min_samples_leaf,
                                                     max_features=max_features)
                        model.fit(X_train, y_train)
                        end = timer()
                        scoring_clf_raw['train_time'].append(end - start)

                        start = timer()
                        model_score = classification_test_score(model)
                        end = timer()

                        scoring_clf_raw['model'].append('Random Forest Classifier')
                        scoring_clf_raw['score'].append(np.round(model_score*100, 3))
                        scoring_clf_raw['params'].append(model_params)
                        scoring_clf_raw['test_time'].append(end - start)
        

def random_forest_classifier_grid(X_train, y_train):
    params={'n_estimators': [100,200,300],
            'max_depth': [10],
            'min_samples_split': [2,4,6],
            'min_samples_leaf': [1,3,5],
            'max_features': ['auto','sqrt','log2']}
    
    start = timer()
    model = GridSearchCV(RandomForestClassifier(random_state=0), params)
    model.fit(X_train, y_train)
    end = timer()
    scoring_clf_raw['train_time'].append(end - start)
        
    start = timer()
    model_score = classification_test_score(model)
    end = timer()
    scoring_clf_raw['model'].append('Random Forest Classifier')
    scoring_clf_raw['score'].append(np.round(model_score*100, 3))
    scoring_clf_raw['params'].append(model.best_params_)
    scoring_clf_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>SVM Classifier<jupyter_code>def svm_classifier(X_train, y_train):
    params={
        'gamma': np.logspace(-4, -1, 3),
        'C': np.logspace(-2, 1, 3),
        'kernel': ['linear']}
    
    for gamma in params['gamma']:
        for C in params['C']:
            for kernel in params['kernel']:
                start = timer()
                model_params = (gamma, C, kernel)
                model = SVC(probability = True, gamma=gamma, C=C, kernel=kernel)
                model.fit(X_train, y_train)
                end = timer()
                scoring_clf_raw['train_time'].append(end - start)

                start = timer()
                model_score = classification_test_score(model)
                end = timer()

                scoring_clf_raw['model'].append('SVM Classifier')
                scoring_clf_raw['score'].append(np.round(model_score*100, 3))
                scoring_clf_raw['params'].append(model_params)
                scoring_clf_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>AdaBoost Classifier<jupyter_code>def adaboost_classifier(X_train, y_train):
    params={
        'n_estimators': [100,200,300],
        'learning_rate': [0.001,0.01,0.1,1],
        'algorithm': ['SAMME', 'SAMME.R']}
    
    for n_estimators in params['n_estimators']:
        for learning_rate in params['learning_rate']:
            for algorithm in params['algorithm']:
                start = timer()
                model_params = (n_estimators, learning_rate, algorithm)
                model = AdaBoostClassifier(random_state=0, n_estimators=n_estimators, 
                                          learning_rate=learning_rate, algorithm=algorithm)
                model.fit(X_train, y_train)
                end = timer()
                scoring_clf_raw['train_time'].append(end - start)

                start = timer()
                model_score = classification_test_score(model)
                end = timer()

                scoring_clf_raw['model'].append('Ada Boost Classifier')
                scoring_clf_raw['score'].append(np.round(model_score*100, 3))
                scoring_clf_raw['params'].append(model_params)
                scoring_clf_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>ExtraTrees Classifier<jupyter_code>def extratrees_classifier(X_train, y_train):
    params={
        'n_estimators': [100,200,300],
        'max_depth': [10],
        'min_samples_split': [2,4,6],
        'min_samples_leaf': [1,3,5],
        'max_features': ['auto','sqrt','log2']}
    
    for n_estimators in params['n_estimators']:
        for max_depth in params['max_depth']:
            for min_samples_split in params['min_samples_split']:
                for min_samples_leaf in params['min_samples_leaf']:
                    for max_features in params['max_features']:
                        start = timer()
                        model_params = (n_estimators, max_depth, min_samples_split, min_samples_leaf, max_features)
                        model = ExtraTreesClassifier(random_state=0, n_estimators=n_estimators,
                                                     max_depth=max_depth, min_samples_split=min_samples_split, min_samples_leaf=min_samples_leaf,
                                                     max_features=max_features)
                        model.fit(X_train, y_train)
                        end = timer()
                        scoring_clf_raw['train_time'].append(end - start)

                        start = timer()
                        model_score = classification_test_score(model)
                        end = timer()

                        scoring_clf_raw['model'].append('Extra Trees Classifier')
                        scoring_clf_raw['score'].append(np.round(model_score*100, 3))
                        scoring_clf_raw['params'].append(model_params)
                        scoring_clf_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>GradientBoosting Classifier<jupyter_code>def gradient_boosting_classifier(X_train, y_train):
    params={
        'n_estimators': [100,200,300],
        'learning_rate': [0.001,0.01,0.1,1],
        'subsample': [0.001,0.1,1],
        'max_depth': [5,10,20]}
    
    for n_estimators in params['n_estimators']:
        for learning_rate in params['learning_rate']:
            for subsample in params['subsample']:
                for max_depth in params['max_depth']:
                    start = timer()
                    model_params = (n_estimators, learning_rate, subsample, max_depth)
                    model = GradientBoostingClassifier(random_state=0, n_estimators=n_estimators,
                                                 learning_rate=learning_rate, subsample=subsample, max_depth=max_depth)
                    model.fit(X_train, y_train)
                    end = timer()
                    scoring_clf_raw['train_time'].append(end - start)

                    start = timer()
                    model_score = classification_test_score(model)
                    end = timer()

                    scoring_clf_raw['model'].append('Gradient Boosting Classifier')
                    scoring_clf_raw['score'].append(np.round(model_score*100, 3))
                    scoring_clf_raw['params'].append(model_params)
                    scoring_clf_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>Stacking Classifier<jupyter_code>def stacking_classifier(X_train, y_train):
    params={
        'n_estimators': [100,200,300]}
    
    for n_estimators in params['n_estimators']:
        start = timer()
        model_params = (n_estimators)
        estimators = [('rf', RandomForestClassifier(n_estimators=n_estimators, random_state=42)),
                  ('svr', LinearSVC(random_state=45))]

        model = StackingClassifier(estimators=estimators, final_estimator=LogisticRegression())
        model.fit(X_train, y_train)
        end = timer()
        scoring_clf_raw['train_time'].append(end - start)

        start = timer()
        model_score = classification_test_score(model)
        end = timer()

        scoring_clf_raw['model'].append('Stacking Classifier')
        scoring_clf_raw['score'].append(np.round(model_score*100, 3))
        scoring_clf_raw['params'].append(model_params)
        scoring_clf_raw['test_time'].append(end - start)<jupyter_output><empty_output><jupyter_text>Neural Network (MLP Classifier)<jupyter_code>def mlp_classifier(X_train, y_train):
    params={
        'hidden_layer_sizes': [(80,20,40,5), (75,30,50,10,3)], 
        'activation': ['identity', 'relu','logistic',], 
        'solver': ['lbfgs','sgd', 'adam'], 
        'alpha': np.logspace(-4,1,5)}
    
    for hidden_layer_sizes in params['hidden_layer_sizes']:
        for activation in params['activation']:
            for solver in params['solver']:
                for alpha in params['alpha']:
                    start = timer()
                    model_params = (hidden_layer_sizes, activation, solver, alpha)
                    model = MLPClassifier(hidden_layer_sizes=hidden_layer_sizes, activation=activation, solver=solver, alpha=alpha, max_iter=20)
                    model.fit(X_train, y_train)
                    end = timer()
                    scoring_clf_raw['train_time'].append(end - start)

                    start = timer()
                    model_score = classification_test_score(model)
                    end = timer()

                    scoring_clf_raw['model'].append('Neural Network (MLP Classifier)')
                    scoring_clf_raw['score'].append(np.round(model_score*100, 3))
                    scoring_clf_raw['params'].append(model_params)
                    scoring_clf_raw['test_time'].append(end - start)
        
# run all Classifiers (more than 8 hours!)
classifiers = [
    ('Logistic Regression', logistic_regression),
    ('Random Forest Classifier', random_forest_classifier),
    ('Neural Network (MLP Classifier)', mlp_classifier),              
    ('AdaBoost Classifier', adaboost_classifier),
    ('Extra Trees Classifier', extratrees_classifier),
    ('Gradient Boosting Classifier', gradient_boosting_classifier),
    ('Stacking Classifier', stacking_classifier),
    # ('SVM Classifier', svm_classifier) => removed as takes a few hours, and with similar results as other models
]

start = timer()
for classifier in classifiers:
    start_clf = timer()
    print(f'Running {classifier[0]}')
    classifier[1](X_train, y_train)
    end_clf = timer()
    print(f'{classifier[0]} done => {np.round(end_clf - start_clf, 4)}s.')
end = timer()

print(f'Running all classifiers: {np.round(end - start, 4)}s')
running_score = pd.DataFrame(scoring_clf_raw)

running_score.to_csv('classification_scores.csv')<jupyter_output>Running Logistic Regression
Logistic Regression done => 35.1231s.
Running Random Forest Classifier
Random Forest Classifier done => 250.337s.
Running Neural Network (MLP Classifier)
Neural Network (MLP Classifier) done => 235.9248s.
Running AdaBoost Classifier
AdaBoost Classifier done => 109.1923s.
Running Extra Trees Classifier
Extra Trees Classifier done => 200.5558s.
Running Gradient Boosting Classifier
Gradient Boosting Classifier done => 1413.9461s.
Running Stacking Classifier
Stacking Classifier done => 62.2184s.
Running all classifiers: 2307.3023s<jupyter_text>Comparing all Machine Learning models used<jupyter_code>import matplotlib.ticker as mtick

def read_csv_dataframe(filename):
    summary = pd.read_csv(filename)
    summary.drop(columns=['Unnamed: 0'],inplace=True)
    groups = summary.groupby(by=['model'])
    summary = groups.apply(lambda x: x[x['score'] == x['score'].max()].iloc[0])
    summary.sort_values(by='score', inplace=True)
    summary.set_index('model', inplace=True)
    return summary

summary_df = read_csv_dataframe('regressions_scores.csv')
summary_clf_df = read_csv_dataframe('classification_scores.csv')

# merge
merged_scores_df = pd.concat([summary_df, summary_clf_df])
merged_scores_df.sort_values(by='score', inplace=True)
merged_scores_df

# plot feature importance 
axis = feature_importance_df.plot(kind='barh', title="F1 Prediction Feature Importance", figsize=(16, 4), color='#05AFF2', grid=True,)
y_label = axis.yaxis.get_label()
y_label.set_visible(False)

# plot precision scores
axis = merged_scores_df[['score']].plot(kind='barh', figsize=(16,10), color=['#00CC99','#9370DB','#CA98BF'], grid=True, linestyle='--')
axis.xaxis.set_major_formatter(mtick.PercentFormatter())
axis.set_title('F1 Prediction Precision Scores', pad=20)
y_label = axis.yaxis.get_label()
y_label.set_visible(False)
merged_scores_df.sort_values(by='score', inplace=True, ascending=False)
merged_scores_df<jupyter_output><empty_output><jupyter_text>ConclusionFollowing our [Champion/Challenger](https://medium.com/@awaiskaleem/mlflow-tips-n-tricks-eb1ac013edd1) model, the winning model is the one built with [Bagging Regressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html) ([Decision Tree Regressor](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html?highlight=decisiontreesklearn.tree.DecisionTreeRegressor)). This is an ensemble model fron SKLearn which uses DecistionTreeRegressor as its base estimator.[Neural Network - MLP Regressor](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html?highlight=mlpsklearn.neural_network.MLPRegressor) ended up with the same precision score, however because it took 10x more in training time, I have delegated it to 2nd place. - **Bagging Regressor (Decision Tree Regressor)**- Train time: **0.97s**- Test time: **0.42s**- **Grid**, **ConstructorRecentForm**, **DriverRecentForm** and **DriverRecentWins** round out the most significant features. The last 3 are one of the features we engineered for this project.- Precision score: **61.9%**- For each race, the finishing order is also available, but omitted here for brevity- Correctly predicting the winner in **13 out of 21** races in 2021 Season  Preparing for Rest API<jupyter_code>def bagging_regressor_pickle(X_train, y_train):
    # now use the winning paramters to build the model
    model = BaggingRegressor(random_state=0, base_estimator=DecisionTreeRegressor(),
         n_estimators=200, max_samples=10, max_features=50, bootstrap=True, bootstrap_features=True)
    model.fit(X_train, y_train)
    pickle.dump(model, open('f1-model.pkl', 'wb'))
    print(model.n_features_in_)
    

start = timer()
bagging_regressor_pickle(X_train, y_train)
end = timer()
print(f'bagging_regressor_pickle took {end - start} s')
start = timer()
from_pickled_model = pickle.load(open('f1-model.pkl', 'rb'))
for i in range(0, 10):
    f1_model_score = regression_test_score(from_pickled_model)
    print(f1_model_score)
end = timer()
print(f'bagging_regressor_inference took {end - start} s')
results_df.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1143496973302075, 'fraction_numerical': 0.015360929353721265, 'mean_word_length': 3.4454036397573495, 'pattern_counts': {'":': 0, '<': 145, '<?xml version=': 0, '>': 161, 'https://': 22, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '477912', 'license': 'MIT', 'n_tokens_mistral': 17989, 'n_tokens_neox': 16874, 'n_words': 3646, 'path': 'final-project-part3-technical-notebook.ipynb', 'repo_name': 'jaeyow/f1-predictor', 'size': '57168'}