<jupyter_start><jupyter_text>Loading, cleaning, and merging the data<jupyter_code>pbp = utils.get_data("pbp")
weather = utils.get_data("weather")
pbp = utils.clean_data("pbp", pbp)
weather = utils.clean_data("weather", weather)
FullData = pbp.merge(weather, how = 'inner', on = ('game_date', 'home_team'))
FullData<jupyter_output><empty_output><jupyter_text>Finding the amounts of made and missed field goals, and calculating the percentage of made field goals in the data<jupyter_code>FullData.loc[:, "field_goal_result"].value_counts()
4122/(759+4122)<jupyter_output><empty_output><jupyter_text>Checking for null values<jupyter_code>FullData.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 4881 entries, 0 to 4880
Data columns (total 14 columns):
 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   posteam                 4881 non-null   object        
 1   game_date               4881 non-null   datetime64[ns]
 2   half_seconds_remaining  4881 non-null   float64       
 3   game_half               4881 non-null   object        
 4   field_goal_result       4881 non-null   object        
 5   kick_distance           4881 non-null   float64       
 6   score_differential      4881 non-null   float64       
 7   kicker_player_name      4881 non-null   object        
 8   kicker_player_id        4881 non-null   object        
 9   home_team               4881 non-null   object        
 10  stadium                 4881 non-null   object        
 11  weather_temperature     4881 non-null   float64       
 12  weather_wind_mph        4881 non-null   float64 [...]<jupyter_text>Seperating the predictors and the target and using LabelEncoder to encode the target<jupyter_code>from sklearn.preprocessing import LabelEncoder
LabEnc = LabelEncoder()
LabEnc.fit(FullData.loc[:, "field_goal_result"])
y = LabEnc.transform(FullData.loc[:, "field_goal_result"])
X = FullData[["posteam", "half_seconds_remaining", "game_half", "kick_distance", "score_differential", "kicker_player_name", "stadium", "weather_temperature", "weather_wind_mph", "weather_detail"]]
#After encoding, a result of made is given the value of 0, while a result of missed is given the value of 1
y<jupyter_output><empty_output><jupyter_text>Creating Train and Test Splits<jupyter_code>from sklearn.model_selection import train_test_split
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size = 0.2, random_state = 52594, stratify = y)<jupyter_output><empty_output><jupyter_text>Finding the number of made and missed field goals in the training data, as well as the percentage of field goals made<jupyter_code>import numpy as np
np.unique(ytrain, return_counts = True)
3297/(3297 + 607)<jupyter_output><empty_output><jupyter_text>Encoding the training data with OneHotEncoding<jupyter_code>CatVar = ['posteam', 'game_half', 'kicker_player_name', 'stadium', 'weather_detail']
OtherVar = ['half_seconds_remaining', "kick_distance", "score_differential", "weather_temperature", "weather_wind_mph"]
cat = utils.get_cat(CatVar, X)#Gets the possible values for everything in the FullFrame so encoding doesn't result in different column numbers
enc_Xtrain = utils.ohe(CatVar, OtherVar, Xtrain, cat)
enc_Xtest = utils.ohe(CatVar, OtherVar, Xtest, cat)<jupyter_output><empty_output><jupyter_text>Scaling the data<jupyter_code>from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaler.fit(enc_Xtrain)
scaled_enc_Xtrain = scaler.transform(enc_Xtrain)
scaled_enc_Xtest = scaler.transform(enc_Xtest)
scaled_enc_Xtrain = pd.DataFrame(scaled_enc_Xtrain, columns = enc_Xtrain.columns)
scaled_enc_Xtrain<jupyter_output><empty_output><jupyter_text>Building a base logistic regression model<jupyter_code>from sklearn.linear_model import LogisticRegression
BaseLogReg = utils.cross_validate(scaled_enc_Xtrain, ytrain, "LogReg", penalty = 'none', score = 'f1')
#The f1 score per cross validation test. The scores are low and highly variable

BaseLogReg
utils.get_average_score(BaseLogReg)
#Repeating for classification accuracy. It looks decent, but it is lower than predicting win on every kick

BaseLogReg = utils.cross_validate(scaled_enc_Xtrain, ytrain, "LogReg", penalty = 'none')
BaseLogReg
utils.get_average_score(BaseLogReg)
#Confusion matrix for this model on the test data
BaseLogRegFit = LogisticRegression(penalty = 'none', max_iter = 10000)
BaseLogRegFit.fit(scaled_enc_Xtrain, ytrain)

ypred = BaseLogRegFit.predict(scaled_enc_Xtrain)

from sklearn.metrics import confusion_matrix

matrix = confusion_matrix(ytrain, ypred)

matrix
from sklearn.metrics import f1_score

f1_score(ytrain, ypred)
BaseLogRegFit.score(scaled_enc_Xtrain, ytrain)<jupyter_output><empty_output><jupyter_text>Grid search for LogReg<jupyter_code>from sklearn.model_selection import GridSearchCV
grid = {'C': [0.0001, 0.001, 0.01, 0.1, 1, 10, 100], 'penalty': ['l1', 'l2']}
LogRegGrid = LogisticRegression(solver = 'saga', max_iter=10000)
GridSearch = GridSearchCV(estimator = LogRegGrid, param_grid = grid, cv = 10, return_train_score = True, verbose = 2, scoring = 'f1')
GridSearch.fit(scaled_enc_Xtrain, ytrain)
pd.DataFrame(GridSearch.cv_results_).sort_values('mean_test_score', ascending = False).T
#Confusion matrix for the winning model from the grid search
GridLogRegFit = LogisticRegression(C = 10, penalty = 'l2', max_iter = 10000)
GridLogRegFit.fit(scaled_enc_Xtrain, ytrain)

ypred = GridLogRegFit.predict(scaled_enc_Xtrain)

from sklearn.metrics import confusion_matrix

matrix = confusion_matrix(ytrain, ypred)

matrix
#This model is almost identical to the base one, performing slightly worse on the confusion matrix
f1_score(ytrain, ypred)
GridLogRegFit.score(scaled_enc_Xtrain, ytrain)<jupyter_output><empty_output><jupyter_text>Installing imbalanced learn for oversampling<jupyter_code>pip install -U imbalanced-learn<jupyter_output>Requirement already up-to-date: imbalanced-learn in c:\users\michael\anaconda3\lib\site-packages (0.7.0)
Requirement already satisfied, skipping upgrade: scipy>=0.19.1 in c:\users\michael\anaconda3\lib\site-packages (from imbalanced-learn) (1.5.0)
Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in c:\users\michael\anaconda3\lib\site-packages (from imbalanced-learn) (1.18.5)
Requirement already satisfied, skipping upgrade: joblib>=0.11 in c:\users\michael\anaconda3\lib\site-packages (from imbalanced-learn) (0.16.0)
Requirement already satisfied, skipping upgrade: scikit-learn>=0.23 in c:\users\michael\anaconda3\lib\site-packages (from imbalanced-learn) (0.23.1)
Requirement already satisfied, skipping upgrade: threadpoolctl>=2.0.0 in c:\users\michael\anaconda3\lib\site-packages (from scikit-learn>=0.23->imbalanced-learn) (2.1.0)
Note: you may need to restart the kernel to use updated packages.<jupyter_text>Importing random over sampler<jupyter_code>from imblearn.over_sampling import RandomOverSampler<jupyter_output><empty_output><jupyter_text>Using RandomOverSampler to over sample the data to improve the f1 score<jupyter_code>#Using f1 as the scorer. The values used for this were determined by experimentation, looking for the best combination of f1 and classification accuracy across the cross validations

ROSLogReg = utils.cross_validate(scaled_enc_Xtrain, ytrain, "LogReg", penalty = 'l1', solver = 'saga', oversample = 'ros', ss = .6, score = 'f1')
utils.get_average_score(ROSLogReg)
ROSLogReg
#Redoing the above to get the classification accuracy score

ROSLogReg = utils.cross_validate(scaled_enc_Xtrain, ytrain, "LogReg", penalty = 'l1', solver = 'saga', oversample = 'ros', CatVar = CatVar, ss = .6, score = 'acc')
utils.get_average_score(ROSLogReg)
ROSLogReg
#Applying the ROS to the testing data

ros = RandomOverSampler(random_state = 52594, sampling_strategy = .6)
rosX, rosy = ros.fit_resample(scaled_enc_Xtrain, ytrain)
rosLogReg = LogisticRegression(penalty = 'l1', solver = 'saga', max_iter = 10000).fit(rosX, rosy)
ypred = rosLogReg.predict(scaled_enc_Xtrain)

f1_score(ytrain, ypred)
rosLogReg.score(scaled_enc_Xtrain, ytrain)
matrix = confusion_matrix(ytrain, ypred)

matrix<jupyter_output><empty_output><jupyter_text>Making a decision tree model using grid search<jupyter_code>from sklearn.tree import DecisionTreeClassifier
DecisionTree = DecisionTreeClassifier(max_leaf_nodes = 100, min_samples_leaf= 200)
DecisionTree.fit(enc_Xtrain, ytrain)
from sklearn.model_selection import cross_validate
dtcv = cross_validate(DecisionTree, scaled_enc_Xtrain, ytrain, cv= 10, return_train_score = True, return_estimator = True, verbose = 2, scoring = 'f1')
#This suggests the model is simply predicting "make" for every kick, which will not work for the purposes of this project

dtcv['test_score']<jupyter_output><empty_output><jupyter_text>Using grid search for Decision Tree<jupyter_code>dtgrid = [{'max_depth':[1,2,3,4,5,6,7,8,9,10,15,20,25,30], 'max_features' :[1,2,3,4,5,6,7,8,9,10,15,20,25,30]}]

gridsearchdt = GridSearchCV(estimator = DecisionTreeClassifier(), param_grid = dtgrid, scoring = 'f1', cv = 10, verbose = 2)

griddt = gridsearchdt.fit(enc_Xtrain, ytrain)
pd.DataFrame(gridsearchdt.cv_results_).sort_values('mean_test_score', ascending = False).T<jupyter_output><empty_output><jupyter_text>Using Random Over Sample on Decision Tree Model<jupyter_code>#Using f1 as the scorer. The values used for this were determined by experimentation, looking for the best combination of f1 and classification accuracy across the cross validations

ROSDecisionTree = utils.cross_validate(scaled_enc_Xtrain, ytrain, "Tree", penalty = 'l1', solver = 'saga', oversample = 'ros', ss = .6, score = 'f1')
ROSDecisionTree
utils.get_average_score(ROSDecisionTree)
#Using f1 as the scorer. The values used for this were determined by experimentation, looking for the best combination of f1 and classification accuracy across the cross validations

ROSDecisionTree = utils.cross_validate(scaled_enc_Xtrain, ytrain, "Tree", penalty = 'l1', solver = 'saga', oversample = 'ros', ss = .6, score = 'acc')
ROSDecisionTree
utils.get_average_score(ROSDecisionTree)<jupyter_output><empty_output><jupyter_text>Building Random Forest Model<jupyter_code>from sklearn.ensemble import RandomForestClassifier
RanFor = RandomForestClassifier(n_estimators = 700, max_features = 'auto', max_depth = 20)
rfcv = cross_validate(RanFor, scaled_enc_Xtrain, ytrain, cv= 10, return_train_score = True, return_estimator = True, verbose = 2, scoring = "f1")
#These scores are very low

rfcv['test_score']<jupyter_output><empty_output><jupyter_text>Using a grid search for a random forest model<jupyter_code>rfcgrid = [{'max_depth':[1,2,3,4,5,6,7,8,9,10,15,20,25,30], 'max_features' :[1,2,3,4,5,6,7,8,9,10,15,20,25,30]}]

gridsearchrfc = GridSearchCV(estimator = RandomForestClassifier(), param_grid = rfcgrid, scoring = 'f1', cv = 10, verbose = 2)

gridRFC = gridsearchrfc.fit(enc_Xtrain, ytrain)
pd.DataFrame(gridsearchrfc.cv_results_).sort_values('mean_test_score', ascending = False).T<jupyter_output><empty_output><jupyter_text>Using Random Over Sampler for Random Forest<jupyter_code>#Using f1 as the scorer. The values used for this were determined by experimentation, looking for the best combination of f1 and classification accuracy across the cross validations

ROSRandomForest = utils.cross_validate(scaled_enc_Xtrain, ytrain, "RF", penalty = 'l1', solver = 'saga', oversample = 'ros', ss = 1.0, score = 'f1')
ROSRandomForest
utils.get_average_score(ROSRandomForest)
#Again for accuracy

ROSRandomForest = utils.cross_validate(scaled_enc_Xtrain, ytrain, "RF", penalty = 'l1', solver = 'saga', oversample = 'ros', ss = 1.0, score = 'acc')
ROSRandomForest
utils.get_average_score(ROSRandomForest)<jupyter_output><empty_output><jupyter_text>The best performing model is the over sampled linear regression model  Fitting the model and using it on the test data<jupyter_code>ros = RandomOverSampler(random_state = 52594, sampling_strategy = .6)
rosX, rosy = ros.fit_resample(scaled_enc_Xtrain, ytrain)
rosLogReg = LogisticRegression(penalty = 'l1', solver = 'saga', max_iter = 10000).fit(rosX, rosy)
ypred = rosLogReg.predict(scaled_enc_Xtest)

f1_score(ytest, ypred)
rosLogReg.score(scaled_enc_Xtest, ytest)
matrix = confusion_matrix(ytest, ypred)
matrix<jupyter_output><empty_output><jupyter_text>Predicting 4 field goals from an NFL game (Ravens @ Eagles, 10/18/2020)  Prediction 1, actual result: made<jupyter_code>Predict1 = pd.DataFrame([['BAL', 732, 'Half1', 46.0, 14, "J.Tucker", "Lincoln Financial Field", 65, 7, "Normal"]], columns = X.columns)
enc_Predict1 = utils.ohe(CatVar, OtherVar, Predict1, cat)
scaled_enc_Predict1 = scaler.transform(enc_Predict1)
rosLogReg.predict_proba(scaled_enc_Predict1)<jupyter_output><empty_output><jupyter_text>Predicting the same field goal, but if it was the other team and other kicker<jupyter_code>Predict2 = pd.DataFrame([['PHI', 732, 'Half1', 46.0, 14, "J.Elliott", "Lincoln Financial Field", 65, 7, "Normal"]], columns = X.columns)
enc_Predict2 = utils.ohe(CatVar, OtherVar, Predict2, cat)
scaled_enc_Predict2 = scaler.transform(enc_Predict2)
rosLogReg.predict_proba(scaled_enc_Predict2)<jupyter_output><empty_output><jupyter_text>While the percentages are suspect, it is clear that the model strongly accounts for the kicker and the kicking team  Prediction 2, actual result: missed<jupyter_code>Predict3 = pd.DataFrame([['PHI', 0, 'Half1', 52.0, -17, "J.Elliott", "Lincoln Financial Field", 65, 7, "Normal"]], columns = X.columns)
enc_Predict3 = utils.ohe(CatVar, OtherVar, Predict3, cat)
scaled_enc_Predict3 = scaler.transform(enc_Predict3)
rosLogReg.predict_proba(scaled_enc_Predict3)<jupyter_output><empty_output><jupyter_text>Predicting the same field goal, but other team other kicker<jupyter_code>Predict4 = pd.DataFrame([['BAL', 0, 'Half1', 52.0, -17, "J.Tucker", "Lincoln Financial Field", 65, 7, "Normal"]], columns = X.columns)
enc_Predict4 = utils.ohe(CatVar, OtherVar, Predict4, cat)
scaled_enc_Predict4 = scaler.transform(enc_Predict4)
rosLogReg.predict_proba(scaled_enc_Predict4)<jupyter_output><empty_output><jupyter_text>Prediction 3, actual result: made<jupyter_code>Predict5 = pd.DataFrame([['BAL', 601, 'Half2', 55.0, 10, "J.Tucker", "Lincoln Financial Field", 65, 7, "Normal"]], columns = X.columns)
enc_Predict5 = utils.ohe(CatVar, OtherVar, Predict5, cat)
scaled_enc_Predict5 = scaler.transform(enc_Predict5)
rosLogReg.predict_proba(scaled_enc_Predict5)<jupyter_output><empty_output><jupyter_text>Predicting the same field goal, but other team, other kicker<jupyter_code>Predict6 = pd.DataFrame([['PHI', 601, 'Half2', 55.0, 10, "J.Elliott", "Lincoln Financial Field", 65, 7, "Normal"]], columns = X.columns)
enc_Predict6 = utils.ohe(CatVar, OtherVar, Predict6, cat)
scaled_enc_Predict6 = scaler.transform(enc_Predict6)
rosLogReg.predict_proba(scaled_enc_Predict6)<jupyter_output><empty_output><jupyter_text>Prediction 4, actual result: made<jupyter_code>Predict7 = pd.DataFrame([['BAL', 437, 'Half2', 46.0, 13, "J.Tucker", "Lincoln Financial Field", 65, 7, "Normal"]], columns = X.columns)
enc_Predict7 = utils.ohe(CatVar, OtherVar, Predict7, cat)
scaled_enc_Predict7 = scaler.transform(enc_Predict7)
rosLogReg.predict_proba(scaled_enc_Predict7)<jupyter_output><empty_output><jupyter_text>Predicting the same field goal, but other team, other kicker<jupyter_code>Predict8 = pd.DataFrame([['PHI', 437, 'Half2', 46.0, 13, "J.Elliott", "Lincoln Financial Field", 65, 7, "Normal"]], columns = X.columns)
enc_Predict8 = utils.ohe(CatVar, OtherVar, Predict8, cat)
scaled_enc_Predict8 = scaler.transform(enc_Predict8)
rosLogReg.predict_proba(scaled_enc_Predict8)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10860564367157144, 'fraction_numerical': 0.033632715459583414, 'mean_word_length': 6.451352634076887, 'pattern_counts': {'":': 0, '<': 112, '<?xml version=': 0, '>': 119, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '334005', 'license': 'MIT', 'n_tokens_mistral': 5781, 'n_tokens_neox': 5369, 'n_words': 1292, 'path': 'Notebooks/Model Building.ipynb', 'repo_name': 'mbrown-amc/Predicting-made-miss-percentages-of-individual-NFL-field-goal-attempts', 'size': '15699'}