<jupyter_start><jupyter_text>The Models---In an attempt to recreate the same models I executed back in December 2020, I will be utilizing the same packages in order to understand what my aim was for the project.This is to improve the final results I had before to see if my problem statement still holds true.<jupyter_code># Import Packages
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

pd.set_option('display.max_columns', None)

from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC, LinearSVC
from sklearn.ensemble import RandomForestClassifier

from xgboost import XGBRFClassifier, XGBClassifier
from sklearn.metrics import plot_confusion_matrix, f1_score, roc_curve

from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.model_selection import RandomizedSearchCV, train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score, f1_score, classification_report

import pickle
df = pd.read_csv('train_test_2021.csv')
# Check to make sure this is the data set that I want
df.head()
df['type'].value_counts(normalize=True)
# Since type is a confusing column name, we will rename it to 'result'
df.rename(columns={'type':'result'}, inplace=True)
df.head()
# We will turn the stand, p_throws, and result into binary columns
# This is so that we can include a batter's stance, a pitcher's stance
# and result of the play as something numeric
make_binary = ['stand','p_throws','result']
pd.get_dummies(df, columns=make_binary, drop_first=True).head()

# This shows that righties (R) are 1 while lefties (L) are 0
# As well as strikes (S) being represented as 1 and balls (B) as 0
# Change DF and rename columns
df = pd.get_dummies(df, columns=make_binary, drop_first=True)
df.rename(columns={'stand_R':'batter_stance','p_throws_R':'pitchers_throwing_hand','result_S':'result'},inplace=True)
df.head()
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 357046 entries, 0 to 357045
Data columns (total 18 columns):
 #   Column                  Non-Null Count   Dtype  
---  ------                  --------------   -----  
 0   release_speed           357046 non-null  float64
 1   release_pos_x           357046 non-null  float64
 2   release_pos_z           357046 non-null  float64
 3   pfx_x                   357046 non-null  float64
 4   pfx_z                   357046 non-null  float64
 5   plate_x                 357046 non-null  float64
 6   plate_z                 357046 non-null  float64
 7   vx0                     357046 non-null  float64
 8   vy0                     357046 non-null  float64
 9   vz0                     357046 non-null  float64
 10  ax                      357046 non-null  float64
 11  ay                      357046 non-null  float64
 12  az                      357046 non-null  float64
 13  release_spin_rate       357046 non-null  float64
 14  pitch_name          [...]<jupyter_text>Since we have `pitch_name` as a qualitative column for data, we will do two models, one where the name of the pitch is known and the other, without, just the pitch mechanics.<jupyter_code># let's make a copy of df as df2, 
# this will be df2 will not have the pitch names included
df2 = df.copy()
df2 = df2.loc[:, df2.columns != 'pitch_name']
df2.head()
# Let's dummify pitch name for the original dataframe
df = pd.get_dummies(df, columns=['pitch_name'], prefix='pitch', drop_first=True)<jupyter_output><empty_output><jupyter_text>Let's ask some questionsCan we predict the call of a baseball pitch thrown based on ____?<jupyter_code>print(df.columns)
print(df2.columns)<jupyter_output>Index(['release_speed', 'release_pos_x', 'release_pos_z', 'pfx_x', 'pfx_z',
       'plate_x', 'plate_z', 'vx0', 'vy0', 'vz0', 'ax', 'ay', 'az',
       'release_spin_rate', 'batter_stance', 'pitchers_throwing_hand',
       'result', 'pitch_Changeup', 'pitch_Curveball', 'pitch_Cutter',
       'pitch_Eephus', 'pitch_Fastball', 'pitch_Knuckle Curve',
       'pitch_Knuckleball', 'pitch_Sinker', 'pitch_Slider',
       'pitch_Split-Finger'],
      dtype='object')
Index(['release_speed', 'release_pos_x', 'release_pos_z', 'pfx_x', 'pfx_z',
       'plate_x', 'plate_z', 'vx0', 'vy0', 'vz0', 'ax', 'ay', 'az',
       'release_spin_rate', 'batter_stance', 'pitchers_throwing_hand',
       'result'],
      dtype='object')<jupyter_text>We will start with `df2` and focus on the mechanics and physics of the pitch.<jupyter_code>X = df2.iloc[:,:-1] # A matrix containing features we want
X.head()
y = df2.result # A vector we are aiming for
# Use train test split to split X and y into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=66)
# Check the splits are correct
X_train.shape, X_test.shape, y_train.shape, y_test.shape
# Since the project and presentation used RandomForest, we will start with that

# We will even use the parameters from the previous models
params = ({
    'n_estimators':[100,150,200],
    'max_depth':[2,3,4,5],
    'max_features':['log2','auto']
})
rf = RandomizedSearchCV(RandomForestClassifier(),
                        param_distributions=params,
                        n_jobs=8,
                        verbose=2,
                        random_state=66)
rf.fit(X_train, y_train)
rf.score(X_train, y_train)
rf.score(X_test, y_test)
rf.best_params_
# Check accuracy of our test but why are they the same?
preds = rf.predict(X_test)
accuracy_score(y_test, preds)<jupyter_output><empty_output><jupyter_text>Let's ensure that we have good accuracy of the model by looking at the **Null Accuracy**.<jupyter_code>print(y_test.value_counts()); y_test.mean() 
# The mean will give a mean of the percentage of 1's
# calculate the percentage of zeros
1 - y_test.mean()
max(y_test.mean(), 1 - y_test.mean())<jupyter_output><empty_output><jupyter_text>The dumb model would always predict 0 and be 68% correct.- The accuracy on the RandomForest model is better than a dumb model- We have overachieved the minimum  Now, A Random Forest but on a GridSearch<jupyter_code>rf2 = GridSearchCV(RandomForestClassifier(random_state=66),
                       param_grid=params,
                       n_jobs=8,
                       verbose=2)
rf2.fit(X_train, y_train)
print(f'Train score: {rf2.score(X_train, y_train)}')
print(f'Test score: {rf2.score(X_test, y_test)}')
rf2.best_params_
preds2 = rf2.predict(X_test)
accuracy_score(y_test, preds2)<jupyter_output><empty_output><jupyter_text>Our accuracy on this model is around the same as our initial model. With that being said, it is also better than our null model meaning that one or more of the features prove influence on the result of the pitch.  Confusion Matrix of our RandomForestClassifiers---Looking at the confusion matricies below, it shows that the false positives and false negatives were small in comparison to the model prediciting a strike and/or ball call. That being said, it seems that during the 2021 MLB season, just looking at the location and understanding the speed of a pitch, how a pitcher throws, and the stance of a batter does in someway determine the influence of a call.<jupyter_code>plot_confusion_matrix(rf, X_test, y_test)
plt.show()
plot_confusion_matrix(rf2, X_test, y_test)
plt.show()<jupyter_output><empty_output><jupyter_text>Next, run the models that have the pitch name includedaka we will use `df` dataframe containing the categorical variable "pitch name" the have been dummified.<jupyter_code>X2 = df.loc[:, df.columns!='result']
X2.head()
X2_train, X2_test, y2_train, y2_test = train_test_split(X2, y, random_state=66)
rf3 = RandomizedSearchCV(RandomForestClassifier(),
                        param_distributions=params,
                        n_jobs=8,
                        verbose=2,
                        random_state=66)
rf3.fit(X2_train, y2_train)
print(f'Train score: {rf3.score(X2_train, y2_train)}')
print(f'Test score: {rf3.score(X2_test, y2_test)}')
rf3.best_params_
preds3 = rf3.predict(X2_test)
accuracy_score(y2_test, preds3)<jupyter_output><empty_output><jupyter_text>A reminder of our null accuracy: `0.6706549259483319`.---Looking at the results, it seems our accuracy has dropped once we include the name of the pitch. It is kind of difficult to predict a pitch as it is being thrown, normally that is something you assess once the pitch reaches the catcher's glove but nonetheless knowing that, it does not beat the initial model. However comparing it to the past models, it does by significant amount.<jupyter_code>rf4 = GridSearchCV(RandomForestClassifier(random_state=66),
                       param_grid=params,
                       n_jobs=8,
                       verbose=2)
rf4.fit(X2_train, y2_train)
print(f'Train score: {rf4.score(X2_train, y2_train)}')
print(f'Test score: {rf4.score(X2_test, y2_test)}')
rf4.best_params_
preds4 = rf4.predict(X2_test)
accuracy_score(y2_test, preds4)<jupyter_output><empty_output><jupyter_text>The accuracy of the models stayed around 88%, better than null but doesn't beat the original. Knowing what type of pitch is coming doesn't help in determining strike vs ball call. But this could explain how an umpire determines a pitch thrown. Some times it is hard to determine a strike or ball if the pitch thrown isn't always a fast ball.<jupyter_code>plot_confusion_matrix(rf3, X2_test, y2_test)
plt.show()
plot_confusion_matrix(rf4, X2_test, y2_test)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09176346356916579, 'fraction_numerical': 0.03157338965153115, 'mean_word_length': 3.856923076923077, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '644411', 'license': 'MIT', 'n_tokens_mistral': 3208, 'n_tokens_neox': 2981, 'n_words': 983, 'path': 'v3/recreate_models.ipynb', 'repo_name': 'laternader/situation_baseball', 'size': '9470'}