<jupyter_start><jupyter_text>ModelingWith the data successfully obtained and wrangled, we can finally put it into some models!<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
df = pd.read_csv('training-data.csv')

print(df.shape)
df.head()
df['Year'].describe() # The year range is accurate--recall that I don't want to touch the 2019 data until the very end.
# I'll take a quick look at my categorical data.

df.describe(exclude=np.number)<jupyter_output><empty_output><jupyter_text>Since the goal is to predict each player's assists for 2019-2020, it makes the most sense to me to do a train/test split on "Year," with the test being the latest year in the training data (i.e. 2018).To avoid over-fitting to the test set, I'll also make a val set. I tried various train/val splits, and putting it at around 2014 ended up being for the best.<jupyter_code>test = df[df['Year'] == 2018]
temp = df[df['Year'] < 2018]
train = temp[temp['Year'] < 2014]
val = temp[temp['Year'] >= 2014]<jupyter_output><empty_output><jupyter_text>Calculating the baselineWhile determining the baseline prediction can be somewhat subjective, in this case I think it makes the most sense to have the baseline be the *previous* year's assists total. In other words, if I want to guess how many assists a player will have this year, a good first pass is to say that they'll have just as many as they did last year.Thankfully, the Assist column already provides us with this information.<jupyter_code>from sklearn.metrics import mean_squared_error

baseline = val['AST']

baseline_mse = mean_squared_error(baseline, val['Target'])

baseline_mse

# The baseline's mean squared error is quite low, so it might be difficult for my models to beat it, at least by much.<jupyter_output><empty_output><jupyter_text>Linear Models<jupyter_code># One-hot encoding is the best way to deal with categorical variables in linear models.
# "PLayer" and "Team" have high cardinality, and I don't think either is very meaningful, so I'll just
# remove them beforehand. I also have to drop "Target" from the input matrix, obviously.

non_features = ['Player', 'Tm', 'Target']
X_train = train.drop(non_features, axis=1)
y_train = train['Target']
X_val = val.drop(non_features, axis=1)
y_val = val['Target']

# After determining my hyperparameters, I'll want to use all the data I have available to predict the test data.
# So along with the above, I'm also going to make "X_train_full" dataframes with everything before 2018 included.
X_train_full = temp.drop(non_features, axis=1)
y_train_full = temp['Target']
X_test = test.drop(non_features, axis=1)
y_test = test['Target']
import category_encoders as ce

encoder = ce.OneHotEncoder(use_cat_names=True)

X_train_encoded = encoder.fit_transform(X_train)
X_val_encoded = encoder.transform(X_val)

X_train_full_encoded = encoder.fit_transform(X_train_full)
X_test_encoded = encoder.transform(X_test)
# First is a basic linear regression. I'll also use Select K Best to determine the optimal number of features.

from sklearn.feature_selection import f_regression, SelectKBest
from sklearn.linear_model import LinearRegression

lowest = [0.9, 1] # This will help me keep track of which k value is best.
for k in range(1, len(X_train_encoded.columns)+1):
    print(f'{k} features')
    
    selector = SelectKBest(score_func=f_regression, k=k)
    X_train_selected = selector.fit_transform(X_train_encoded, y_train)
    X_val_selected = selector.transform(X_val_encoded)
    
    model = LinearRegression()
    model.fit(X_train_selected, y_train)
    y_pred = model.predict(X_val_selected)
    
    mse = mean_squared_error(y_val, y_pred)
    
    if mse < lowest[0]:
        lowest[0] = mse
        lowest[1] = k
    
    print(f'Val MSE: {mse} \n')
    print(f'Lowest so far: k = {lowest[1]} with mse = {lowest[0]}' )
# Looks like k=100 is best.

selector = SelectKBest(score_func=f_regression, k=100)

X_train_full_selected = selector.fit_transform(X_train_full_encoded, y_train_full)
X_test_selected = selector.transform(X_test_encoded)

model = LinearRegression()

model.fit(X_train_full_selected, y_train_full)
# I'm going to be testing a lot of models, so let's make the next part a function.

def determine_error(test, model):

    y_pred = model.predict(test)

    model_mse = mean_squared_error(y_test, y_pred)

    improvement = baseline_mse - model_mse

    percent_improve = (improvement / baseline_mse) * 100

    print('Model mean squared error:', model_mse)
    print('Baseline mean squared error:', baseline_mse)
    print('Improvement over baseline:', improvement)
    print(f'Percent improvement: {percent_improve}%')
    
determine_error(test=X_test_selected, model=model)

# Nine percent is actually pretty good, considering how accurate the baseline was. But let's see if we can do better!
# Let's do a Ridge regression next. I'll do a similar for loop to get the optimal alpha value.

from sklearn.linear_model import Ridge

lowest = [1, 0]
for alpha in range(0, 1000, 1):
    ridge_reg_split = Ridge(alpha=alpha).fit(X_train_encoded, y_train)
    mse = mean_squared_error(y_val, ridge_reg_split.predict(X_val_encoded))
    if mse < lowest[0]:
        lowest[0] = mse
        lowest[1] = alpha
    print(f'For alpha = {alpha}, mse = {mse}')
    print(f'Lowest so far: alpha = {lowest[1]}, mse = {lowest[0]}')
# Let's go with alpha=504, then!

model = Ridge(alpha=504)

model.fit(X_train_full_encoded, y_train_full)

determine_error(test=X_test_encoded, model=model)

# Just a little better than the basic linear regression.<jupyter_output>Model mean squared error: 0.7727204860069009
Baseline mean squared error: 0.8528051787916153
Improvement over baseline: 0.08008469278471442
Percent improvement: 9.390737155019467%<jupyter_text>Random forest models<jupyter_code># Random forests do fine with ordinal encoding, so we can include the Team column this time.

non_features = ['Player', 'Target']
X_train = train.drop(non_features, axis=1)
y_train = train['Target']
X_val = val.drop(non_features, axis=1)
y_val = val['Target']

X_train_full = temp.drop(non_features, axis=1)
y_train_full = temp['Target']
X_test = test.drop(non_features, axis=1)
y_test = test['Target']
# I'll do cross-validation to determine the optimal hyperparameters.

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import make_pipeline
from scipy.stats import randint, uniform

pipeline = make_pipeline(
    ce.OrdinalEncoder(),
    RandomForestRegressor(random_state=100)
)

param_distributions = {
    'randomforestregressor__n_estimators': randint(50, 500),
    'randomforestregressor__max_depth': [5, 10, 15, 20, None],
    'randomforestregressor__max_features': uniform(0,1),
}

search = RandomizedSearchCV(
    pipeline,
    param_distributions=param_distributions,
    n_iter=20,
    cv=5,
    scoring='neg_mean_squared_error',
    verbose=10,
    return_train_score=True,
    n_jobs=-1
)

search.fit(X_train_full, y_train_full);
print('Best hyperparameters', search.best_params_)
print('Cross-validation mse', -search.best_score_)
param_values = list(search.best_params_.values())

pipeline = make_pipeline(
    ce.OrdinalEncoder(),
    RandomForestRegressor(max_depth=param_values[0], max_features=param_values[1],
                          n_estimators=param_values[2], random_state=100)
)

pipeline.fit(X_train_full, y_train_full)

determine_error(test=X_test, model=pipeline)

# Worse than both linear models.
# And finally, XGBoost.
# Since it doesn't play great with pipelines, I'll encode in advance.

encoder2 = ce.OrdinalEncoder()
X_train_encoded2 = encoder2.fit_transform(X_train)
X_val_encoded2 = encoder2.transform(X_val)

from xgboost import XGBRegressor

eval_set = [(X_train_encoded2, y_train),
            (X_val_encoded2, y_val)]

model = XGBRegressor(
    n_estimators=1000,
    max_depth=3,
    learning_rate=0.1, # I tested various combinations and this seemed to be the best.
    n_jobs=-1
)

model.fit(X_train_encoded2, y_train, eval_set=eval_set, eval_metric='rmse', early_stopping_rounds=50)
# Now to do it on my test set.

X_train_full_encoded2 = encoder2.fit_transform(X_train_full)
X_test_encoded2 = encoder2.transform(X_test)

eval_set = [(X_train_full_encoded2, y_train_full),
            (X_test_encoded2, y_test)]

model = XGBRegressor(
    n_estimators=1000,
    max_depth=3,
    learning_rate=0.1,
    n_jobs=-1
)

model.fit(X_train_encoded2, y_train, eval_set=eval_set, eval_metric='rmse', early_stopping_rounds=50)
determine_error(test=X_test_encoded2, model=model)

# Better than the random forests, but still a bit worse than the linear models.<jupyter_output>Model mean squared error: 0.7794797007022337
Baseline mean squared error: 0.8528051787916153
Improvement over baseline: 0.07332547808938161
Percent improvement: 8.598151126765007%<jupyter_text>Final modelSo it's been decided: my model is a Ridge Regression with alpha=504.Now the final step is to bring all my data together, put it into the model, and make predictions for the 2019-2020 season!<jupyter_code>df2 = pd.read_csv('wrangled-data.csv')

test_final = df2[df2['Year'] == 2019]
non_features = ['Player', 'Tm', 'Target']
X_train_final = df.drop(non_features, axis=1)
y_train_final = df['Target']

X_test_final = test_final.drop(non_features, axis=1)
encoder = ce.OneHotEncoder(use_cat_names=True)

X_train_final_encoded = encoder.fit_transform(X_train_final)
X_test_final_encoded = encoder.transform(X_test_final)
model = Ridge(alpha=504)

model.fit(X_train_final_encoded, y_train_final)
y_pred = model.predict(X_test_final_encoded)
test_final['preds'] = y_pred

test_final = test_final.drop('Target', axis=1)

test_final.head()
test_final.to_csv('final-predictions.csv', index=None, header=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09573498125823118, 'fraction_numerical': 0.031506432985513116, 'mean_word_length': 5.381383322559794, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '657282', 'license': 'MIT', 'n_tokens_mistral': 3541, 'n_tokens_neox': 3259, 'n_words': 930, 'path': 'notebooks/Modeling.ipynb', 'repo_name': 'davidanagy/nba-assists-prediction', 'size': '9871'}