<jupyter_start><jupyter_text>Load the Packages<jupyter_code># import packages
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import roc_auc_score
from sklearn.preprocessing import StandardScaler
from joblib import dump  
import joblib 
# Adjusting the jupyter notebook display to see more data
pd.set_option('display.max_rows', 600)
pd.set_option('display.max_columns', 100)
pd.set_option('display.width', 500)<jupyter_output><empty_output><jupyter_text>Importing the data<jupyter_code>#import the data
train = pd.read_csv("/home/jovyan/work/data/raw/train.csv")
test = pd.read_csv("/home/jovyan/work/data/raw/test.csv")
# observe the shape of the train data
train.shape
# observe the shape of the test data
test.shape
train.info()
test.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 3799 entries, 0 to 3798
Data columns (total 20 columns):
Id         3799 non-null int64
GP         3799 non-null int64
MIN        3799 non-null float64
PTS        3799 non-null float64
FGM        3799 non-null float64
FGA        3799 non-null float64
FG%        3799 non-null float64
3P Made    3799 non-null float64
3PA        3799 non-null float64
3P%        3799 non-null float64
FTM        3799 non-null float64
FTA        3799 non-null float64
FT%        3799 non-null float64
OREB       3799 non-null float64
DREB       3799 non-null float64
REB        3799 non-null float64
AST        3799 non-null float64
STL        3799 non-null float64
BLK        3799 non-null float64
TOV        3799 non-null float64
dtypes: float64(18), int64(2)
memory usage: 593.7 KB<jupyter_text>**Data Glossary**Id_old - Previous Player Identifier (apparently not in data)Id - Player IdentifierGP - Games PlayedMIN - Minutes PlayedPTS - Points Per GameFGM - Field Goals MadeFGA - Field Goals AttemptsFG% - Field Goals Percent3P Made - 3-Points Made3PA - 3-Points Attempts3P% - 3-Points PercentFTM - Free Throw MadeFTA - Free Throw AttemptsFT% - Free Throw PercentOREB - Offensive ReboundsDREB - Defensive ReboundsREB - ReboundsAST - AssistsSTL - StealsBLK - BlocksTOV - TurnoversTARGET_5Yrs - Outcome: 1 if career length >= 5 years, 0 otherwise<jupyter_code># identifying if there is any null values for each feature
test.isnull().sum()<jupyter_output><empty_output><jupyter_text>Correlation PlotUsing the train dataset to check any association with the Target variable "TARGET_5Yrs" to the predictor variables.<jupyter_code># This is another Correlation plot I preferred this one as it does give a lot more information
corr = train.corr()
corr.style.background_gradient(cmap='coolwarm')
# Sample of using a correlation plot to identify which feature is associated with our Target variable "TARGET_5Yrs"
import seaborn as sns
import matplotlib.pyplot as plt

f, ax = plt.subplots(figsize=(10, 10))
corr = train.corr()
sns.heatmap(corr, mask=np.zeros_like(corr, dtype=np.bool), cmap=sns.diverging_palette(220, 10, as_cmap=True),
            square=True, ax=ax)<jupyter_output><empty_output><jupyter_text>*Lets examine the details of the dataset*<jupyter_code># observing the first 5 records 
train.head()
# the first 5 rows for test data
test.head()<jupyter_output><empty_output><jupyter_text>Check the **TARGET_5Yrs** variable only has 0 or 1.<jupyter_code># Check Target variable to see the unique values in that column, should be only 1 or 0
train["TARGET_5Yrs"].unique()

# Thats great there is no artifacts in that column besides 1 and 0<jupyter_output><empty_output><jupyter_text>Produce a crosstab to show the data imbalance for target variable<jupyter_code># Using crosstab function lets check the train data of the sum of the group
pd.crosstab(train['TARGET_5Yrs'],train['TARGET_5Yrs'],margins=True)<jupyter_output><empty_output><jupyter_text>From the crosstab above, in the train data we have an imbalanced dataset of 1331 NBA rookie players of whom do not make it to the NBA after 5 years. compared to 6669 NBA rookie players that do play on after 5 years with the NBA profession.This represents about 79% success rate of the 8000 players which adds to the issue that we have an **imbalanced dataset** that needs to be balanced out in order for make our models to be reliable and reasonably accurate as possible.  *Summary statistics*<jupyter_code># Summary statistics for train data
train.describe()
# Summary statistics for test data
test.describe()<jupyter_output><empty_output><jupyter_text>Preparing the data<jupyter_code># Create a copy of the train data
df = train.copy()

# Seperate the TARGET_5YRS column from df and naming it as "target"
target = df.pop('TARGET_5Yrs')
# review df 
df.head()<jupyter_output><empty_output><jupyter_text>6. *Logistic Regression Feature Importance*These coefficients can provide the basis for a crude feature importance score. This assumes that the input variables have the same scale or have been scaled prior to fitting a model.source: https://machinelearningmastery.com/calculate-feature-importance-with-python/  *Note* GO TO FEATURE SELECTION 2 AS THIS WILL HAVE THE HIGHEST KAGGLE SCORE<jupyter_code># Create a copy of the train data
df = train.copy()

# Separate the TARGET_5YRS column from df and naming it as "target"
target = df.pop('TARGET_5Yrs')

X = df

y = target
# logistic regression for feature importance
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from matplotlib import pyplot
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.metrics import roc_auc_score
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV


# define dataset
# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# define the model
model = LogisticRegression()
# fit the model
model.fit(X, y)
# get importance
importance = model.coef_[0]
# summarize feature importance
for i,v in enumerate(importance):
	print('Feature: %0d, Score: %.5f' % (i,v))
# plot feature importance
plt_1 = pyplot.figure(figsize=(20, 10))
pyplot.bar([x for x in range(len(importance))], importance,tick_label=X.columns)
pyplot.title("Logistic Regression Feature importance")
pyplot.ylabel("Feature importance scores")
pyplot.show()
X.columns
# find feature number relating to which columns
for n in enumerate(df.columns):
    print(n)<jupyter_output>(0, 'Id')
(1, 'GP')
(2, 'MIN')
(3, 'PTS')
(4, 'FGM')
(5, 'FGA')
(6, 'FG%')
(7, '3P Made')
(8, '3PA')
(9, '3P%')
(10, 'FTM')
(11, 'FTA')
(12, 'FT%')
(13, 'OREB')
(14, 'DREB')
(15, 'REB')
(16, 'AST')
(17, 'STL')
(18, 'BLK')
(19, 'TOV')<jupyter_text>Seems the features **'3P Made', '3PA', '3P%'** and **'FT%'** is more associated in predicting players less than 5 years and should be likely to be excluded if to predict players that would survive a minimum 5 years in the NBA.   Creating the feature selection LETS USE ALL THESE<jupyter_code># We select the features that are positive to see if these will improve the roc auc score for 
# the NBA rookie players predictions for at least 5 years.

# First selection feature
selected_features1 = ['Id','GP','MIN','FGM','FG%','3P Made','3P%','FTA','FT%','OREB','DREB','AST','BLK']<jupyter_output><empty_output><jupyter_text>*Preparing the data*<jupyter_code># Create a copy of the train data
df = train.copy()

# Seperate the TARGET_5YRS column from df and naming it as "target"
target = df.pop('TARGET_5Yrs')
# Save the prepared dataframe in the data / interim folder

df.to_csv('../data/interim/df.csv')


target.to_csv('../data/interim/training_target.csv')
# saving the selected feature 1 as train_selection1

train_selection1 = train.loc[:,selected_features1]<jupyter_output><empty_output><jupyter_text>***Scaling, train, test split and model creation for the feature selection 1 for Logistic Regression*** ***Scaling the training data***<jupyter_code>#Save standard scaler as scaler 
scaler = StandardScaler()

#  save it to models folder as scaler.joblib
dump(scaler, '../models/scaler.joblib')

# scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_sel1 = scaler.fit_transform(train_selection1)
# Create train, test, split

Xsf1_train, Xsf1_test, ysf1_train, ysf1_test = train_test_split(X_sel1, y, test_size=0.2, random_state=42)
# save the different sets into processed folder
np.save('../data/processed/Xsf1_train', Xsf1_train)
np.save('../data/processed/Xsf1_test',  Xsf1_test)
np.save('../data/processed/ysf1_train', ysf1_train)
np.save('../data/processed/ysf1_test',  ysf1_test)
# logistic regression model
model_logreg = LogisticRegression(solver='saga') # saga has penalties like elasticnet, l1, l2 and none

# fit model
model_logreg.fit(Xsf1_train, ysf1_train)

# predict probabilities
pred_logreg = model_logreg.predict_proba(Xsf1_test)
#Save Logistic regression model as model_logreg 
model_logreg = LogisticRegression(solver='saga')

#  save it to models folder as model_logreg.joblib
dump(model_logreg, '../models/model_logreg.joblib')<jupyter_output><empty_output><jupyter_text>**Import roc auc score metric**<jupyter_code>from sklearn.metrics import roc_auc_score
auc_score_logreg = roc_auc_score(ysf1_test, pred_logreg[:,1])
# auc roc score logistic regression with selected features 1
print(f' Logistic Regression for selected feature importance 1 is : {auc_score_logreg}')<jupyter_output>Logistic Regression for selected feature importance 1 is : 0.7116892424160555<jupyter_text>Apply the model to the  test dataset  - Remember to select the features first  - scale the data (Test dataset) - predict the probability using Logistic Regression model  **Prepare the data for test dataset**<jupyter_code># make a copy of the test dataset as "test_data"
test_data =test.copy()
# Use the selected features as you done previously for the training data
test_selection1 = test_data.loc[:,selected_features1]

import os
os.getcwd()<jupyter_output><empty_output><jupyter_text>*** Feature Selection 2: Scaling, train, test split and model creation for logistic regression*** ***Preparing Data***<jupyter_code># load the df from interim folder and save it to 'df'
df = pd.read_csv('../data/interim/df.csv')
# load the target from train dataset from interim folder and save it to  "target"
target = pd.read_csv('../data/interim/training_target.csv')<jupyter_output><empty_output><jupyter_text>***Creating feature selection***<jupyter_code># We select the features that are positive to see if these will improve the roc auc score for 
# the NBA rookie players predictions for at least 5 years.

# second selection feature

selected_features2 = ['GP','FGM','3P Made','OREB','DREB'] 


# selected_features2 = ['GP','MIN','PTS','3P Made','STL','AST']
# selected_features2 = ['GP','FGM','3P Made','OREB','DREB'] # BEST SCORE
# selected_features2 = ['GP','MIN','PTS','FG%','3P Made','STL','AST','TOV'] # kaggle scored" 0.70307
# selected_features2 = ['GP','MIN','PTS','FGA','FG%','REB','OREB'] # 7 features scored 0.70636
# selected_features2 = ['GP','MIN','FGM'] # No Good
# selected_features2 = ['Id','GP','MIN','PTS','FGA','FGM','FG%','REB','FTA','OREB'] # 10 features scored:  0.70443
# selected_features2 = ['GP','MIN','PTS','FGA','FGM','FG%','REB','FTA','OREB'] # 9 features  scored:  0.70612
# selected_features2 = ['DREB','BLK','GP','MIN','PTS','FGA','FGM','FG%','REB','FTA','OREB'] # 11 features scored: 0.70591
# selected_features2 = ['GP','MIN','PTS','FG%'] # kaggle scored 0.70339
train_selection2 = train.loc[:,selected_features2]
train_selection2<jupyter_output><empty_output><jupyter_text>***Scaling the training data***<jupyter_code># Load the scaler model
scaler = joblib.load('../models/scaler.joblib')
# scale the data
# from sklearn.preprocessing import StandardScaler
# scaler = StandardScaler()
X_sel2 = scaler.fit_transform(train_selection2)
# train test split
Xsf2_train, Xsf2_test, ysf2_train, ysf2_test = train_test_split(X_sel2, y, test_size=0.2, random_state=42)
# logistic regression model
model_logreg2 = LogisticRegression(solver='liblinear')#'liblinear' 

# fit model
model_logreg2.fit(Xsf2_train, ysf2_train)

# predict probabilities
pred_logreg2 = model_logreg2.predict_proba(Xsf2_test)
from sklearn.metrics import roc_auc_score
auc_score_logreg2 = roc_auc_score(ysf2_test, pred_logreg2[:,1])<jupyter_output><empty_output><jupyter_text>Save the roc_auc_score metrics using joblib and name it as "***roc_auc_score***" in the models folder<jupyter_code># save roc_auc_score to models folder as roc_auc_score.joblib
dump(roc_auc_score, '../models/roc_auc_score.joblib')<jupyter_output><empty_output><jupyter_text>Save the Logistic Regression model using joblib and name it as "***model_logreg2***" in the models folder<jupyter_code># save logistic regression with liblinear solver to models folder as model_logreg2.joblib
dump(model_logreg2, '../models/model_logreg2.joblib')
# auc roc score logistic regression with selected features 2
auc_score_logreg2<jupyter_output><empty_output><jupyter_text>***ROC AUC Scores reference for selected features***0.7093259695533102 for selected features ['GP','FGM','3P Made','OREB','DREB']0.711289675824     for selected features ['GP','MIN','FGM']**0.7135235516777532 for selected features  ['GP','MIN','PTS','FGA','FGM','FG%','REB','FTA','OREB']** 0.7127955157537305 for selected features  ['Id','GP','MIN','PTS','FGA','FGM','FG%','REB','FTA','OREB'] 0.7133898888323271 for selected features  ['DREB','BLK','GP','MIN','PTS','FGA','FGM','FG%','REB','FTA','OREB']   Scaling the data (Test dataset)<jupyter_code># make a copy of the test dataset as "test_data"
test_data =test.copy()
# test_selection1 = test_data.loc[:,selected_features1]
test_selection2 = test_data.loc[:,selected_features2]
test_selection2<jupyter_output><empty_output><jupyter_text>***Scale the data for the feature selection 2***<jupyter_code># # scale the data
# from sklearn.preprocessing import StandardScaler
# Load the scaler model
scaler = joblib.load('../models/scaler.joblib')
test_sel2 = scaler.fit_transform(test_selection2)
test_sel2
# After standard scaling the test_data we now do a prediction on it based on the model was tested. 
# In this case this is Logistic Regression model "model1" as the AUC-ROC score was better than KNN model.

logreg_sf2_pred = model_logreg2.predict_proba(test_sel2)
pd.DataFrame(logreg_sf2_pred[:,1])<jupyter_output><empty_output><jupyter_text>Create a dataframe to attach predicted results to the players id in the test dataset.<jupyter_code># We want the "Id" column only in test data
test.iloc[:,0]
#creating a dataframe combining test['Id'] and the results from the model
frames = [test.iloc[:,0], pd.DataFrame(logreg_sf2_pred[:,1])]
logreg_sf2_test_result = pd.concat(frames, axis=1)
logreg_sf2_test_result.columns = ['Id','TARGET_5Yrs']

logreg_sf2_test_result["TARGET_5Yrs"] = logreg_sf2_test_result["TARGET_5Yrs"].round(2)
logreg_sf2_test_result<jupyter_output><empty_output><jupyter_text>Save to csv file as ***'group3_log_reg_sf2.csv'***<jupyter_code># save to csv
logreg_sf2_test_result.to_csv('../data/processed/group3_log_reg_sf2_6f.csv',index = False)

# check current directory
import os
os.getcwd()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11288423806409417, 'fraction_numerical': 0.034270765206017004, 'mean_word_length': 6.023886081763895, 'pattern_counts': {'":': 0, '<': 95, '<?xml version=': 0, '>': 96, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '401778', 'license': 'FTL', 'n_tokens_mistral': 5441, 'n_tokens_neox': 5075, 'n_words': 1509, 'path': 'notebooks/Group3_Logistic_regression_feature_importance_best_model.ipynb', 'repo_name': 'JKaur1992/adsi_at1', 'size': '15290'}