<jupyter_start><jupyter_text>Explore DataUnderstand the training set and test set, and what issues there are to determine what data preparation steps are required.<jupyter_code>ex.explore("train.csv")
ex.explore("test.csv")<jupyter_output>=== dataframe info ===
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3799 entries, 0 to 3798
Data columns (total 21 columns):
 #   Column   Non-Null Count  Dtype  
---  ------   --------------  -----  
 0   Id_old   3799 non-null   int64  
 1   Id       3799 non-null   int64  
 2   GP       3799 non-null   int64  
 3   MIN      3799 non-null   float64
 4   PTS      3799 non-null   float64
 5   FGM      3799 non-null   float64
 6   FGA      3799 non-null   float64
 7   FG%      3799 non-null   float64
 8   3P Made  3799 non-null   float64
 9   3PA      3799 non-null   float64
 10  3P%      3799 non-null   float64
 11  FTM      3799 non-null   float64
 12  FTA      3799 non-null   float64
 13  FT%      3799 non-null   float64
 14  OREB     3799 non-null   float64
 15  DREB     3799 non-null   float64
 16  REB      3799 non-null   float64
 17  AST      3799 non-null   float64
 18  STL      3799 non-null   float64
 19  BLK      3799 non-null   float64
 20  TOV      3799 non-null   floa[...]<jupyter_text>Data Quality Concerns- GP, 3PM, 3PA, 3p%, FT%, BLK have negative value as minimum - **Convert to absolute value** - There are potential outliers in the value between 75% mark and MAX is huge jump across all features- Check if all made value is smaller than attempt value- All % values are bit off and not close to Made/attempt - **Consider dropping these fields or recreate them**- BLK has outliers - **Fix it or drop these**- 3PA and FTA has 0 value - Need to ensure 3PM and FTM are also 0 in these cases- Imbalanced data - 21 : 4 Ratio  Data PreparationConvert to absolute value<jupyter_code>df_train_abs = getAbsolute.abs(pd.read_csv("../data/raw/train.csv"),'train')
df_test_abs = getAbsolute.abs(pd.read_csv("../data/raw/test.csv"),'test')<jupyter_output><empty_output><jupyter_text>Feature Engineering - Clustering- Standardization as later we're going to use LogisticRegression (The best model from week 1)- Use KNN to add a feature called 'cluster'- Calculate the distance between each observation and the centrion of the cluster they belong<jupyter_code>df_train_clean = kpw_build_features.build(df_train_abs)
df_test_clean = kpw_build_features.build(df_test_abs)
y = df_train_clean.pop('TARGET_5Yrs')
X = df_train_clean.iloc[:,2:] 
X_test = df_test_clean.iloc[:,2:]
columns = X.columns
sc,X_std,X_test_std = standardization.standardScale(X,X_test)
X_train, X_val, y_train, y_val = train_test_split(X_std,y,test_size=0.2, random_state=8, stratify=y)
gmm = clustering.findCluster(X_train)
df_X_train = clustering.addClusterFeatures(gmm, X_train, columns)
df_X_val = clustering.addClusterFeatures(gmm, X_val, columns)
X = np.asarray(X_std)
X_train = np.asarray(df_X_train)
X_val = np.asarray(df_X_val)
y = np.asarray(y)
y_train = np.asarray(y_train)
y_val = np.asarray(y_val)<jupyter_output><empty_output><jupyter_text>Dump all processed data<jupyter_code>joblib.dump(X, "../data/processed/X")
joblib.dump(X_train, "../data/processed/X_train")
joblib.dump(X_val, "../data/processed/X_val")
joblib.dump(y, "../data/processed/y")
joblib.dump(y_train, "../data/processed/y_train")
joblib.dump(y_val, "../data/processed/y_val")<jupyter_output><empty_output><jupyter_text>Feature Engineering - Stacking TP and TN prediction resultsSimplified week 2 approach by justing adding two classifiers, and one is stronger in TP and another one is stronger in TN.<jupyter_code>classifier_TP = LogisticRegression(penalty='elasticnet', solver='saga', l1_ratio=0.3, C=0.01)
model_TP,TP_tr_auroc,TP_v_auroc = eval_model.eval_model(classifier_TP, X_train, y_train, X_val, y_val)
classifier_TN = xgb.XGBRFClassifier(max_depth=4, learning_rate=0.01, objective='binary:logistic', scale_pos_weight=0.4)
model_TN,TN_tr_auroc,TN_v_auroc = eval_model.eval_model(classifier_TN, X_train, y_train, X_val, y_val)<jupyter_output>[23:30:58] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.<jupyter_text>Use the trained model to add the prediction probability as new features of TP and TN to both training set and validation set<jupyter_code>df_PN_train = df_X_train
df_PN_train['TP'] = model_TP.predict_proba(X_train)[:,1]
df_PN_train['TN'] = model_TN.predict_proba(X_train)[:,1]
X_PN_train = np.asarray(df_PN_train)
df_PN_val = df_X_val
df_PN_val['TP'] = model_TP.predict_proba(X_val)[:,1]
df_PN_val['TN'] = model_TN.predict_proba(X_val)[:,1]
X_PN_val = np.asarray(df_PN_val)<jupyter_output><empty_output><jupyter_text>Experiment on all features plus TP and TN new features.<jupyter_code>classifier_stacking = xgb.XGBRFClassifier(max_depth=1, learning_rate=0.1, objective='binary:logistic', scale_pos_weight=0.6)
model_stacking,stacking_tr_auroc,stacking_v_auroc = eval_model.eval_model(classifier_stacking, X_PN_train, y_train, X_PN_val, y_val)<jupyter_output>[23:31:01] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.<jupyter_text>😔 **Result is not ideal as still no TN detected when hyperparameters are tuned to remove overfitting.**  Experiment on using feature **cluster** to create different models and combine them- The assumption here is that each group is naturally/factually closer to each other- Running a classifier in each group shall achieve better performance in classification- When it comes to validating, we use if else condition to determine which model to use  Step 1 - Recalculate the clusters in training set and test set<jupyter_code>gmm = clustering.findCluster(X_std)
df_X_std = clustering.addClusterFeatures(gmm, X_std, columns)
X_std = np.asarray(df_X_std)
df_X_test = clustering.addClusterFeatures(gmm, X_test_std, columns)
X_test = np.asarray(df_X_test)<jupyter_output><empty_output><jupyter_text>Step 2 - Added in TP and TN prediction results<jupyter_code>df_X_std.head()
model_TP = classifier_TP.fit(X_std, y)
model_TN = classifier_TN.fit(X_std, y)
model_TN.feature_importances_
df_PN_std = df_X_std
df_PN_std['TP'] = model_TP.predict_proba(X_std)[:,1]
df_PN_std['TN'] = model_TN.predict_proba(X_std)[:,1]
X_PN_std = np.asarray(df_PN_std)
df_PN_test = df_X_test
df_PN_test['TP'] = model_TP.predict_proba(X_test)[:,1]
df_PN_test['TN'] = model_TN.predict_proba(X_test)[:,1]
X_PN_test = np.asarray(df_PN_test)
df_PN_std['TARGET'] = y<jupyter_output><empty_output><jupyter_text>Step 1 - Train multiple classifiers based on cluster grouping<jupyter_code>model_0 = eval_model.eval_model_with_df(LogisticRegression(penalty='elasticnet', solver='saga', l1_ratio=0.7, C=0.4), df_PN_std[df_PN_std['cluster']==0])
model_1 = eval_model.eval_model_with_df(xgb.XGBRFClassifier(max_depth=2, learning_rate=1, scale_pos_weight=10,objective='binary:logistic'), df_PN_std[df_PN_std['cluster']==1])
model_2 = eval_model.eval_model_with_df(xgb.XGBRFClassifier(max_depth=2, learning_rate=0.1, scale_pos_weight=1.6,objective='binary:logistic'), df_PN_std[df_PN_std['cluster']==2])
model_3 = eval_model.eval_model_with_df(xgb.XGBRFClassifier(max_depth=2, learning_rate=10, scale_pos_weight=10,objective='binary:logistic'), df_PN_std[df_PN_std['cluster']==3])
model_4 = eval_model.eval_model_with_df(xgb.XGBRFClassifier(max_depth=1, learning_rate=1, scale_pos_weight=10,objective='binary:logistic'), df_PN_std[df_PN_std['cluster']==4])
model_5 = eval_model.eval_model_with_df(xgb.XGBRFClassifier(max_depth=2, learning_rate=9, scale_pos_weight=3,objective='binary:logistic'), df_PN_std[df_PN_std['cluster']==5])
model_6 = eval_model.eval_model_with_df(xgb.XGBRFClassifier(max_depth=3, learning_rate=1, scale_pos_weight=10,objective='binary:logistic'), df_PN_std[df_PN_std['cluster']==6])<jupyter_output>[23:32:30] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.<jupyter_text>Step 2 - Combined and use the cluster driven models<jupyter_code>clfs = [model_0,model_1,model_2,model_3,model_4,model_5,model_6]
def combineClfs(df, clfs):
    y_clfs = np.ones(df.shape[0])
    for counter in range(df.shape[0]):
        cluster = int((df.loc[counter])['cluster'])
        model = clfs[cluster]
        X_clfs = np.asarray(df.loc[counter])
        X_clfs = X_clfs.reshape(1,25)
        y_clfs[counter] = model.predict_proba(X_clfs)[0,1]
    return y_clfs
df_clfs = df_PN_std
y_clfs = df_clfs.pop('TARGET')
y_pred_clfs = combineClfs(df_clfs,clfs)
roc_auc_score(y, y_pred_clfs)
from numpy import vectorize
vfunc = vectorize(lambda x: 1 if x >= 0.5 else 0)
confusion_matrix(y, vfunc(y_pred_clfs))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.111597853466214, 'fraction_numerical': 0.03504544956740773, 'mean_word_length': 5.685212298682284, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '637411', 'license': 'MIT', 'n_tokens_mistral': 3563, 'n_tokens_neox': 3331, 'n_words': 765, 'path': 'notebooks/Wang_Kai-Ping-Week3_assignmentC.ipynb', 'repo_name': 'ronvoluted/kaggle-nba', 'size': '9131'}