<jupyter_start><jupyter_text>Overview- The goal of this notebook is to experiment with non parametric machine learning models.  - These models make no assumptions about the scaling of the data and require minimal preprocessing.      - However, they are prone to overfitting.  - I use a relatively small dataset with a mix of continuous and categorical features where the only preprocessing is to use One-Hot Encoding on the categorical features.  The small dataset with multiple features will be likely to overfitting with no intervention- We will examine overfitting behavior and explain how we can reduce it Table of contents=================   * [ImportPackagesAndData](ImportPackagesAndData)   * [ExploratoryAnalysis](ExploratoryAnalysis)    * [ModelFitting](ModelFitting)       * [DecisionTree](DecisionTree)        * [RandomForest](RandomForest)        * [ADABOOST](ADABOOST)   ImportPackagesAndData  Datasethttps://archive.ics.uci.edu/ml/datasets/heart+diseaseThis dataset is freely available courtesy of UC Irive Machine Learning Repository  Features Age : displays the age of the individual.Sex : displays the gender of the individual using the following format : 1 = male 0 = female.Chest-pain type : displays the type of chest-pain experienced by the individual using the following format : 1 = typical angina 2 = atypical angina 3 = non - anginal pain 4 = asymptoticResting Blood Pressure : displays the resting blood pressure value of an individual in mmHg (unit)Serum Cholestrol : displays the serum cholestrol in mg/dl (unit)Fasting Blood Sugar : compares the fasting blood sugar value of an individual with 120mg/dl. If fasting blood sugar > 120mg/dl then : 1 (true) else : 0 (false)Resting ECG : 0 = normal 1 = having ST-T wave abnormality 2 = left ventricular hyperthrophyMax heart rate achieved : displays the max heart rate achieved by an individual.Exercise induced angina : 1 = yes 0 = noST depression induced by exercise relative to rest : displays the value which is integer or float.Peak exercise ST segment : 1 = upsloping 2 = flat 3 = downslopingNumber of major vessels (0-3) colored by flourosopy : displays the value as integer or float.Thal : displays the thalassemia : 3 = normal 6 = fixed defect 7 = reversable defectDiagnosis of heart disease : Displays whether the individual is suffering from heart disease or not : 0 = absence 1,2,3,4 = present.<jupyter_code>import pandas as pd
import numpy as np

data_path = "https://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.cleveland.data"
heart_data = pd.read_csv(data_path, delimiter = ",", header=None)
heart_data = heart_data.replace({'?':np.NaN})
heart_data.columns = ['age', 'sex', 'cp', 'trestbps', 'chol',
              'fbs', 'restecg', 'thalach', 'exang', 
              'oldpeak', 'slope', 'ca', 'thal', 'target']
heart_data['target'] = heart_data.target.map({0: 0, 1: 1, 2: 1, 3: 1, 4: 1})
heart_data
print(heart_data.isna().sum())
print(heart_data.nunique())

heart_data['thal'] = pd.to_numeric(heart_data['thal'])
heart_data['thal'] = heart_data['thal'].fillna(heart_data['thal'].mean())

heart_data['ca'] = pd.to_numeric(heart_data['ca'])
heart_data['ca'] = heart_data['ca'].fillna(heart_data['ca'].mean())
heart_data['thal'].unique()
#heart_data['ca']<jupyter_output><empty_output><jupyter_text>ExploratoryAnalysis<jupyter_code>heart_data["target"].value_counts().sort_index(ascending=False).plot(kind='barh')
heart_data["target"].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>ModelFitting<jupyter_code>import category_encoders as ce
X = heart_data.iloc[:, :-1]
X = ce.OneHotEncoder(use_cat_names=True, cols = ['sex','cp','fbs','restecg','exang','slope','thal']).fit_transform(X)

y = heart_data.iloc[:, -1]

X.columns
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, test_size=.2, stratify=y)<jupyter_output><empty_output><jupyter_text>DecisionTreeFirst, I will examine basic Decision Trees using the CART algorithmCurrent model is wrong because it does >,< on categorical variables.  This is nonsense<jupyter_code>from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV

decision_tree = DecisionTreeClassifier(random_state = 42)

param_dict = {
    #"criterion":['gini','entropy'],
    "max_depth":range(1,20)
}


gridDtNoCv = GridSearchCV(decision_tree,
                   param_grid = param_dict,
                   cv=[(slice(None), slice(None))],
                   verbose=1,
                   n_jobs = -1)
gridDtNoCv.fit(X_train,y_train)

gridDtCv = GridSearchCV(decision_tree,
                   param_grid = param_dict,
                   cv=10,
                   verbose=1,
                   n_jobs = -1)

gridDtCv.fit(X_train,y_train)
print('Best depth with no cross validation: ' + str(gridDtNoCv.best_params_) + ' Score: ' + str(gridDtNoCv.best_score_) )
print('Best depth with cross validation: ' + str(gridDtCv.best_params_) + ' Score: ' + str(gridDtCv.best_score_) )
from matplotlib import pyplot as plt

max_depth = []
for i in gridDtNoCv.cv_results_['params']:
    val = sum( i.values() ) 
    max_depth.append(val)

Score_Dt_NoCV = gridDtNoCv.cv_results_['mean_test_score']   
Score_Dt_wCV = gridDtCv.cv_results_['mean_test_score']   


plt.plot(max_depth, Score_Dt_NoCV, label = "Training Score")
plt.plot(max_depth, Score_Dt_wCV, label = "Cross Validated Score")
plt.xlabel('Decision Tree Depth')
# Set the y axis label of the current axis.
plt.ylabel('Accuracy Score')
# Set a title of the current axes.
plt.title('Decision Tree Overfitting')
# show a legend on the plot
plt.legend()
plt.show()
from sklearn import tree
fig = plt.figure(figsize=(25,20))
_ = tree.plot_tree(gridDtCv.best_estimator_, 
                   feature_names=X.columns,  
                   class_names=['Present', 'Absent'],
                   filled=True,
                   impurity=False, 
                   proportion=True)<jupyter_output><empty_output><jupyter_text>RandomForestensemble of Decision Trees bagging - sampling with replacement - 63% of points get used uses subest of features.  Default is sqrt(num features)<jupyter_code>from sklearn.ensemble import RandomForestClassifier
random_forest = RandomForestClassifier(
    random_state = 42, 
    n_estimators = 100, 
    bootstrap = True)
random_forest.fit(X_train, y_train)

rf_param_dict = {
    #"criterion":['gini','entropy'],
    #"n_estimators":range(1,50),
    "max_depth":range(1,20)
}


gridRFNoCv = GridSearchCV(random_forest,
                   param_grid = rf_param_dict,
                   cv=[(slice(None), slice(None))],
                   verbose=1,
                   n_jobs = -1)
gridRFNoCv.fit(X_train,y_train)

gridRFCv = GridSearchCV(random_forest,
                   param_grid = rf_param_dict,
                   cv=10,
                   verbose=1,
                   n_jobs = -1)

gridRFCv.fit(X_train,y_train)
print('Best # of esitmators with no cross validation: ' + str(gridRFNoCv.best_params_) + ' Score: ' + str(gridRFNoCv.best_score_) )
print('Best # of esitmators with cross validation: ' + str(gridRFCv.best_params_) + ' Score: ' + str(gridRFCv.best_score_) )
n_estimators = []
for i in gridRFNoCv.cv_results_['params']:
    val = sum( i.values() ) 
    n_estimators.append(val)

Score_RF_NoCV = gridRFNoCv.cv_results_['mean_test_score']   
Score_RF_wCV = gridRFCv.cv_results_['mean_test_score']   


plt.plot(n_estimators, Score_RF_NoCV, label = "Training Score")
plt.plot(n_estimators, Score_RF_wCV, label = "Cross Validated Score")
plt.xlabel('Max Depth')
# Set the y axis label of the current axis.
plt.ylabel('Accuracy Score')
# Set a title of the current axes.
plt.title('Random Forest, Still Overfitting')
# show a legend on the plot
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>ADABOOST<jupyter_code>from sklearn.ensemble import AdaBoostClassifier

ada_clf = AdaBoostClassifier(
    RandomForestClassifier(max_depth = 3), 
    n_estimators = 100,
    random_state = 42, 
    algorithm = "SAMME.R")


ada_param_dict = {
    "learning_rate":np.linspace(.1,1,9,endpoint=False)
}


gridAdaNoCv = GridSearchCV(ada_clf,
                   param_grid = ada_param_dict,
                   cv=[(slice(None), slice(None))],
                   verbose=1,
                   n_jobs = -1)
gridAdaNoCv.fit(X_train,y_train)

gridAdaCv = GridSearchCV(ada_clf,
                   param_grid = ada_param_dict,
                   cv=10,
                   verbose=1,
                   n_jobs = -1)

gridAdaCv.fit(X_train,y_train)
print('Best # of esitmators with no cross validation: ' + str(gridAdaNoCv.best_params_) + ' Score: ' + str(gridAdaNoCv.best_score_) )
print('Best # of esitmators with cross validation: ' + str(gridAdaCv.best_params_) + ' Score: ' + str(gridAdaCv.best_score_) )
learning_rate = []
for i in gridAdaNoCv.cv_results_['params']:
    val = sum( i.values() ) 
    learning_rate.append(val)

Score_ADA_NoCV = gridAdaNoCv.cv_results_['mean_test_score']   
Score_ADA_wCV = gridAdaCv.cv_results_['mean_test_score']   


plt.plot(learning_rate, Score_ADA_NoCV, label = "Training Score")
plt.plot(learning_rate, Score_ADA_wCV, label = "Cross Validated Score")
plt.xlabel('Learning Rate')
# Set the y axis label of the current axis.
plt.ylabel('Accuracy Score')
# Set a title of the current axes.
plt.title('ADABOOST, Random Forest')
# show a legend on the plot
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Model Comparison: Decision Tree, Random Forest, Adaboost<jupyter_code>from sklearn.metrics import accuracy_score
%matplotlib inline

optimized_decison_tree = gridDtCv.best_estimator_
dt_training_score = accuracy_score(optimized_decison_tree.predict(X_train), y_train)
dt_test_score = accuracy_score(optimized_decison_tree.predict(X_test), y_test)

optimised_random_forest = gridRFCv.best_estimator_
rf_training_score = accuracy_score(optimised_random_forest.predict(X_train), y_train)
rf_test_score = accuracy_score(optimised_random_forest.predict(X_test), y_test)

optimised_adaboost = gridAdaCv.best_estimator_
ada_training_score = accuracy_score(optimised_adaboost.predict(X_train), y_train)
ada_test_score = accuracy_score(optimised_adaboost.predict(X_test), y_test)



print('cross validated decision tree: ' + str(optimized_decison_tree) )
print(' ')
print('_____cross validated training accuracy: ' + str(dt_training_score) )
print('_____testing accuracy: ' + str(dt_test_score) )
dt_feat_importances = pd.Series(optimized_decison_tree.feature_importances_, index=X.columns)
dt_feat_importances.nlargest(20).plot(kind='barh')
plt.show()

print(' ')
print('cross validated random forest: ' + str(optimised_random_forest) )
print(' ')
print('_____cross validated training accuracy: ' + str(rf_training_score) )
print('_____testing accuracy: ' + str(rf_test_score) )
rf_feat_importances = pd.Series(optimised_random_forest.feature_importances_, index=X.columns)
rf_feat_importances.nlargest(20).plot(kind='barh')
plt.show()

print(' ')
print('cross validated Adaboosted Random Forest: ' + str(optimised_adaboost)  )
print(' ')
print('_____cross validated training accuracy: ' + str(ada_training_score) )
print('_____testing accuracy: ' + str(ada_test_score) )
ada_feat_importances = pd.Series(optimised_adaboost.feature_importances_, index=X.columns)
ada_feat_importances.nlargest(20).plot(kind='barh')
plt.show()<jupyter_output>cross validated decision tree: DecisionTreeClassifier(max_depth=3, random_state=42)
 
_____cross validated training accuracy: 0.8471074380165289
_____testing accuracy: 0.8524590163934426
{'fraction_non_alphanumeric': 0.10950785879927853, 'fraction_numerical': 0.011938503822038994, 'mean_word_length': 4.630560928433269, 'pattern_counts': {'":': 6, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '28535', 'license': 'MIT', 'n_tokens_mistral': 4064, 'n_tokens_neox': 3773, 'n_words': 877, 'path': 'heart_disease_non_linear_models.ipynb', 'repo_name': 'branclark/heartDisease_nonLinearModels', 'size': '11643'}