<jupyter_start><jupyter_text>- **age**- **sex**  * 0 = female  * 1 = male- **cp**: chest pain  * 1 = typical angina  * 2 = atypical angina  * 3 = non-anginal pain  * 4 = asymptomatic- **restbp**: resting blood pressure, in mm Hg- **chol**: serum cholesterol in mg/dl- **fbs**: fasting blood sugar  * 0 = fbs >= 120 mg/dl  * 1 = fbs < 120 mg/dl- **restecg**: resting ecg results  * 1 = normal  * 2 = having ST-T wave abnormality  * 3 = showing probable or definite left ventricular hyperthropy- **thalach**: maximum heart rate achieved- **exang**: exercise induced angina  * 0 = no  * 1 = yes- **oldpeak**: ST depression induced by exercise relative to rest- **slope**: the slope of the peak exercise ST segment  * 1 = upsloping  * 2 = flat  * 3 = downsloping- **ca**: number of major vessels (0-3) colored by fluoroscopy- **thal**: thalium heart scan  * 3 = normal (no cold spots)  * 6 = fixed defect (colp spots during rest and exercise)  * 7 = reversible defect (when cold spots only appear during exercise)- **hd**: diagnosis of Heart Diseases, target value<jupyter_code>df.columns = ['age', 'sex', 'cp', 'restbp', 'chol', 'fbs', 'restecg', 'thalach', 'exang', 'oldpeak', 'slope', 'ca', 'thal', 'hd']
df.head()
df.dtypes
df['ca'].unique()
df['thal'].unique()
missing_mask = (df['ca'] == '?') | (df['thal'] == '?')
df[missing_mask].shape[0]
df[missing_mask]
print(f"{df[missing_mask].shape[0]/df.shape[0]*100:0.2f} % missing values")
df_no_missing = df[~missing_mask]
len(df_no_missing)
df_no_missing = df_no_missing.astype({'ca': np.float64,
                                      'thal': np.float64})
X = df_no_missing.drop('hd', axis=1)
X.head()
y = df_no_missing['hd']
y.head()
X_dummies = pd.get_dummies(X, columns = ['cp',
                             'restecg',
                             'slope',
                             'thal'
                            ])
X_dummies.head()
y.unique()<jupyter_output><empty_output><jupyter_text>Turning the problem into binary classification, either has heart diseases or not.<jupyter_code>y.replace(to_replace=[2, 3, 4], value=1, inplace=True)
y.unique()
X_train, X_test, y_train, y_test = train_test_split(X_dummies, y, random_state=42)
tree = DecisionTreeClassifier(random_state=42)
fitted_tree = tree.fit(X_train, y_train)
plt.figure(figsize=(15, 7.5))
plot_tree(fitted_tree,
         filled = True,
         rounded = True,
         class_names = ['No HD', 'Yes HD'],
         feature_names = X_dummies.columns)
plt.show()
plot_confusion_matrix(fitted_tree,
                      X_test,
                      y_test,
                      display_labels=['No HD', 'Yes HD'],
                      cmap=plt.cm.Reds,
                      normalize=None)
plt.title('Confusion Matrix')
plt.show()
plot_confusion_matrix(fitted_tree,
                      X_test,
                      y_test,
                      display_labels=['No HD', 'Yes HD'],
                      cmap=plt.cm.Reds,
                      normalize='true')
plt.title('Confusion Matrix, Normalized')
plt.show()<jupyter_output><empty_output><jupyter_text>Cost Complexity Pruning Visualizing alpha<jupyter_code>path = fitted_tree.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas, impurities = path.ccp_alphas, path.impurities
ccp_alphas = ccp_alphas[:-1] # Exclusing the max value for alpha that would prune all leaves, leaving only a root instead of a tree.
trees = []
for ccp_alpha in ccp_alphas:
    tree = DecisionTreeClassifier(ccp_alpha = ccp_alpha,
                                  random_state = 42)
    tree.fit(X_train, y_train)
    trees.append(tree)
train_scores = [tree.score(X_train, y_train) for tree in trees]
test_scores = [tree.score(X_test, y_test) for tree in trees]
fig, ax = plt.subplots()
ax.set_xlabel('Alpha')
ax.set_ylabel('Accuracy')
ax.set_title("Accuracy vs. Alpha for train and test")
ax.plot(ccp_alphas, train_scores, marker = 'd', label = 'train', drawstyle = 'steps-post') #, drawstyle = 'steps-post'
ax.plot(ccp_alphas, test_scores, marker = 'o', label = 'test', drawstyle = 'steps-post')
ax.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>For the test set, the accuracy is maximum when alpha is around 0.016. When applying **Cost Complexity Pruning** on a single **Classification Tree**, values for alpha go from 0 to 1 included, because GINI scores go from 0 to 1.  Values for alpha for a **Regression Tree** can be much larger, since the sum of squared residuals can go from 0 to positive infinity.<jupyter_code>tree = DecisionTreeClassifier(ccp_alpha = 0.016,
                              random_state = 42)

scores = cross_val_score(tree, X_train, y_train, cv=5)
df_cv = pd.DataFrame(data = {'tree': np.arange(5), 'accuracy': scores})
df_cv.plot(x = 'tree', y = 'accuracy', marker = 'o', linestyle = '--')
plt.show()<jupyter_output><empty_output><jupyter_text>The graph shows that using different **train** and **test** data with the same alpha resulted in different accuracies, suggesting that alpha is sensitive to the datasets.  Let's use cross validation to find the optimal value for ccp_alpha.<jupyter_code>alpha_values = []

for ccp_alpha in ccp_alphas:
    tree = DecisionTreeClassifier(ccp_alpha = ccp_alpha,
                                  random_state = 42)
    scores = cross_val_score(tree, X_train, y_train, cv = 5)
    alpha_values.append([ccp_alpha, np.mean(scores), np.std(scores)])

alpha_results = pd.DataFrame(alpha_values, columns = ['alpha', 'mean_accuracy', 'std'])
alpha_results.plot(x = 'alpha',
                   y = 'mean_accuracy',
                   yerr = 'std',
                   marker = 'o',
                   linestyle = '--'
                  )
plt.show()
alpha_results.head()
max_mask = alpha_results['mean_accuracy'] == np.max(alpha_results['mean_accuracy'])
alpha_results[max_mask]
best_alpha = np.float64(alpha_results[max_mask]['alpha'])
tree_pruned = DecisionTreeClassifier(ccp_alpha = best_alpha,
                                     random_state = 42)
tree_pruned = tree_pruned.fit(X_train, y_train)
plot_confusion_matrix(tree_pruned,
                      X_test,
                      y_test,
                      display_labels=['No HD', 'Yes HD'],
                      cmap=plt.cm.Reds,
                      normalize=None)
plt.title('Confusion Matrix')
plt.show()
plot_confusion_matrix(tree_pruned,
                      X_test,
                      y_test,
                      display_labels=['No HD', 'Yes HD'],
                      cmap=plt.cm.Reds,
                      normalize='true')
plt.title('Confusion Matrix, Normalized')
plt.show()
plt.figure(figsize=(15, 7.5))
plot_tree(tree_pruned,
         filled = True,
         rounded = True,
         class_names = ['No HD', 'Yes HD'],
         feature_names = X_dummies.columns)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1290842490842491, 'fraction_numerical': 0.012161172161172162, 'mean_word_length': 2.9252443933295, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '114645', 'license': 'MIT', 'n_tokens_mistral': 2326, 'n_tokens_neox': 2164, 'n_words': 564, 'path': 'DecisionTree.ipynb', 'repo_name': 'giacomomiolo/data-science-trove', 'size': '6825'}