<jupyter_start><jupyter_text>OverviewWe will use the Wisconsin Diagnostic Breast Cancer (WDBC) dataset, which you can download from http://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic) . The dataset consists of 569 samples of biopsied tissue. The tissue for each sample is imaged and 10 characteristics of the nuclei of cells present in each image are characterized. These characteristics are:1. Radius2. Texture3. Perimeter4. Area5. Smoothness6. Compactness7. Concavity8. Number of concave portions of contour 9. Symmetry10. Fractal dimensionEach of the 569 samples used in the dataset consists of a feature vector of length 30. The first 10 entries in this feature vector are the mean of the characteristics listed above for each image. The second 10 are the standard deviation and last 10 are the largest value of each of these characteristics present in each image.Each sample is also associated with a label: a label of value 1 indicates the sample was for malignant (cancerous) tissue. A label of value 0 indicates the sample was for benign tissue. Training a TreeStart by training a basic decision tree. For training, use two data sets. Data set 1 uses 80% of the data for training and 20% for test. Data set 2 uses 90% of the data for training and 10% for test.<jupyter_code># load first dataset
train_1 = pd.read_csv('cancer_datasets/training_1.csv')
val_1 = pd.read_csv('cancer_datasets/validation_1.csv')
test_1 = pd.read_csv('cancer_datasets/testing_1.csv')

feature_names = train_1.columns[1:]
X_train = train_1.values[:, 1:]
y_train = train_1.values[:, 0]
X_val = val_1.values[:, 1:]
y_val = val_1.values[:, 0]
X_test = test_1.values[:, 1:]
y_test = test_1.values[:, 0]
X_train.shape, X_val.shape, X_test.shape
# load second data set
train = pd.read_csv('cancer_datasets/training_2.csv')
val = pd.read_csv('cancer_datasets/validation_2.csv')
test = pd.read_csv('cancer_datasets/testing_2.csv')

feature_names = train.columns[1:]
X_train2 = train.values[:, 1:]
y_train2 = train.values[:, 0]
X_val2 = val.values[:, 1:]
y_val2 = val.values[:, 0]
X_test2 = test.values[:, 1:]
y_test2 = test.values[:, 0]
X_train2.shape, X_val2.shape, X_test2.shape
# define a function to automate the process of training and evaluating each dataset
def report_training(X_train0, y_train0, X_test0, y_test0, criterion = 'gini'):
    print('Find optiaml split using criterion:', criterion)
    cancertree = decisionTree()
    cancertree.grow(X_train0, y_train0, criterion)
    print("total number of nodes: ", cancertree.count_nodes())
    print("total number of leaves: ", cancertree.count_leaves())

    y_pred = cancertree.predict(X_test0)
    acc_test = cancertree.cal_accuracy(y_pred, y_test0)
    y_pred_train = cancertree.predict(X_train0)
    acc_train = cancertree.cal_accuracy(y_pred_train, y_train0)
    print("Accuracy of training data:", acc_train)
    print('Accuracy of testing data:', acc_test)

# fit training set 1 to the tree
print("Decision tree using data set 1")
report_training(X_train, y_train, X_test, y_test, 'entropy')
# fit training set 2 to the tree
print("Decision tree using data set 2")
report_training(X_train2, y_train2, X_test2, y_test2, 'entropy')
# fit training set 1 to the tree
print("Decision tree using data set 1")
report_training(X_train, y_train, X_test, y_test, 'gini')
# fit training set 2 to the tree
print("Decision tree using data set 2")
report_training(X_train2, y_train2, X_test2, y_test2, 'gini')<jupyter_output>Decision tree using data set 2
Find optiaml split using criterion: gini
total number of nodes:  33
total number of leaves:  17
Accuracy of training data: 1.0
Accuracy of testing data: 0.9122807017543859<jupyter_text>InterpretationFrom the result, we can see that dats set 2 is achieving better performance than data set 1, measured by accuracy of testing set. This is because data set 2 uses more data for training, which can beat overfitting in a way. The accuracy of training data is all 1, since decision tree has large power of expression. But it is also prone to overfitting, thus resulting in accuracy only arround 0.9 in testing sets.  Pruning a treeContinue iterations until a degenerate tree with only a single root node remains. For each tree that is produced, calculate the classification accuracy for that tree on the training, validation and testing datasets.<jupyter_code>from sklearn.metrics import precision_recall_fscore_support
import matplotlib.pyplot as plt

# define a function that demontrates pruning results given the dataset
def draw_pruning_results(X_train0, y_train0, X_test0, y_test0, X_val0, y_val0, criterion = 'gini'):
    num_leaves = []
    all_accuracy = []  # store training, validaiton, and testing accuracy
    all_fscore = [] #storetraining, validaiton, and testing f score
    all_precision = [] #storetraining, validaiton, and testing precision
    all_recall = []  #storetraining, validaiton, and testing recall

    # strore the best tree
    best_tree_acc = None
    best_tree_f1 = None
    best_val_accu = float('-inf')
    best_val_f1 = float('-inf')
    
    # the initial complete tree
    current_tree = decisionTree()
    current_tree.grow(X_train0, y_train0, criterion)
    # iteratively prune the tree by one best node
    while current_tree:
        num_leaves.append(current_tree.count_leaves())    
        # predict
        yh_test = current_tree.predict(X_test0)
        yh_val = current_tree.predict(X_val0)
        yh_train = current_tree.predict(X_train0)
        # calcualte metrics
        all_accuracy.append([current_tree.cal_accuracy(yh_train, y_train0), 
                             current_tree.cal_accuracy(yh_val, y_val0),
                            current_tree.cal_accuracy(yh_test, y_test0)])
        prec_train, rec_train, fscre_train, _ = precision_recall_fscore_support(y_train0, yh_train, pos_label=1, average='binary')
        prec_val, rec_val, fscre_val,_ = precision_recall_fscore_support(y_val0, yh_val, pos_label=1, average='binary')
        prec_test, rec_test, fscre_test,_ = precision_recall_fscore_support(y_test0, yh_test, pos_label=1, average='binary')      
        all_fscore.append([fscre_train, fscre_val, fscre_test])
        all_precision.append([prec_train, prec_val, prec_test])
        all_recall.append([rec_train, rec_val, rec_test])
        
        # update best tree
        val_acc = current_tree.cal_accuracy(yh_val, y_val0)
        if val_acc >= best_val_accu:
            best_tree_acc = current_tree
            best_val_accu = val_acc
        if fscre_val >= best_val_f1:
            best_tree_f1 = current_tree
            best_val_f1 = fscre_val
        # prune the tree
        current_tree = current_tree.pruneSingleGreedyNode(X_val0, y_val0)

    # plot metrics
    f, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
    x_leaves = np.array(num_leaves)
    ax1.plot(x_leaves, np.array(all_accuracy)[:,0], label = "training")
    ax1.plot(x_leaves, np.array(all_accuracy)[:,1], label = "validation")
    ax1.plot(x_leaves, np.array(all_accuracy)[:,2], label = "testing")
    ax1.set_title('accuracy vs. number of leaves')
    ax1.legend()
    ax1.set_xlabel('number of leaves')
    ax1.set_ylabel('accuracy')
    ax1.set_xlim([1, max(x_leaves)])
    ax1.set_ylim([0, 1])
    
    ax2.plot(x_leaves, np.array(all_fscore)[:,0], label = "training")
    ax2.plot(x_leaves, np.array(all_fscore)[:,1], label = "validation")
    ax2.plot(x_leaves, np.array(all_fscore)[:,2], label = "testing")
    ax2.set_title('F score vs. number of leaves')
    ax2.legend()
    ax2.set_xlabel('number of leaves')
    ax2.set_ylabel('F score')
    ax2.set_xlim([1, max(x_leaves)])
    ax2.set_ylim([0, 1])
    plt.show()
    return [best_tree_acc, best_tree_f1]
# prune tree of dataset 1, gini
print('pruning results using data set 1')
[pruned_acc_tree1, pruned_f1_tree1] = draw_pruning_results(X_train, y_train, X_test, y_test, X_val, y_val, 'gini')
# prune tree of dataset 2, gini
print('pruning results using data set 2')
[pruned_acc_tree2, pruned_f1_tree2] = draw_pruning_results(X_train2, y_train2, X_test2, y_test2, X_val2, y_val2,'entropy' )
pr1 = sum(y_val == 1) / len(y_val)
pr2 = sum(y_val2 == 1) / len(y_val2)
print("positive rate in validation set 1 is",  pr1)
print("positive rate in validation set 2 is", pr2)
# PRC and ROC curve
from sklearn.metrics import precision_recall_curve, roc_curve, auc
from inspect import signature
def plot_curves(acc_tree, f1_tree, X_test0, y_test0):
    #compute PR curve
    y_score_acc = acc_tree.predict(X_test0)
    y_score_f1 = f1_tree.predict(X_test0)
    precision_acc, recall_acc, _ = precision_recall_curve(y_test0, y_score_acc)
    precision_f1, recall_f1, _ = precision_recall_curve(y_test0, y_score_f1)

    step_kwargs = ({'step': 'post'}
                   if 'step' in signature(plt.fill_between).parameters
                   else {})

    f, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
    ax1.step(recall_acc, precision_acc, color='b', alpha=0.2,
             where='post')
    ax1.fill_between(recall_acc, precision_acc, alpha=0.2, color='b', **step_kwargs)
    ax1.set_title('PR curve of accuracy pruned tree')
    ax1.set_xlabel('Recall')
    ax1.set_ylabel('Precison')

    ax2.step(recall_f1, precision_f1, color='b', alpha=0.2,
             where='post')
    ax2.fill_between(recall_f1, precision_f1, alpha=0.2, color='b', **step_kwargs)
    ax2.set_title('PR curve of F1 score pruned tree')
    ax2.set_xlabel('Recall')
    ax2.set_ylabel('Precison')
    plt.show()

    # compute ROC curve
    fpr_acc, tpr_acc, _ = roc_curve(y_test0, y_score_acc)
    roc_auc_acc = auc(fpr_acc, tpr_acc)
    fpr_f1, tpr_f1, _ = roc_curve(y_test0, y_score_f1)
    roc_auc_f1 = auc(fpr_f1, tpr_f1)

    f, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
    ax1.plot(fpr_acc, tpr_acc, color='darkorange', label='ROC curve (area = %0.2f)' % roc_auc_acc)
    ax1.plot([0, 1], [0, 1], color='navy', linestyle='--')
    ax1.set_title('ROC curve of accuracy pruned tree')
    ax1.set_xlabel('False Positive Rate')
    ax1.set_ylabel('True Positive Rate')
    ax1.legend()

    ax2.plot(fpr_f1, tpr_f1, color='darkorange', label='ROC curve (area = %0.2f)' % roc_auc_f1)
    ax2.plot([0, 1], [0, 1], color='navy', linestyle='--')
    ax2.set_title('ROC curve of F1 score pruned tree')
    ax2.set_xlabel('False Positive Rate')
    ax2.set_ylabel('True Positive Rate')
    ax2.legend()
    plt.show()

print('resuls for dataset 1, validation set. ')
print('The left column is the best tree selected by accuracy, and the right by F1 score.')
plot_curves(pruned_acc_tree1, pruned_f1_tree1, X_val, y_val)
print('resuls for dataset 1, test set')
print('The left column is the best tree selected by accuracy, and the right by F1 score.')
plot_curves(pruned_acc_tree1, pruned_f1_tree1, X_test, y_test)
# 'resuls for dataset 2'
print('results for dataset 2, validation set')
print('The left column is the best tree selected by accuracy, and the right by F1 score.')
plot_curves(pruned_acc_tree2, pruned_f1_tree2, X_val2, y_val2)
# 'resuls for dataset 2'
print('results for dataset 2, testing set')
print('The left column is the best tree selected by accuracy, and the right by F1 score.')
plot_curves(pruned_acc_tree2, pruned_f1_tree2, X_test2, y_test2)<jupyter_output>results for dataset 2, testing set
The left column is the best tree selected by accuracy, and the right by F1 score.<jupyter_text>Interperatationgenerally, with the number of leaf nodes doing down, the accuary of testing set first goes up, then goes down. This is because initially we have a fully grown tree, with is likely to be overfitting. In such situation, training accuracy is high, but testing accuracy is low. As we prune it, the tree are not fitting too much noise in the training data, and performs better. But if we prune it too much, it becomes underfitting, and can't predict well. The F score drops to zero when there is only one node, because that one node predicts all samples to be negative, thus precision and f score is set to be 0.  The first validaiton set is unbalanced, having over 90% negative rate. Thus the validation accuracy always stays high. In this way the accuracy is not informative, and is not a good indicator of the tree performance. Judging by both ROC curve and PRC curve, the best pruned tree selected by validation accuracy performs worse than the tree selected by validation F1 score, having less area. As a comparasion, the seconde validation set is balaced, having 40% positive rate. Thus the two measures generate trees with the same performance.  Draw the Tree<jupyter_code>print("The original tree trained from dataset 1: \n")
current_tree = decisionTree()
current_tree.grow(X_train, y_train)
current_tree.print_tree(feature_names)
print("Best pruned tree selected by validaiton accuracy, dataset 1: \n")
pruned_acc_tree1.print_tree(feature_names)
print("Best pruned tree selected by validaiton F1 score, dataset 1: \n")
pruned_f1_tree1.print_tree(feature_names)
print("The original tree trained from dataset 2: \n")
current_tree = decisionTree()
current_tree.grow(X_train2, y_train2)
current_tree.print_tree(feature_names)
print("Best pruned tree selected by validaiton accuracy, dataset 2: \n")
pruned_acc_tree2.print_tree(feature_names)
print("Best pruned tree selected by validaiton accuracy, dataset 2: \n")
pruned_f1_tree2.print_tree(feature_names)<jupyter_output>Best pruned tree selected by validaiton accuracy, dataset 2: 

Number_of_Concavities__max<=0.14
├── Perimeter__max<=127.2
│   ├── Area__stddev<=54.74
│   │   ├── Smoothness__stddev<=0.0
│   │   │   ├── 1
│   │   │   └── Area__max<=929.8
│   │   │       ├── 0
│   │   │       └── 1
│   │   └── 1
│   └── 1
└── Perimeter__mean<=71.82
    ├── 0
    └── 1
{'fraction_non_alphanumeric': 0.09852395840907438, 'fraction_numerical': 0.028284737875372647, 'mean_word_length': 4.378959718420023, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '314263', 'license': 'MIT', 'n_tokens_mistral': 4882, 'n_tokens_neox': 4668, 'n_words': 1451, 'path': '1_decisionTree/train_and_evaluation.ipynb', 'repo_name': 'yuehu9/Machine-Learning-Porjects', 'size': '13753'}