<jupyter_start><jupyter_text>Exercise Sheet 10 Loading and pre-processing of data<jupyter_code>import pandas as pd
from sklearn.datasets import load_breast_cancer
cancer = load_breast_cancer()
# some preprocessing
X = pd.DataFrame(cancer.data, columns = cancer.feature_names)
Y = pd.DataFrame(cancer.target, columns = ["class"])
Y = Y.replace({0: "malignant", 1 : "benign"})
class_names = ['malignant','benign']<jupyter_output><empty_output><jupyter_text>A nice explaination about decision trees and random forests can be found here: https://towardsdatascience.com/an-implementation-and-explanation-of-the-random-forest-in-python-77bf308a9b76  Question 1 - Decision Trees  Q1 - 1<jupyter_code>from sklearn.tree import DecisionTreeClassifier
# hint: https://scikit-learn.org/stable/modules/tree.html

#Creating the classifier object wiht a max depth of 2
clf = DecisionTreeClassifier(max_depth=2, random_state=20)

# Fitting the model
clf.fit(X,Y)<jupyter_output><empty_output><jupyter_text>Q1 - 2<jupyter_code>import graphviz 
from sklearn import tree

my_data = tree.export_graphviz(clf, out_file=None, 
                      feature_names=X.columns,
                      class_names = class_names,
                      filled=True, rounded=True,
                      special_characters=True)  
graph = graphviz.Source(my_data) 
graph<jupyter_output><empty_output><jupyter_text>Q1 - 3  a) *   In case of sample i, worst radius = 10, which is less than that of the root value. So we traverse the left subtree of the root node. Then we check for the worst concave points which is 0.1 and compare it with 0.136. Since it is smaller, we again traverse the left subtree. So, checking the third condition is immaterial now and we choose the class as **malignant**.*   In case of sample ii, worst radius = 20. We compare it with the root node value which is 16.795. Since, it is greeater than the root node value, we traverse the right subtree of the root node. Then we check for the mean texture which is 35 and compare it with 16.11. Since, it is again greater, we traverse the right subtree again. So, now checking the third condition is immaterial now and we choose the class as **benign**.  b) *   In case of sample iii, worst radius = 12, so we traverse the left subtree of the root node. Then we check for the worst concave points which is 0.25 and hence we traverse the right subtree. So, now checking the third condition is immaterial now and we choose the class as **benign**.*   In case of sample iv, worst radius = 35, so we traverse the right subtree of the root node. Then we check for the mean texture which is 16 and hence we traverse the left subtree. So, now checking the third condition is immaterial now and we choose the class as **malignant**.  Q1 - 4<jupyter_code>from sklearn.model_selection import train_test_split

# Randomly splitting the data into training and testing with a ratio of 3:1

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25, random_state = 20)<jupyter_output><empty_output><jupyter_text>Q1 - 5<jupyter_code>from sklearn.model_selection import GridSearchCV

# Hyperparameter list for optimization
params = {'max_leaf_nodes': list(range(2, 100)), 'min_samples_split': [2, 3, 4]}

# Creating the object for the decision tree classifier
clf2 = DecisionTreeClassifier(random_state=20)

# Instantiating the GridSearchCV object
clf2_cv = GridSearchCV(clf2,params, cv=5)

# Fitting it to data
clf2_cv.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Q1 - 6<jupyter_code># Displaying the best parameters after the 5-fold cross validation
print("The best parameters are as follows")
print(clf2_cv.best_params_)
print("\n")
print("The best score for the best parameters is " + str(clf2_cv.best_score_))<jupyter_output>The best parameters are as follows
{'max_leaf_nodes': 7, 'min_samples_split': 2}


The best score for the best parameters is 0.94593707250342<jupyter_text>*   Hence, the best hyperparameters are max_leaf_nodes = 7 and min_samples_split = 2. *   Also the best score for the best parameters  = 0.946  Q1 - 7<jupyter_code>from sklearn.metrics import accuracy_score

# Training the model on the hyperparameters found
clf3 = DecisionTreeClassifier(random_state=20, max_leaf_nodes=7, min_samples_split=2).fit(X_train,y_train)

# Testing it on the test data
y_pred = clf3.predict(X_test)
accuracy = accuracy_score(y_test,y_pred)

# Displaying the test accuracy
print("The model's accuracy on the test set is " + str(accuracy))<jupyter_output>The model's accuracy on the test set is 0.9020979020979021<jupyter_text>Q1 - 8 The disadvantages of decision tree is as follows.*   **Lack of smoothness**.*   **Lack of stability, high variance** - A small change in the data can cause a large change in the structure of the decision tree causing instability. *   It prone to **overfitting**.*   **Affected by noise** - Little bit of noise can make it unstable which leads to wrong predictions.*   **Not suitable for large dataset**s - If data size is large, then one single tree may grow complex and lead to overfitting. So in this case, we should use Random Forest instead of a single Decision Tree.To overcome the disadvantages:*   **To overcome over fitting** (too many splits): If we can limit the node size and the depth of the tree by pruning using cost-complexity criterion which defines the trade-off between goodness of fit and model .*   We make the tree more robust by building an ensemble using the bootstarp aggregation where each tree is treated individually which lead to the decrease in variance  and also limit over-fitting without substantially increasing error due to bias; which is called the **random forest**.   Question 2 - Random Forests  Q2 - 1 *   **Forest**  is a collection of many Decision trees rather averaging the prediction of trees.*   **Random** means that we take a random sample of our training data before building the tree and also take a subset of features when splitting .*   A **random forest** is simply a collection of decision trees whose results are aggregated into one final result. Their ability to limit over-fitting without substantially increasing error due to bias is why they are such powerful models. One way Random Forests reduce variance is by training on different samples of the data. It combines output class labels of the individual trees to a final consensus class label. *   Random forest uses **randomness** if there are many features and when we want to classify, random forest uses a subset of all the features for splitting each node in each decision tree. So it combines hundreds of decision trees and trains each one separately using the bootstrapping by taking a limited number of features, then making a final prediction by averaging the prediction of each individual tree. So, independent predictions from multiple decision trees are combined to produce a final classification.  Q2 - 2 There are two popular arguments on why bagging works.*   A limitation of Decision Trees is that they suffer from high variance. Using the method of bagging, the variance is averaged from multiple trees on different variants of the training set thereby improving the overall variance. *   Bagging is capable of having an equalization(also known as secondary effect) on the potential influence on examples. This in turn reduces the variance.From a Bayesian point of view, Bagging works because:*   It is an approximation to the optimal procedure of Bayesian model averaging, with an appropriate implicit prior.*  Bagging effectively shifts the prior to a more appropriate region of model space.   Q2 - 3<jupyter_code># Importing necessary packages
from sklearn.ensemble import RandomForestClassifier
from itertools import chain 
import numpy as np

# Learning a classification of the breast cancer data with the RandomForestClassifier

#Creating the classifier object wiht a max depth of 2
clf4 = RandomForestClassifier(random_state=20)

# Converting target dataframe to numpy array
y = Y.values.tolist()
y = list(chain.from_iterable(y))
y = np.array(y)

# Fitting the classifier to the data
clf4.fit(X,y)
# Hyperparameter list for optimization
param_grid = { 
    'min_samples_split': [2, 3, 4],
    'max_features': ['auto', 'sqrt', 'log2'],
    'max_leaf_nodes': list(range(10, 20)) #run before with wider range, but restricted to a subset for students to reduce run time
}


# Converting target dataframe of y_train to numpy array
y2_train = y_train.values.tolist()
y2_train = list(chain.from_iterable(y2_train))
y2_train = np.array(y2_train)


# Creating the object for the random forest classifier
clf5 = RandomForestClassifier(random_state=20)

# Instantiating the GridSearchCV object
clf5_cv = GridSearchCV(clf5,param_grid, cv=5)

# Fitting it to data
clf5_cv.fit(X_train,y2_train)<jupyter_output><empty_output><jupyter_text>Q2 - 4<jupyter_code># Displaying the best parameters after the 5-fold cross validation
print("The best parameters are as follows")
print(clf5_cv.best_params_)
print("\n")
print("The best score for the best parameters is " + str(clf5_cv.best_score_))<jupyter_output>The best parameters are as follows
{'max_features': 'auto', 'max_leaf_nodes': 15, 'min_samples_split': 2}


The best score for the best parameters is 0.955403556771546<jupyter_text>*   Hence, the best hyperparameters are max_features = auto, max_leaf_nodes = 15 and min_samples_split = 2. *   Also the best score for the best parameters  = 0.955  Q2 - 5<jupyter_code>from sklearn.metrics import accuracy_score

# Fitting the model on the best hyperparameters found
clf6 = RandomForestClassifier(random_state=20, max_features='auto', max_leaf_nodes=15, min_samples_split=2).fit(X_train,y2_train)<jupyter_output><empty_output><jupyter_text>Q2 - 6<jupyter_code># Predicting the classes of the test set
y_pred =clf6.predict(X_test)
y_pred2 = y_pred.tolist()

# Storing the indices of the test dataset
idx = X_test.index.to_list()

for i in range(0,len(idx)):
  print("For index = " + str(idx[i]) + ", class = "+ str(y_pred2[i]))

# Printing the number of samples in test data assigned to each class
d = {x:y_pred2.count(x) for x in y_pred2}
print(d)<jupyter_output>{'benign': 84, 'malignant': 59}<jupyter_text>Q2 - 7<jupyter_code>from sklearn.metrics import confusion_matrix

accuracy = accuracy_score(y_test,y_pred)
cm = confusion_matrix(y_test,y_pred)

# Displaying the test accuracy
print("The model's accuracy on the test set is " + str(accuracy))
print()
# Displaying the confusion matrix
print("The confusion matrix is as follows:")
print(cm)<jupyter_output>The model's accuracy on the test set is 0.972027972027972

The confusion matrix is as follows:
[[83  3]
 [ 1 56]]<jupyter_text>Q2 - 8<jupyter_code># Storing all the feature names in a list
feature_names = X.columns.to_list()

# Extract feature importances
f = pd.DataFrame({'features': feature_names, 'importance': clf6.feature_importances_}).sort_values('importance', ascending = False)

print(f)<jupyter_output>features  importance
7       mean concave points    0.155255
22          worst perimeter    0.121300
23               worst area    0.115884
27     worst concave points    0.109546
6            mean concavity    0.075353
20             worst radius    0.066466
26          worst concavity    0.044549
0               mean radius    0.039732
13               area error    0.038943
3                 mean area    0.034723
2            mean perimeter    0.027355
12          perimeter error    0.025323
10             radius error    0.019157
24         worst smoothness    0.015974
28           worst symmetry    0.015184
1              mean texture    0.014486
21            worst texture    0.014091
5          mean compactness    0.013392
25        worst compactness    0.008686
29  worst fractal dimension    0.005334
4           mean smoothness    0.004986
11            texture error    0.004959
15        compactness error    0.004729
16          concavity error    0.004558
[...]
{'fraction_non_alphanumeric': 0.06993824069437489, 'fraction_numerical': 0.03780671006509765, 'mean_word_length': 4.302212389380531, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '185826', 'license': 'MIT', 'n_tokens_mistral': 3788, 'n_tokens_neox': 3457, 'n_words': 1436, 'path': 'Exercise_10.ipynb', 'repo_name': 'itsroops/Biomedical_Data_Science_and_AI', 'size': '11982'}