<jupyter_start><jupyter_text>**Exercise set 8**==============>The goal of this exercise is to gain familiarity with some>classification methods and the different ways we can assess and compare them.**Exercise 8.1**In this exercise, we will consider the[UCI ML Breast Cancer Wisconsin (Diagnostic) dataset](https://goo.gl/U2Uwz2).This data set contains $569$ tumors which have been classifiedas malignant or benign. In addition, $30$ variables have beenmeasured and it is our goal to make a predictive model whichcan classify new tumors as being malignant or benign.An overview of the different variables can be foundon the [`sklearn` website](https://scikit-learn.org/stable/datasets/index.htmlbreast-cancer-dataset).**(a)**  Begin the exercise by loading the raw data, and creating a test set.The data set itself can be loaded directly from `sklearn`as follows:```pythonfrom sklearn.datasets import load_breast_cancerdata = load_breast_cancer()X = data['data'] "Rename" y so that 0 = benign and 1 = malignant:y = [0 if i == 1 else 1 for i in data['target']]class_names = ['benign', 'malignant']print('Classes:')print(class_names)print('Variables:')print(data['feature_names'])```The test set can be created as in previous exercises by using the method`train_test_split` which can be found in the module`sklearn.model_selection`.Create the test setusing $33$\% of the available data points for the test set.What is important to consider when creating this test set?Check that this is indeed satisfied for your test set andfor your training set.<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>**Your answer to question 8.1(a):** *Double click here* **(b)** In this case, we have to determine whatquantity we are going to useto compare the different classification methods. Briefly explainhow the following possible metrics for assessing classifiers arecalculated:* (i)  Accuracy.* (ii)  Precision.* (iii)  Recall.* (iv)  F1.* (v)  The confusion matrix.Can you an example of a classification task where false positivesshould be avoided and false negative are tolerable? Can you givean example of the opposite (i.e. where false positives are tolerableand false negatives should be avoided)?In the following, we will calculate all these metrics for thedifferent classification methods we consider. At the end of theexercise, you will be asked to compare the different classifiersusing these metrics. But before we do that: Which of theaforementioned metrics would you say is most important forthe classification task we have here?<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>**Your answer to question 8.1(b):** *Double click here* **(c)**  Create a $k$-nearest neighbor classifier(This classifier is available from`sklearn.neighbors.KNeighborsClassifier`) with $3$ neighbors andfit it using your training set. Evaluate the classifier using theaforementioned metrics (with the test set) and plot the confusion matrix.(The different metrics are available in the `sklearn.metrics` module. Here, there is also a method, `plot_confusion_matrix` which youcan use for plotting the confusion matrix.)<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>**Your answer to question 8.1(c):** *Double click here* **(d)**  We will now try to optimize the $k$ for a $k$-nearest neighbor classifier.This can be done using the method `GridSearchCV` which we used inexercise set $6$. Use this method to find the "best" value for $k$, anduse the metric you deemed most important in part **(b)** forscoring what "best" is. Consider $k$ values in the range $1 \leq k \leq 10$for your search for the best $k$.Evaluate the classifier with the best $k$ using theaforementioned metrics (with the test set) and plot the confusion matrix.What value for $k$ did you find in this case?<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>**Your answer to question 8.1(d):** *Double click here* **(e)**  Create a decision tree classifier (This classifier is available from`sklearn.tree.DecisionTreeClassifier`)and fit it using your training set.Evaluate the classifier using theaforementioned metrics (with the test set) and plot the confusion matrix.<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>**Your answer to question 8.1(e):** *Double click here* **(f)**  We will alsotry to tune the `DecisionTreeClassifier`by determining the maximum depthwe should use for the tree. Again, you can use the method`GridSearchCV` to optimize the parameter`max_depth` for the `DecisionTreeClassifier`.Use the metric you deemed most importantin question **(b)** and consider depthsin the range `max_depth = range(1, 21)`, and, in addition,a depthwhere you set `max_depth = None` (this lets thetree expand as far down as possible).Evaluate the classifier with the best `max_depth` using theaforementioned metrics (with the test set) and plot the confusion matrix.What is the best `max_depth` you find in this case? Is it differentfrom letting the tree expand as far down as possible and is this as youwould expect?<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>**Your answer to question 8.1(f):** *Double click here* **(g)**  In addition to the classifiers we have considered so far,we will try $3$ alternative classifiers.For these, we will not tryto optimize any extra parameters.For each of the following classifiers:* (i)  Naive Bayes. (Available as `GaussianNB` from `sklearn.naive_bayes`)* (ii)  LDA. (Available as `LinearDiscriminantAnalysis` from `sklearn.discriminant_analysis`)* (iii)  Random forest. (Available as `RandomForestClassifier` from `sklearn.ensemble`)Train the classifier using your training set, and evaluate it usingthe aforementioned metrics (with the test set) and plot the confusion matrix.For the Random forest classifier: Investigate the most importantvariables for this classifier. This information is available through theproperty `feature_importances_` of your`RandomForestClassifier` object.<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>**Your answer to question 8.1(g):** *Double click here* **(h)**  Compare the accuracy, precision, recall and the F1 scores for allthe classifiers you have considered.If you were to select oneclassifier to put into real-life use, which one would you choose and why?<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>**Your answer to question 8.1(h):** *Double click here* **(i)**  Two extra tasks for the curious student:* (i)  Visualize the decision tree you have created. Thiscan be done by using the method `export_graphviz`from `sklearn.tree`, or the method `plot_tree`from `sklearn.tree`. (Please see https://scikit-learn.org/stable/modules/tree.html and https://scikit-learn.org/stable/modules/generated/sklearn.tree.export_graphviz.html )* (ii)  Create an additional classifier using a support vector machine. This isavailable via `SVC` from `sklearn.svm`.<jupyter_code># Your code here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08725879682179341, 'fraction_numerical': 0.006242905788876277, 'mean_word_length': 6.0139303482587065, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 39, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '364024', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1979, 'n_tokens_neox': 1876, 'n_words': 869, 'path': 'exercises_2020/08_Exercise_Set_8.ipynb', 'repo_name': 'sroet/chemometrics', 'size': '7048'}