<jupyter_start><jupyter_text>Evaluation for Skewed Dataset **CS5483 Data Warehousing and Data Mining**___<jupyter_code>%reset -f
from IPython import display
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# produce vector inline graphics
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')
from weka.core import dataset
import weka.core.jvm as jvm
from weka.core.converters import Loader
from weka.classifiers import Classifier, Evaluation
from weka.core.classes import Random
from weka.core.classes import complete_classname<jupyter_output><empty_output><jupyter_text>Class imbalance problem In this notebook, we will analyze a skewed dataset for detecting microcalcifications in mammograms. The goal is to build a classifier to identify whether a bright spot in a mammogram is a micro-calcification, which is an early sign of breast cancer as shown below.  The dataset can be downloaded from[OpenML](https://www.openml.org/d/310) in [ARFF format](https://www.openml.org/data/download/52214/phpn1jVwe). The following loads the data using `python-weka-wrapper`.<jupyter_code>jvm.start() 
loader = Loader(classname="weka.core.converters.ArffLoader")
data = loader.load_url('https://www.openml.org/data/download/52214/phpn1jVwe')
data.class_is_last()
print(data.summary(data))<jupyter_output><empty_output><jupyter_text>There are 7 attributes and over 11 thousand instances. To understand the dataset, refer to Section 4 of the original paper:- Woods, <NAME>., et al. "Comparative evaluation of pattern recognition techniques for detection of microcalcifications." Biomedical Image Processing and Biomedical Visualization. Vol. 1905. International Society for Optics and Photonics, 1993. [(Available via CityU VPN.)](https://www.worldscientific.com/doi/abs/10.1142/9789812797834_0011) A set of 24 mammograms were segmented to locate small bright spots, which are the candidates for the classification of malignant clusters of micro-calcifications. The classification problem is based on attributes of the image segments:- Area (number of pixels)- Average grey level- Gradient strength (of perimeter pixels)- Root mean square noise (fluctuation of the pixel values)- Root mean square noise of local background- Contrast (average grey level minus average of a 2-pixel wide border)- (Low order moment-based) Shape descriptor To compute the 10-fold cross-validation accuracy for `J48`:<jupyter_code>clf = Classifier(classname="weka.classifiers.trees.J48")
evl = Evaluation(data)
evl.crossvalidate_model(clf, data, 10, Random(1))

print(f'Accuracy: {evl.percent_correct:.3g}%')<jupyter_output><empty_output><jupyter_text>You should see that the accuracy is close to 100%. To show the detailed confusion matrix:<jupyter_code>confusion_matrix = pd.DataFrame(evl.confusion_matrix, dtype = int,
    columns=[ f'predicted class "{v}"' for v in data.class_attribute.values],
    index=[f'class "{v}"' for v in data.class_attribute.values])
confusion_matrix<jupyter_output><empty_output><jupyter_text>Each row of the confusion matrix corresponds to a class value (1: malignant, -1: benign) and each column corresponds to a predicted class. Each entry is a count of instances belonging to certain class and having certain predicted class. **Exercise** Assign to `percent_of_malignant_detected` the percentage of instances of class 1 predicted as class 1:<jupyter_code># YOUR CODE HERE
raise NotImplementedError()
print(
    f'Percentage of malignant detected: {percent_of_malignant_detected:.3g}%'
)
# tests<jupyter_output><empty_output><jupyter_text>Different Performance Metrics Since a high accuracy can be misleading for a skewed dataset, we should use other performance metrics to evaluate the performance of a classification algorithm:<jupyter_code>pos_class = 1  # specify the postive class value
performance = {
    'precision': evl.precision(pos_class),
    'recall': evl.recall(pos_class),
    'specificity': evl.true_negative_rate(pos_class)
}
performance<jupyter_output><empty_output><jupyter_text>Although specificity is close to 100%, precision and recall are below 80% and 60% respectively. These metrics can be computed from the numbers of true/false positives/negatives:\begin{align}\text{precision} &= \frac{\text{TP}}{\text{TP} + \text{FP}}\\ \text{recall} &= \frac{\text{TP}}{\text{TP} + \text{FN}}\\ \text{specificity} &= \frac{\text{TN}}{\text{TN} + \text{FP}}\end{align}<jupyter_code>TP = evl.num_true_positives(pos_class)
FN = evl.num_false_negatives(pos_class)
FP = evl.num_false_positives(pos_class)
TN = evl.num_true_negatives(pos_class)

assert np.isclose(performance['precision'], TP / (TP + FP))
assert np.isclose(performance['recall'], TP / (TP + FN))
assert np.isclose(performance['specificity'], TN / (TN + FP))

TFPN = pd.DataFrame(
    [[TP, FN],
     [FP, TP]],
    dtype = int,
    columns=['predicted +ve', 'predicted -ve'],
    index=['+ve', '-ve'])
TFPN<jupyter_output><empty_output><jupyter_text>The above table is not the same as a confusion matrix since a confusion matrix- does not specify a positive class, and- can have more than two rows/columns in multi-class classification problems. **Exercise** Modify `performance` to include the negative predictive value. You can add the value using:```Pythonperformance['NPV'] = ...```<jupyter_code># YOUR CODE HERE
raise NotImplementedError()
print(f"negative predictive value (NPV): {performance['NPV']:.3g}")
# tests<jupyter_output><empty_output><jupyter_text>$F_{\beta}$ score is another measure that captures the performance in both precision and recall:\begin{align}F_{\beta} &:= \left( \frac{\text{precision}^{-1} + \beta^2 \cdot \text{recall}^{-1}}{\beta^2 + 1}\right)^{-1} = \frac{(\beta^2+1)\cdot \text{precision}\cdot \text{recall} }{\beta^2\text{precision} + \text{recall}}\\F := F_1 &= \left( \frac{\text{precision}^{-1} + \text{recall}^{-1}}{2}\right)^{-1} = \frac{2\cdot \text{precision}\cdot \text{recall} }{\text{precision} + \text{recall}},\end{align}which is the harmonic mean of precision and recall. This is useful in training a classifier to maximize both precision and recall.<jupyter_code>performance['F'] = evl.f_measure(pos_class)
print(f"F-score: {performance['F']:.3g}")<jupyter_output><empty_output><jupyter_text>**Exercise** For the $F_{\beta}$ score, we can increase $\beta$ to put more weight on recall. Modify `performance` to include the $F_2$ score. You can set the value using:```Pythonperformance['F_2'] = ...```<jupyter_code># YOUR CODE HERE
raise NotImplementedError()
print(f"F_2 score: {performance['F_2']:.3g}")
# tests<jupyter_output><empty_output><jupyter_text>**Exercise** Using `ZeroR` as the classifier, assign to `ZeroR_performance` a dictionary of precision, recall, and specificity. You can create the dictionary as follows:```PythonZeroR_performance = {    'precision': ___,    'recall': ___,    'specificity': ___}```Use 10-fold cross-validation with a random seed of `1`. If the value is not a number, you may enter it as `np.nan`.<jupyter_code># YOUR CODE HERE
raise NotImplementedError()
ZeroR_performance
# tests<jupyter_output><empty_output><jupyter_text>**Exercise** Is ZeroR a good baseline classifier?[*Hint: Is the accuracy misleading? Can a random decision maker do better than zeroR?*] YOUR ANSWER HERE  Operating Curves for Probabilistic Classifier For a probabilistic classifier that returns probabilities of different classes, we can obtain a trade-off between precision and recall by changing a threshold $\gamma$ for positive prediction, i.e., predict positive if and only if the probability estimate for positive class is larger than $\gamma$. The following plots the [precision-recall curve](https://waikato.github.io/weka-wiki/roc_curves/) and prints the area under the curve.<jupyter_code>import weka.plot.classifiers as plcls

plcls.plot_prc(evl, class_index=[1])
performance['PRC'] = evl.area_under_prc(pos_class)
print(f"area under precision-recall curve (PRC): {performance['PRC']:.3g}")<jupyter_output><empty_output><jupyter_text>**Exercise** One can operate the classifier at any point on the curve by an appropriate choice of $\gamma$. Is it a good idea to operate at the initial part of the curve where the slope is strictly positive? YOUR ANSWER HERE **Exercise** The above curve shows that the classifier can achieve 100% recall but not 100% precision. If you can choose any classifier, is it always possible to achieve 100% recall for any given data set? How about 100% precision? YOUR ANSWER HERE We can also plot the ROC (receiver operator characteristics) curve to show the trade-off between recall (true positive rate) and false positive rate:<jupyter_code>plcls.plot_roc(evl, class_index=[1])
performance['AUC'] = evl.area_under_roc(pos_class)
print(f"area under ROC curve (AUC): {performance['AUC']:.3g}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1114250197762459, 'fraction_numerical': 0.014012882811617132, 'mean_word_length': 6.040572792362768, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 5, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '104989', 'license': 'MIT', 'n_tokens_mistral': 2814, 'n_tokens_neox': 2597, 'n_words': 904, 'path': 'Tutorial5/Evaluation for Skewed Dataset.ipynb', 'repo_name': 'ccha23/cs5483', 'size': '8850'}