<jupyter_start><jupyter_text>Evaluating Classifiers<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>* **The first model** predicts a high risk of cancer for 800 out of 1000 patients’ images in the test set.* Of these 800 images 50 actually show signs of skin cancer.* Hence, all problematic images are correctly identified.<jupyter_code>TP = 50
TN = 200
FN = 0
FP = 750

acc = (TP + TN) / 1000
acc   # not such a great metric when there is class imbalance (in y_train)
# confusion matrix
data = [[TN, FN],
        [FP, TP]]

pd.DataFrame(data, columns=['real 0', 'real 1'], index=['predict 0', 'predict 1'])
# precision and recall

# precision: how many of our positive predictions (cancers) are really positives
# 0.0 .. 1.0
precision = TP / (TP + FP)
precision
# recall: how many of the real positives (cancers) do we detect?
# 0.0 .. 1.0
recall = TP / (TP + FN)
recall<jupyter_output><empty_output><jupyter_text>* **The second classifier** categorizes 100 out of 1000 images into the high risk group. * 40 of the 100 images show real signs of cancer. * 10 images are not identified and falsely classified as low-risk.<jupyter_code>TP = 40
TN = 890
FP = 60
FN = 10

acc = (TP + TN) / (TP + TN + FP + FN)
acc<jupyter_output><empty_output><jupyter_text>accuracy = correct predictions / total predictions<jupyter_code># the titanic data is class imbalanced
dead = 550
alive = 350

dead / (alive + dead)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10555174777244689, 'fraction_numerical': 0.045236463331048665, 'mean_word_length': 4.488721804511278, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '494945', 'license': 'MIT', 'n_tokens_mistral': 555, 'n_tokens_neox': 462, 'n_words': 188, 'path': 'week2/EvaluatingClassifiers.ipynb', 'repo_name': 'boraatlay/bootcamp', 'size': '1459'}