<jupyter_start><jupyter_text>Kitchen Confusion Matrix A game about classification errors and biasInspired by [Lights, Camera, Algorithm](https://gist.github.com/rshorey/6fbf25b7a35ff67fbda435a1553d9564) by <NAME> and <NAME>.<jupyter_code>import pandas as pd
import numpy.random as random
%matplotlib inline
# Part of getting good results is picking a seed that makes the calibration,
# and overall accuracy, FPV, FNR, and PPV come out well.
# Any particular draw of random data will give us a better or worse situation.
random.seed(121)
#random.seed(123)
df = pd.read_csv('lights-camera-algorithm-2.csv')<jupyter_output><empty_output><jupyter_text>This is the original data for Lights, Camera, Algorithm -- fruits and vegatables plus their feature. I've added the feauture "exotic"<jupyter_code>df.head()
# There are already more "exotic" fruits than vegetables
pd.crosstab(df.truth, df.exotic)
# A function that returns a random True/False, biased so that
# fruits get probability of true = p_fruit and veg gets 1-p_fruit
def fruit_veg_random(fruit_or_veg, p_fruit):
    if fruit_or_veg=='fruit':
        return random.uniform() <= p_fruit
    else:
        return random.uniform() >= p_fruit

# We are going to make further imbalanced columns that are correleted with f/v, thus eaten
df['in_season'] = df['truth'].apply(lambda f_or_v: fruit_veg_random(f_or_v, 0.7))
df['imported'] = df['truth'].apply(lambda f_or_v: fruit_veg_random(f_or_v, 0.2))
# Now we calculate the results of our "classifier"
# Exotic foods are much more likely to be eaten, in season foods are somewhat more,
# imported foods somewhat less (because they are less frehs)
df['score'] = 2*df['exotic'] + df['in_season'] - df['imported']
df['score'].hist(bins=range(-1,5))
# Now generate the ground truth "eaten"
# For teaching purposes, we need two things to be true:
#  - We want the predictor to be good but not perfect. So we base `eaten` on `score`
#    so that the score is calibrated, like a properly trained model.
#  - There is an imblance between fruit and veg. This will come automatically because 
#    we have imbalanced the feature distributions, so the score is unbalanced
# 

# Being able to map the scores to monotonic probabilities makes the classifier "calibrated"
score_probabilities = {
    -1: 0.1,
    0:  0.25,
    1:  0.5,
    2:  0.75,
    3:  0.9
}

df['eaten'] = df['score'].apply(lambda score: random.uniform() <= score_probabilities[score])

pd.crosstab(df.eaten, df.score)
pd.crosstab(df.truth, df.eaten)
# We can now turn this score into a classifier by thresholding at some value
thresh = 1
df['predicted'] =  df.score >= thresh
# Is this classifier balanced?
f = df[df.truth=='fruit'].groupby('predicted').mean()['eaten']
v = df[df.truth=='veg'].groupby('predicted').mean()['eaten']

# We can actually look at the calibration now
a = pd.concat([f,v], axis=1)
a.columns = ['Fruit','Vegetable']
a.plot.bar()<jupyter_output><empty_output><jupyter_text>We can get six different classifiers by setting our score threshold from  3. But 3 is always predict not eaten, so there are really only four thresholds that make sense. Let's see the confusion matrix for one of these levels.<jupyter_code># cm is a confusion matrix. The rows are guessed, the columns are actual 
def print_ppv_fpv(cm):
    # the indices here are [col][row] or [actual][guessed]
    TN = cm[False][False]   
    TP = cm[True][True]
    FN = cm[True][False]
    FP = cm[False][True]
    print('Accuracy: ', (TN+TP)/(TN+TP+FN+FP))
    print('PPV: ', TP / (TP + FP))
    print('FPR: ', FP / (FP + TN))
    print('FNR: ', FN / (FN + TP))
    print()

def print_metrics(guessed, actual):
    cm = pd.crosstab(guessed, actual, rownames=['guessed'], colnames=['actual'])
    print(cm)
    print()
    print_ppv_fpv(cm)    

print('Everyone')
print_metrics(df.predicted, df.eaten)

print('Fruits')
subset = df[df.truth == 'fruit']
print_metrics(subset.predicted, subset.eaten)

print('Vegetables')
subset = df[df.truth == 'veg']
print_metrics(subset.predicted, subset.eaten)
# Cleanup and save
df = df.rename(index=str, columns={"truth": "fruit_or_veg", "fruit": "name"})
df = df[['name','fruit_or_veg','exotic','in_season','imported','score','eaten']]
df.head()
df.to_csv('kitchen-confusion-matrix.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1303538175046555, 'fraction_numerical': 0.012802607076350093, 'mean_word_length': 4.767785234899329, 'pattern_counts': {'":': 2, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '517690', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 1526, 'n_tokens_neox': 1429, 'n_words': 455, 'path': 'week-5/kitchen-confusion-matrix.ipynb', 'repo_name': 'adrianblanco/lede-algorithms', 'size': '4311'}