<jupyter_start><jupyter_text>Image prediction accuracy analysis- <NAME> (MIT License)In this notebook, we'll look at the TSV dump from the mysql db that recorded the prediction accuracies (top1, top5, none) from the webapp.Note: the dialog and some of the choices were specific to my data. The outputs should be sufficiently general that they work with your data, though. Feel free to modify the text and conclusions if it bothers you!<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

plt.style.use('bmh')
%matplotlib inline
project_dir = '/path/to/repo/'

# we don't need the id or URL for this
results = (pd.read_csv(project_dir + 'rdata/db-results-export.tsv', sep='\t')
           .drop(['id','link'], axis=1)
           )

results.head()
# make sure they're in time order (snowflake id encodes timing)
results = results.sort_values('tweet_id', ascending=True)
#results.tail()
# distribution of labels?
results['label_id'].value_counts().sort_index()<jupyter_output><empty_output><jupyter_text>Remember that:```0 = top11 = top52 = None```Also recall that the `3`s are from a bug in the UI buttons, and should be `2`s. I think there were < 10 total uses of the buttons, so there are probably a handful of 2s that are 1s and 1s that are 0s. But since they're one to two OOM less, who cares.Just drop the 3s for now.<jupyter_code>results = results.query('label_id != 3')
results['label_id'].value_counts().sort_index()<jupyter_output><empty_output><jupyter_text>Quick (and unsurprising) take: most of the predictions are incorrect. Given the random things people post on Twitter, this was expected from the outset.  Analyze score distributionsLet's start by looking at the scores and their distributions. Start by recalling the format of the data.We can begin by simply looking at the distributions of the prediction probabilities. Recall that these are the ordered (1 to 5), most-probable predictions for each image. For annoying historical reasons, the remainder of the notebook uses a dataframe named `adj_results`, so make that connection here.<jupyter_code>adj_results = results
adj_results.head()
axes = (adj_results[['score1','score2','score3','score4','score5']]
        .plot.hist(bins=100, 
                   subplots=True, 
                   #sharey=True, 
                   figsize=(8,8))
       )

plt.xlabel('prediction probability')
axes[0].set_title('distribution of top-5 prediction probabilities ({} images)'.format(len(adj_results)))<jupyter_output><empty_output><jupyter_text>These distributions are consistent with the notion that `score1` is always the most probable result, `score2` less so, and so on. There is never a case in which the fifth-most likely label is predicted to be highly likely.Interestingly, note that most of the distribution weight for `score1` (the most probably prediction) is still at very low probability. This indicates that the model is generally not confident in it's predictions for our data set. Another way of saying it is that this out-of-the-box model was not designed for the task of labeling random twitter images, and as such it doesn't do a spectacular job of it. But that's fine, and expected. We're looking to see if we can still get some use out of it.If you uncomment the `sharey` kwarg, you can also see more clearly that all the probability masses do sum to 1, but the `score1` distribution is just much more spread out over the probability range.Note that this view doesn't account for (or display) the ground truth that we applied. Let's encode that in the frame with a text label.<jupyter_code># add a column that maps the label_id to the text labels 
label_dict = {0:'top1', 1:'top5', 2:'none'}

adj_results['truth'] = adj_results['label_id'].apply(lambda x: label_dict[x])

adj_results[['label_id','truth']].head()<jupyter_output><empty_output><jupyter_text>Prediction probability distributions per labelNow that we have the labels, let's use them in a similar set of distributions as above.<jupyter_code>adj_results.head()
cols = ['score1','score2','score3','score4','score5']
labels = ['top1','top5','none']

# make 3 separate charts (one for each text label) 
#  each one will have subplots for the 5 scores
for label in labels:
    axes = (adj_results.query('truth == @label')[cols].plot.hist(bins=100,
                                                                    subplots=True, 
                                                                    #sharey=True, 
                                                                    figsize=(8,6)                                                                    
                                                                    )
            )
    # put the title on the top subplot
    axes[0].set_title('score distribution for label={}'.format(label))

plt.legend()    
plt.xlabel('prediction probability')<jupyter_output><empty_output><jupyter_text>Pretty interesting shapes. Given that the distributions for each label are pretty unique, we could probably train some other model to predict the labels based on all the top 5 scores. Interesting idea, but not the point right now; we want to know more about how to use the `top1` predictions. Of note: in both `top5` and `none`, the `score1` (highest probability prediction) can be as high as 0.9 or more. Similarly, even with the `top1` label, some `score1` values are as low as 0.1. **This makes it clear that we can't blindly use the `score1` probability.**We need something like an ROC curve (FPR v TPR) on `score1` to understand the tradeoffs.  ROC curveHow do we construct an ROC curve for this data? [Wiki ref for calculations](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)- assume `score1` is the prediction probability of a binary classifier- assume ground truth is binary: the label is `top1` or it's not - vary probability decision-making threshold from 0 to 1- at each threshold    - everything above threshold is "prediction positive"    - everything below threshold is "prediction negative"- at all times (fixed)    - actual data with `top1` / "1" labels are the "condition positive"    - everything else is "condition negative"- TPR = TP / cond pos- FPR = FP / cond neg- plot TPR vs. FPR at each threshold value<jupyter_code>sub_df = adj_results[['score1','truth']].copy(deep=True)

# for easier comparisons, let's convert 'top1' to 1 and everything else to 0
sub_df['truth'] = sub_df['truth'].apply(lambda x: 1 if x == 'top1' else 0)

#del sub_df['pred_acc']
sub_df.rename(columns={'score1':'pred'}, inplace=True)

sub_df.head()
# threshold range
t_rng = np.linspace(0,1,100)

# ground truth (fixed)
cond_pos = len(sub_df.query('truth == 1'))
cond_neg = len(sub_df.query('truth == 0'))

# use this copy of the frame above for flipping the preds at each threshold
# `tmp_df` will be our binary label holder
tmp_df = sub_df.copy(deep=True)
tpr_list = []
fpr_list = []

for t in t_rng:
    # flip to 0 or 1 based on this threshold
    tmp_df['pred'] = sub_df['pred'].apply(lambda x: 1 if x > t else 0)
    # calculate TPR (TP / cond pos)
    tp = len(tmp_df.query('pred == 1 and truth == 1'))
    tpr = tp / cond_pos
    tpr_list.append(tpr)
    # calculate FPR (FP / cond neg)
    fp = len(tmp_df.query('pred == 1 and truth == 0'))
    fpr = fp / cond_neg
    fpr_list.append(fpr)

#print('tpr_list: ', tpr_list)
#print('fpr_list: ', fpr_list)
fig = plt.figure(figsize=(6,6))

plt.plot(fpr_list, tpr_list, '--.', label='model')
plt.plot(t_rng, t_rng, ':', c='k', label='diagonal')

plt.xlabel('FPR')
plt.ylabel('TPR')
# rough estimate of auc
auc = sum(tpr_list)/len(tpr_list)
plt.title('ROC curve for top1 predictions (AUC ~ {:.2f})'.format(auc))
plt.legend()<jupyter_output><empty_output><jupyter_text>The last step would be to choose a threshold value for the prediction probability and then say that's where we operate. How do we choose the best threshold?Some googling suggests there the "optimal" choice is typically dependent on the industry and context of the problem. That said, [some texts](https://books.google.com/books?id=JzT_CAAAQBAJ&lpg=PT43&dq=Selecting%20an%20Optimal%20Threshold%20ROC%20curve&pg=PT43v=onepage&q=Selecting%20an%20Optimal%20Threshold%20ROC%20curve&f=false) suggest there are two common answers:- the point that is closest to "ideal" (i.e. (0,1))- the point that is furthest from the "informationless diagonal"Let's calculate both and see what they look like. Closest to "ideal"<jupyter_code># create an array of the TPR and FPR values
fptp_list = []
for f,t in zip(fpr_list, tpr_list):
    fptp_list.append([f, t])

# NB: fptp starts at (1,1) at t=0; reverse order (start at (0,0)) for 
#  same convention as t_rng  
fptp = np.array(fptp_list)[::-1]    
# create an array of the "ideal" values
ideal_list = []
for _ in fptp:
    ideal_list.append([0,1])

ideal = np.array(ideal_list)
# calculate pair-wise (euclidean) distances
ideal_dist = np.array([np.linalg.norm(a-b) for a,b in zip(ideal, fptp)])

# get the index of the smallest distance
ideal_position = np.argmin(ideal_dist)
# ideal_dist should be 1.0 at both ends!
#plt.plot(ideal_dist)<jupyter_output><empty_output><jupyter_text>Furthest from diagonal<jupyter_code># create an array of the diagnoal values
# NB: diag starts at (0,0)
diag = np.array([[x, x] for x in t_rng])
# calculate pair-wise (euclidean) distances
# (reverse fptp since it starts goes upper right => lower left)
#diag_dist = np.array([np.linalg.norm(a-b) for a,b in zip(diag, fptp[::-1])])
diag_dist = np.array([np.linalg.norm(a-b) for a,b in zip(diag, fptp)])

# get the index of the largest distance
diag_position = np.argmax(diag_dist)
# diag_dist should be 0 at both ends!
#plt.plot(diag_dist)<jupyter_output><empty_output><jupyter_text>Compare results<jupyter_code>print('The threshold closest to the "ideal" point in the ROC curve is at:')
print(' (FPR, TPR)={}'.format(fptp[ideal_position]))
print(' with prediction threshold={:.3f}'.format(t_rng[ideal_position]))
print('The threshold furthest from the "informationless diagonal" in the ROC curve is at:')
print(' (FPR, TPR)={}'.format(fptp[diag_position]))
print(' with prediction threshold={:.3f}'.format(t_rng[diag_position]))
fig = plt.figure(figsize=(6,6))

plt.plot(fpr_list, tpr_list, '--.', label='model')
plt.plot(t_rng, t_rng, ':', c='k', label='diagonal')

# diagonal threshold
plt.plot(*fptp[diag_position], 'o', markersize=10, alpha=0.6,
         label='furthest from diagonal (t={:.2f})'.format(t_rng[diag_position])
        )
plt.plot([fptp[diag_position][0], fptp[diag_position][0]],
         fptp[diag_position],
         'k--', alpha=0.5
        )

# ideal threshold
plt.plot(*fptp[ideal_position], 'o', markersize=10, alpha=0.6,
         label='closest to ideal (t={:.2f})'.format(t_rng[ideal_position])
         )
plt.plot([ideal[0][0], fptp[ideal_position][0]],
         [ideal[0][1], fptp[ideal_position][1]],
        'k--', alpha=0.5
        )

plt.xlabel('FPR')
plt.ylabel('TPR')
# rough estimate of auc
auc = sum(tpr_list)/len(tpr_list)
plt.title('ROC curve for top1 predictions (AUC ~ {:.2f})'.format(auc))
plt.legend()<jupyter_output><empty_output><jupyter_text>Since there is some random sampling involved in generating the first region of data, these threshold values can vary a bit. In repeated runs, they typically range from 0.5 to 0.6, and sometimes they are the same point.Let's use the "furthest from the diagonal" point and move on to recalculate other parts of the confusion matrix params.<jupyter_code># set threshold
t = t_rng[diag_position]

# ground truth still fixed at cond_pos, cond_neg

# use this copy of the frame above for flipping the preds at each threshold
tmp_df = sub_df.copy(deep=True)
tpr_list = []
fpr_list = []

# flip to 0 or 1 based on this threshold
tmp_df['pred'] = sub_df['pred'].apply(lambda x: 1 if x > t else 0)
# calculate TPR (TP / cond pos)
tp = len(tmp_df.query('pred == 1 and truth == 1'))
tpr = tp / cond_pos
tpr_list.append(tpr)
# calculate FPR (FP / cond neg)
fp = len(tmp_df.query('pred == 1 and truth == 0'))
fpr = fp / cond_neg
fpr_list.append(fpr)
# calculate TN + FN
tn = len(tmp_df.query('pred == 0 and truth == 0'))
fn = len(tmp_df.query('pred == 0 and truth == 1'))
# accuracy 
accuracy = (tp + tn) / len(tmp_df)

print('total accuracy: {:.3f}'.format(accuracy))<jupyter_output><empty_output><jupyter_text>Common categories of predictionsOk, so we've got a mid- to high-70s% accuracy image classifier by setting the threshold (reflected in `tmp_df`), but it's possible that there are class imbalances or other sorts of bias in the *types* of images we can accurately predict. What are the most common accurately predicted images, and what fraction of the data do they comprise?We can use the index of tmp_df to join back onto the larger frame with the actual labels.**Most common TP labels**<jupyter_code># tmp_df still has our binary labels

# most common labels for TP
(tmp_df.query('pred == 1 and truth == 1')
 #.head()
 .join(adj_results[['tweet_id','keyword1']], how='left')
 .groupby(by='keyword1')
 .count()
 .sort_values('tweet_id', ascending=False)[['tweet_id']]
 .rename(columns={'tweet_id':'count'})
 .head(15)
 )<jupyter_output><empty_output><jupyter_text>From this, we see some data that feels consistent with my experience hand-labeling things: the model is really good at recognizing suits and websites in images. Recall that in my hand-labeling, I counted anything that was a mobile app or deskstop screenshot as "web_site". Then, however, the counts drop off pretty quickly.**Most common FP categories**<jupyter_code># tmp_df still has our binary labels

# most common labels for FP
(tmp_df.query('pred == 1 and truth == 0')
 #.head()
 .join(adj_results[['tweet_id','keyword1']], how='left')
 .groupby(by='keyword1')
 .count()
 .sort_values('tweet_id', ascending=False)[['tweet_id']]
 .rename(columns={'tweet_id':'count'})
 .head(15)
 )<jupyter_output><empty_output><jupyter_text>Not a huge shock, there are a bunch of screenshots, so web_site also is at the top of the false positive list. I'm not sure what to make of envelope and menu making it into the top of that list, either.**Most common FN categories**<jupyter_code># tmp_df still has our binary labels

# most common labels for FN
(tmp_df.query('pred == 0 and truth == 1')
 #.head()
 .join(adj_results[['tweet_id','keyword1']], how='left')
 .groupby(by='keyword1')
 .count()
 .sort_values('tweet_id', ascending=False)[['tweet_id']]
 .rename(columns={'tweet_id':'count'})
 .head(15)
 )<jupyter_output><empty_output><jupyter_text>**Most common TN categories**<jupyter_code># tmp_df still has our binary labels

# most common labels for TN
(tmp_df.query('pred == 0 and truth == 0')
 #.head()
 .join(adj_results[['tweet_id','keyword1']], how='left')
 .groupby(by='keyword1')
 .count()
 .sort_values('tweet_id', ascending=False)[['tweet_id']]
 .rename(columns={'tweet_id':'count'})
 .head(15)
 )<jupyter_output><empty_output><jupyter_text>ConclusionSo, is this image classifier useful? Maybe. It's pretty noisy, though. I do think there's probably utility in attaching labels to tweets when above, say, one of these thresholds. To see what you think: - run the cell below- copy/paste the link and compare to the corresponding `keyword1`    - (it's ok that it has my username, it works anyway!)Re-run the cell as many times as necessary.<jupyter_code># for true positives
tw_id, kw1 = (tmp_df.query('pred == 1 and truth == 1')
# for false positives
#tw_id, kw1 = (tmp_df.query('pred == 1 and truth == 0')
                 .join(adj_results[['tweet_id','keyword1']], how='left')
                 .sample(n=1)[['tweet_id','keyword1']].values[0]
                 )

print('prediction: {}\n'.format(kw1))
print('URL to copy-paste: {}'.format('https://www.twitter.com/jrmontag/status/{}'.format(tw_id)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12050699962164207, 'fraction_numerical': 0.014188422247446084, 'mean_word_length': 4.325386165211551, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 65, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '31994', 'license': 'MIT', 'n_tokens_mistral': 5222, 'n_tokens_neox': 4950, 'n_words': 1759, 'path': 'bin/prediction-analysis.ipynb', 'repo_name': 'jrmontag/tweet-image-labels', 'size': '15865'}