<jupyter_start><jupyter_text>Evaluation Metric Testing The point of this notebook is to walk through an evaluation metric taken from one of the kernels posted on [Kaggle](https://www.kaggle.com/wcukierski/example-metric-implementation) to ensure that was it was functioning correctly and gain a deeper undertanding of the [IoU](https://www.kaggle.com/c/data-science-bowl-2018evaluation) metric.<jupyter_code>!pwd
import skimage
import importlib
import numpy as np

from scipy import stats
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import matplotlib as mpl
import pandas as pd
%matplotlib inline
from utils import imaging
from utils import evaluate
importlib.reload(imaging)
importlib.reload(evaluate)<jupyter_output><empty_output><jupyter_text>Notes on method below When we are calculating the the intersection of objects between images we use a 2D histogram of the two images. This takes the two images flattened and compares the pixel values at each location. It reutrns an $\mathbf{n}$ $\times$ $\mathbf{m}$ matrix where $\mathbf{n}$ is the number of true objects and $\mathbf{m}$ is the number of predicted objects. The values of this matrix are counts of the paired pixel values between the two images. So if row 1, column 2 = 50, this means a pixel value of 1 in the true image was given a value of 2 in the predicted image.$\textbf{Note:}$ It doesn't matter if the pixel value of the predicted mask is different than the ground truth mask (unless it is 0). All we care about is that a predict object has pixels that overlaps with a ground truth object.  Evaluate a single image Evaluate a single image to check the result of our evaluation metric is reasonable.<jupyter_code>image_id = '0a7d30b252359a10fd298b638b90cb9ada3acced4e0c0e5a3692013f432ee4e9'
gt_path = imaging.get_path('output_train_1_lab_gt')
seg_path = imaging.get_path('output_train_1_lab_seg')

gt_image_1 = skimage.io.imread(gt_path + image_id + '.png' )
seg_image_1 = skimage.io.imread(seg_path + image_id + '.png' )
f, axarr = plt.subplots(1,2,figsize=(15,15))
axarr[0].imshow(gt_image_1)
axarr[0].set_title('Ground Truth')
axarr[1].imshow(seg_image_1)
axarr[1].set_title('Segmented')<jupyter_output><empty_output><jupyter_text>Evaluate test cases We examine how the evaluation metric peforms in a few scenarios.1. Perfect overlap with nonmatching class labels.2. Not predicting one the ground truth objects (False Negative)3. How a 50% overlap performs with a threshold of .54. Two predicted objects that lay over the ground truth object.<jupyter_code>n = 1000 # matrices will be nxn                                                                                                                             
gt = np.zeros((n,n))
gt[300:700,300:700] = 1
gt[800:850,800:850] = 2


t1 = np.zeros((n,n))
t2 = np.zeros((n,n))
t3 = np.zeros((n,n))
t4 = np.zeros((n,n))                                                                                                                                                 

# perfect prediction
t1[300:700,300:700] = 2
t1[800:850,800:850] = 1


# different labels                                                                                                                                          
t2[300:700,300:700] = 20                                                                                         

# 50% overlap
t3[300:700,500:900] = 1

# Having to small sub regions where the 1 truth region is.
t4[300:500,300:700] = 1 # creating first small sub region
t4[500:700,300:700] = 2 # creating second small sub region
test_cases = [t1,t2,t3,t4]
f, axarr = plt.subplots(1,5,figsize=(15,15))
axarr[0].imshow(gt)
axarr[0].set_title('gt')
axarr[1].imshow(t1)
axarr[1].set_title('t1')
axarr[2].imshow(t2)
axarr[2].set_title('t2')
axarr[3].imshow(t3)
axarr[3].set_title('t3')
axarr[4].imshow(t4)
axarr[4].set_title('t4')
f.tight_layout()<jupyter_output><empty_output><jupyter_text>Test case 1<jupyter_code>evaluate.evaluate_image(gt, t1)<jupyter_output># true nuclei: 2
# predicted pred: 2

thresh	tp	fp	fn	p
0.500	2	0	0	1.000
0.550	2	0	0	1.000
0.600	2	0	0	1.000
0.650	2	0	0	1.000
0.700	2	0	0	1.000
0.750	2	0	0	1.000
0.800	2	0	0	1.000
0.850	2	0	0	1.000
0.900	2	0	0	1.000
0.950	2	0	0	1.000<jupyter_text>As we would hope, inverting the labels has no effect on the evaluation.  Test case 2<jupyter_code>evaluate.evaluate_image(gt, t2)<jupyter_output># true nuclei: 2
# predicted pred: 1

thresh	tp	fp	fn	p
0.500	1	0	1	0.500
0.550	1	0	1	0.500
0.600	1	0	1	0.500
0.650	1	0	1	0.500
0.700	1	0	1	0.500
0.750	1	0	1	0.500
0.800	1	0	1	0.500
0.850	1	0	1	0.500
0.900	1	0	1	0.500
0.950	1	0	1	0.500<jupyter_text>Since there is one correctly predicted object and 1 missed sobject, so 1 TP and 1 FN, the average precision is .5 as expected.  Test case 3<jupyter_code>evaluate.evaluate_image(gt, t3)<jupyter_output># true nuclei: 2
# predicted pred: 1

thresh	tp	fp	fn	p
0.500	0	1	2	0.000
0.550	0	1	2	0.000
0.600	0	1	2	0.000
0.650	0	1	2	0.000
0.700	0	1	2	0.000
0.750	0	1	2	0.000
0.800	0	1	2	0.000
0.850	0	1	2	0.000
0.900	0	1	2	0.000
0.950	0	1	2	0.000<jupyter_text>The object has a 50$\%$ overlap with the ground truth object so the IoU is .3, which gives 0 TPs for all thresholds used.  Test case 4<jupyter_code>evaluate.evaluate_image(gt, t4)<jupyter_output># true nuclei: 2
# predicted pred: 2

thresh	tp	fp	fn	p
0.500	0	2	2	0.000
0.550	0	2	2	0.000
0.600	0	2	2	0.000
0.650	0	2	2	0.000
0.700	0	2	2	0.000
0.750	0	2	2	0.000
0.800	0	2	2	0.000
0.850	0	2	2	0.000
0.900	0	2	2	0.000
0.950	0	2	2	0.000<jupyter_text>Neither predicted object has an IoU with the ground truth object that satisifies any of the thresholds so there are 0 TPs for each threshold.  Evaluate all images Evaluate all images in stage 1 to test the `evaluate_images` function and see the distribution of scores.<jupyter_code>scores = evaluate.evaluate_images(stage_num=1)
scores.head()
f, axarr = plt.subplots(1,2,figsize=(15,5))
axarr[0].hist(scores.score, bins=50)
axarr[0].set_title('Histogram of scores')
axarr[0].set_xlabel('score')
axarr[0].set_ylabel('# of images')
axarr[1].boxplot(scores.score, 0, 'rs', 0)
axarr[1].set_title('Box plot of scores')
axarr[1].set_xlabel('score')
f.tight_layout()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09610139579656667, 'fraction_numerical': 0.11326808920263115, 'mean_word_length': 3.1339522546419096, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '15800', 'license': 'MIT', 'n_tokens_mistral': 2647, 'n_tokens_neox': 2284, 'n_words': 771, 'path': 'notebooks/evaluation_test.ipynb', 'repo_name': 'williamgrimes/nuclei_segmentation', 'size': '6233'}