<jupyter_start><jupyter_text>Evaluate MegaDetector v2 and v3 on the v3 validation setThis validation set is not the perfect measure of performance for either version, since there are some images in there that are in the training set of MDv2, and that MDv3's checkpoint was selected based on results on ~10% of this validation set.This validation set is relatively noise-less however, since they were bounding box labeled by iMerit.<jupyter_code># using results that exclude the 'person' category, since this is newly added in v3

with open('/Users/siyuyang/Source/temp_data/CameraTrap/megadetectorv3/no_human_images/v2_on_val_v3/per_category_metrics.p', 'rb') as f:
    v2_results = pickle.load(f)
    
with open('/Users/siyuyang/Source/temp_data/CameraTrap/megadetectorv3/no_human_images/v3/per_category_metrics.p', 'rb') as f:
    v3_results = pickle.load(f)
v2_animal = v2_results[0][1]
v2_animal
v2_recall_at_90_prec = utils.find_precision_at_recall(v2_animal['precision'], v2_animal['recall'], recall_level=0.9)
v2_recall_at_90_prec
v3_animal = v3_results[0][1]
v3_animal
v3_recall_at_90_prec = utils.find_precision_at_recall(v3_animal['precision'], v3_animal['recall'], recall_level=0.9)
v3_recall_at_90_prec
# percentage gain in AP
100 * (v3_animal['average_precision'] - v2_animal['average_precision']) / v2_animal['average_precision']
# percentage gain in precision at 90% recall
100 * (v3_recall_at_90_prec - v2_recall_at_90_prec)/v2_recall_at_90_prec<jupyter_output><empty_output><jupyter_text>Plot precision-recall for the `animal` category on one graph<jupyter_code>plt.figure(figsize=(5, 5), dpi=150)

plt.plot(v2_animal['recall'], v2_animal['precision'], label='v2')
plt.plot(v3_animal['recall'], v3_animal['precision'], label='v3')

plt.xlabel('Recall')
plt.ylabel('Precision')     
plt.xlim([0, 1.05])
plt.ylim([0, 1.05])
plt.title('MegaDetector v2 and v3 precision-recall \non v3 validation set')
plt.axes().set_aspect('equal')
plt.legend(loc='lower left')
plt.tight_layout()
plt.savefig('/Users/siyuyang/Source/temp_data/CameraTrap/megadetectorv3/no_human_images/v2_v3_animal.png')<jupyter_output><empty_output><jupyter_text>TP at 90% confidence and 0.5 IoU<jupyter_code>len(v2_animal['tp_fp'])
len(v3_animal['tp_fp'])
conf_threshold = 0.9

tp_fp_confident = [[], []]

for i_version, version in enumerate((v2_animal, v3_animal)):
    for s, tf in zip(version['scores'], version['tp_fp']):
        if s > 0.9:
            tp_fp_confident[i_version].append(tf)
len(tp_fp_confident[0])
len(tp_fp_confident[1])
total_gt = 19562

v2_tp = sum(tp_fp_confident[0]) / total_gt
v3_tp = sum(tp_fp_confident[1]) / total_gt

v2_tp
v3_tp<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12312874251497007, 'fraction_numerical': 0.04303892215568862, 'mean_word_length': 6.8617647058823525, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '432488', 'license': 'MIT', 'n_tokens_mistral': 1105, 'n_tokens_neox': 1015, 'n_words': 202, 'path': 'archive/detection/detector_eval/detection_eval_mdv2_mdv3.ipynb', 'repo_name': 'FFI-Vietnam/CameraTraps-FFIVietnamAdaptation', 'size': '2672'}