<jupyter_start><jupyter_text>Detect and analyse imagesThis notebook is part of the _Automated plant stage labelling of herbarium samples in the family *Brassicaceae*_ made at [Propulsion Academy Zurich](https://propulsion.academy/?gclid=Cj0KCQiAwf39BRCCARIsALXWETyIhnHT7bA3VYXXOC415brejc6qYXnX7kEpqJmmJ5d5kAcYgoiLhI4aAmPxEALw_wcB) in collaboration with [ETH Library](https://library.ethz.ch/en/)This notebook shows the process to use the trained model to detect fruits and flowers in herabrium sample pictures. For simplicity you can run the `run_detection.py` script using a command line interface. See the README for more details.<jupyter_code># Import modules and packages
import os
import sys
import json
import numpy as np
import time
from PIL import Image, ImageDraw
import matplotlib.pyplot as plt
plt.style.context("fivethirtyeight")
from skimage.io import imsave, imread<jupyter_output><empty_output><jupyter_text>Directory pathsFill in & test the directory needed for inference and data analysis:- ROOT_DIR is the folder where the Mask_RCNN model is saved together with the settings- PROJECT_DIR (full path) is the folder where the images are located and where the output of the model will be saved- INPUT_DIR_NAME is a subdirectory of the PROJECT_DIR where the images that you want to process are- INPUT_ANNOTATIONS_FILE is the (OPTIONAL) name of the JSON files with the annotations in COCO style format- OUTPUT_DIR_NAME is a subdirectory of the PROJECT_DIR where the output information and (OPTIONALLY) the masks will be saved<jupyter_code># directory of input and output images 
PROJECT_DIR = os.path.join("..", "data")
assert os.path.exists(PROJECT_DIR), 'PROJECT_DIR does not exist. Did you forget to read the instructions above? ;)'
%cd {PROJECT_DIR}

# MASK R CNN dir
ROOT_DIR =  os.path.join("..", "src", "Mask_RCNN")
assert os.path.exists(ROOT_DIR), 'ROOT_DIR does not exist. Did you forget to read the instructions above? ;)'
 
INPUT_DIR_NAME= "test"
INPUT_ANNOTATIONS_FILE = "test/test.json"
OUTPUT_DIR_NAME = "OUTPUT_Masks"
# import mask-r-cnn modules and libraries. 
# These are the files that make the model work. If not available they can be downloaded 
# from : https://github.com/akTwelve/Mask_RCNN. The notebvbok will attempt to do it automatically
sys.path.append("../src")
sys.path.append(ROOT_DIR) 

import herbaria as hb
try:
  from mrcnn.config import Config
  import mrcnn.utils as utils
  from mrcnn import visualize
  import mrcnn.model as modellib

except ModuleNotFoundError:
  print("modules for MaskRCNN not found, downloading mask-RCNN from source")
  !git clone https://github.com/akTwelve/Mask_RCNN

  sys.path.append(os.path.join(ROOT_DIR, Mask_RCNN))
  from mrcnn.config import Config
  import mrcnn.utils as utils
  from mrcnn import visualize
  import mrcnn.model as modellib<jupyter_output><empty_output><jupyter_text>Below are the configs to create the model and the dataset. They are based on the main class of the Mask_RCNN `utils.Config()` and are present in the `herbaria.py` module. Here we import them from `herbaria.py` and then add some configs specifically for detection.<jupyter_code>config = hb.HerbariaConfig()
config.display()<jupyter_output>Configurations:
BACKBONE                       resnet50
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     2
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
COMPUTE_BACKBONE_SHAPE         None
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.7
DETECTION_NMS_THRESHOLD        0.3
FPN_CLASSIF_FC_LAYERS_SIZE     1024
GPU_COUNT                      1
GRADIENT_CLIP_NORM             5.0
IMAGES_PER_GPU                 2
IMAGE_CHANNEL_COUNT            3
IMAGE_MAX_DIM                  1024
IMAGE_META_SIZE                15
IMAGE_MIN_DIM                  1024
IMAGE_MIN_SCALE                0
IMAGE_RESIZE_MODE              square
IMAGE_SHAPE                    [1024 1024    3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.01
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE          [...]<jupyter_text>The following are additional settings for inference, based on the **HerbariaConfig** class<jupyter_code>class InferenceConfig(hb.HerbariaConfig):
    GPU_COUNT = 1
    IMAGES_PER_GPU = 1
    IMAGE_MIN_DIM = 1024
    IMAGE_MAX_DIM = 1024
    DETECTION_MIN_CONFIDENCE = 0.5
    
inference_config = InferenceConfig()<jupyter_output><empty_output><jupyter_text>DATASET creationThe HerbariaDataset class holds the procedure for creating a dataset to feed into the model (OPTIONAL: suggested only for model evaluation and training). It can be imported from `herbaria.py`.<jupyter_code># Load dataset
fpath_test_annotations = os.path.join(PROJECT_DIR,INPUT_ANNOTATIONS_FILE)
fldr_path_test_images = os.path.join(PROJECT_DIR, INPUT_DIR_NAME)

dataset = HerbariaDataset()

dataset.load_data(fpath_test_annotations, fldr_path_test_images)
dataset.prepare()<jupyter_output><empty_output><jupyter_text>Create modelHere we first create the model using the HerbariaConfig settings, than we load the model using the trained_weights argument.This is equivalent to the `herbaria.load_trained_model` function<jupyter_code># load_trained_model function
def load_trained_model(model_path, mode="inference", config=None):
  """function to load a model with pre_trained weights.
  Params:
  - model_path : STRING path saved weights
  - mode : STRING "inference" or "train" DEFAULT "inference"
  - config :  object of class Config, defaults to InferenceConfig for inference and HerbariaConfig for training
  Returns a pre-trained model ready for training or inference"""
  
  assert os.path.exists(model_path), "Provide path to trained weights"

  if config == None:
    if mode == 'train':
      config = HerbariaConfig()
    elif mode == 'inference':
      config = InferenceConfig()
    else :     
        assert " 'mode' should be 'train' or 'inference' "

  print("Loading weights from ", model_path)
  model = modellib.MaskRCNN(mode="inference", 
                          config=inference_config,
                          model_dir=MODEL_DIR)
  # load weights
  model.load_weights(model_path, by_name=True)
  return model
  
model_path = "../logs/Mask_RCNN/m_image_augm20201116T0937/mask_rcnn_m_image_augm_0009.h5"
model=load_trained_model(model_path)
model = hb.load_trained_model()<jupyter_output>loading weights from ../src/model_weights/m_image_augm20201116T0937/mask_rcnn_m_image_augm_0009.h5
Loading weights from  ../src/model_weights/m_image_augm20201116T0937/mask_rcnn_m_image_augm_0009.h5
WARNING:tensorflow:From /home/matt/anaconda3/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py:574: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
Re-starting from epoch 9<jupyter_text>Load images and run inference using the model  Settings for inferenceThe function `detect_flowers_fruits()` allows to elaborate images and save the result in a systematic manner. It can be accessed through the `run_detection.py` script using the command line interface. See the README for mmore details<jupyter_code>def detect_flowers_fruits(input_images_dir, output_folder, model=None, **kwargs):
    """ Runs predictions on images and save the results in output folder.
    Optionally can:
         Filter the outputs based on the score (confidence of the predictions),
         Save masks as png, Display images and outputs (not through Command Line interface)

    Returns the filename of the images processed correctly and optionally

    Params:
    -input_images_dir: path to folder of input images
    -output_folder: path to output destination folder
    -model : model to use to run predictions (predefined if using CLI)

    OPTIONAL params:
    -save_predictions: BOOL default True
    -save masks: BOOL default False
    -filter_scores : REAL between 0 and 1 default 0.5
    -save stats : BOOL default True saves a csv of detection statistics for each image"""

    save_predictions = kwargs.get("save_predictions", True)
    save_masks = kwargs.get("save_masks", False)
    filter_scores = kwargs.get("filter_scores", 0.5, )
    show_predictions = kwargs.get("show_predictions", False)
    save_stats = kwargs.get("save_stats", True)
    labels = kwargs.get("labels", ['flowers', 'fruits'])

    print(f" running inference with the following options:")
    print(f" image source : ", input_images_dir)
    print(f" destination folder: {output_folder}")
    for k, v in kwargs.items():
        print(k, v)

    if not model:
        model = hb.load_trained_model()

    # checks
    assert os.listdir(input_images_dir), "input folder is empty. Please check path or extension of files"
    if not os.path.exists(output_folder):
        'dest folder does not exist. It will be created ;)'
        os.mkdir(output_folder)

    processed_list = []  # list of processed images
    stats_list = []
    for filename in os.listdir(input_images_dir)[:2]:
        if os.path.splitext(filename)[1].lower() in ['.png', '.jpg', '.jpeg']:
            image_path = os.path.join(input_images_dir, filename)
        else:
            continue

        img = skimage.io.imread(image_path)
        img_arr = np.array(img)
        results = model.detect([img_arr], verbose=0)  # this is equivalent to .predict() and actually does the inference
        # filter by scores

        r = results[0]  # this contains all the predictions including the masks for each image
        # filter results by confidence score
        if filter_scores > 0:
            print(f"removing predictions with confidence below {filter_scores}")

            sc_ = r['scores']
            lim_ = len(sc_[r['scores'] > filter_scores]) - 1
            r['masks'] = r['masks'][:, :, :lim_]
            r['rois'] = r['rois'][:lim_]
            r['scores'] = r['scores'][:lim_]
            r['class_ids'] = r['class_ids'][:lim_]

        # create stats table for output
        if save_stats:
            # create dictionary with labels
            stats_dic = {
                'img_id': filename.split(".")[0],
                'min_confidence': filter_scores,
                'max_confidence': r['scores'][0],

            }
            for i in labels:
                stats_dic[str(i)+"_count"] = 0
                stats_dic[str(i)+"_area"] = 0
            # add count of flowers and fruits
            labs, counts = np.unique(r['class_ids'], return_counts=True)
            for n in range(len(labs)):
                column_head = labels[labs[n]-1]+"_count"
                # print("column line", column_head)
                stats_dic[column_head] = counts[n]

            # add area of flowers and fruits(from boxes)
            area_sum=[0]*len(labels)
            for n, box in enumerate(r['rois']):
                lab = r['class_ids'][n]
                area = (box[2]-box[0])*(box[3]-box[1])
                area_sum[lab-1] = area_sum[lab-1]+area
            column_head = labels[lab-1] + "_area"
            for n,i in enumerate(labels):
                stats_dic[ i + "_area"] = area_sum[n]
            print(stats_dic)
            stats_list.append(stats_dic)

        # display images
        if show_predictions:
            visualize.display_instances(img, r['rois'], r['masks'], r['class_ids'],
                                        labels, figsize=(100, 100))

        # save images
        if save_predictions:
            print(f"\n saving predictions for image {filename} to {output_folder}")
            # create dictionary ( MASKS are EXCLUDED)
            json_pred = {'categories': labels,
                         'image': filename.split(".")[0],
                         'rois': r['rois'].tolist(),
                         'labels': r['class_ids'].tolist(),
                         'scores': r['scores'].tolist()
                         }
            processed_list.append(filename)

            json_path = os.path.join(output_folder, filename.split(".")[0] + ".json")
            with open(os.path.join(json_path), "w+") as file:
                json.dump(json_pred, file)
        # saving masks
        if save_masks:
            # create a single mask with category values
            complete_mask = np.zeros(img_arr.shape[:-1], dtype=np.uint8)
            for n in range(r['masks'].shape[2]):
                single_mask = r['masks'][:, :, n].astype(np.uint8)
                single_mask = single_mask * r['class_ids'][n]
                complete_mask = complete_mask + single_mask
            # save image
            mask_path = os.path.join(output_folder, "MASK-" + filename.split(".")[0] + ".png")
            imsave(mask_path, complete_mask)
            print(f"\n saved predicted masks for image {filename} as {mask_path}")
            processed_list.append(filename)
    # save stats as csv
    if save_stats:
        stat_path = os.path.join(output_folder, "detection_stats.csv")
        keys = stats_list[0].keys()
        with open(stat_path, 'w+', newline="")as output_file :
            dict_writer = csv.DictWriter(output_file, keys)
            dict_writer.writeheader()
            dict_writer.writerows(stats_list)
    proc_images_n = len(set(processed_list))
    print("processing finished !!")
    print(f"{proc_images_n} images elaborated")
    return set(processed_list)<jupyter_output><empty_output><jupyter_text>Detection parameters<jupyter_code># Set to true to display all images !! Can be slow with many images
show_predictions = True

# Set to true if you want to save detction outputs
save_predictions = False
#Set to true if you want to save the mask as image. This can be memory intensive
save_masks = False
# Set to true if you want to filter the prediction ba scores. Suggested threshold 0.5
filter_scores= True 
filter_score_thresh=0.5
assert filter_score_thresh >0, "Set filter score above 0"
 
import skimage
input_images = os.path.join(PROJECT_DIR,INPUT_DIR_NAME)
output_folder= os.path.join(PROJECT_DIR,OUTPUT_DIR_NAME)

fpath_test_annotations = os.path.join(PROJECT_DIR,INPUT_ANNOTATIONS_FILE)
fldr_path_test_images = os.path.join(PROJECT_DIR, INPUT_DIR_NAME)

detect_flowers_fruits(input_images, output_folder, model, show_predictions=True, save_predictions=False )
img_dir = os.path.join(PROJECT_DIR,INPUT_DIR_NAME)
gt_json_path= os.path.join(PROJECT_DIR,INPUT_ANNOTATIONS_FILE)
 
 # create dataset
dataset = HerbariaDataset()
dataset.load_data(gt_json_path, img_dir)
dataset.prepare()
for image_id in dataset.image_ids[:2]:
  # load image and annotations from GT and detections
  image, image_meta, gt_class_id, gt_bbox, gt_mask =\
          modellib.load_image_gt(dataset, config,
                                  image_id)
  results = model.detect([image], verbose=0)
  r = results[0]
  # display gt
  visualize.display_instances(image, gt_bbox, gt_mask, gt_class_id, 
                                  dataset.class_names,figsize=(100,100), show_mask=False)
  # display pred
  visualize.display_instances(image, r['rois'], r['masks'], r['class_ids'], 
                                  dataset.class_names,figsize=(100,100), show_mask=False)<jupyter_output><empty_output><jupyter_text>Model performancebelow a selection of functions to evaluate model performance in a more thorough way. More functions are available i9n the original Mask_RCNN repo. See thi readme for more details<jupyter_code># Compute VOC-style Average Precision
def compute_batch_ap(image_ids):
    sp_eval = {}
    sp_eval['AP']=[]
    sp_eval['precisions']=[]
    sp_eval['recalls']=[]
    sp_eval['overlaps']=[]

    for image_id in image_ids:
        # Load image
        image, image_meta, gt_class_id, gt_bbox, gt_mask =\
            modellib.load_image_gt(dataset, config,
                                   image_id)
        # Run object detection
        results = model.detect([image], verbose=0)
        # Compute AP
        r = results[0]
        AP, precisions, recalls, overlaps =\
            utils.compute_ap(gt_bbox, gt_class_id, gt_mask,
                              r['rois'], r['class_ids'], r['scores'], r['masks'])
        sp_eval['AP'].append(AP)
        sp_eval['precisions'].append(precisions)
        sp_eval['recalls'].append(recalls)
        sp_eval['overlaps'].append(overlaps)
    return sp_eval

# # Pick a set of random images
# image_ids = np.random.choice(dataset.image_ids, 10)
# APs = compute_batch_ap(image_ids)
# print("mAP @ IoU=50: ", np.mean(APs))
from utils import compute_matches
from sklearn.metrics import f1_score
import seaborn as sns
sns.color_palette("Spectral", as_cmap=True)
def evaluate_model_performance(img_dir, gt_json_path, model, filter_score=0.5, viz=False):
  """ evaluates the model peroformance against a test set of images ad annotations
  Parameters:
  - model ??
  - detections folder with json files as retruned by <detect_flower_fruits> function
  - ground truth data (annotations) in COCO format

  OPTIONAL Parameters:
  -plot outputs
  -masks
  """


  # create dataset
  dataset = HerbariaDataset()
  dataset.load_data(gt_json_path, img_dir)
  dataset.prepare()
  
  # for det in os.listdir(detections_dir):
  #   if det.split(".") == "json":
  n_record=[]
  f1_boxes=[]
  recall_boxes=[]
  for image_id in dataset.image_ids:
  # load image and annotations from GT and detections
    image, image_meta, gt_class_id, gt_bbox, gt_mask =\
            modellib.load_image_gt(dataset, config,
                                   image_id)
    results = model.detect([image], verbose=0)
    r = results[0]

    if filter_score:
        print(f"removing predictions below {filter_score}")
        sc_=r['scores']
        lim_=len(sc_[ sc_ > filter_score ])-1
        r['masks']=r['masks'][:,:,:lim_]
        r['rois']=r['rois'][:lim_]
        r['scores']= r['scores'][:lim_]
        r['class_ids'] = r['class_ids'][:lim_]
  # boxes classification
    max_len =min(len(gt_class_id), len(r['class_ids']))
    print(max_len)
    f1_=f1_score(gt_class_id[:max_len],r['class_ids'][:max_len])
    f1_boxes.append(f1_)
    print("\n f1 score : ", f1_)
  
  # Overlaps boxes and masks
  # sp_eval = compute_batch_ap(dataset.image_ids)
  # IOU masks
  
    iou_recall,_=utils.compute_recall(r['rois'], gt_bbox, 0)
    print("recall boxes:", iou_recall)
    recall_boxes.append(iou_recall)
  # break
  # compare areas
    # break


  print(f" mean f1 score of batch : {np.mean(f1_boxes)}")
  print(f"mean recall score of batch : {np.mean(recall_boxes)}")
  print(f1_boxes)

  if viz:
    pass
    
    
  eval_results={
      "f1_score": f1_boxes, 
      "recall_boxes": recall_boxes
                  
                  }
  return eval_results

img_dir = os.path.join(PROJECT_DIR,INPUT_DIR_NAME)
gt_json_path= os.path.join(PROJECT_DIR,INPUT_ANNOTATIONS_FILE)
perf=evaluate_model_performance(img_dir, gt_json_path, model, 0.5, False)

import matplotlib
matplotlib.rc('axes',edgecolor="#FFFFFF")
matplotlib.rc('lines', color= "#FFFFFF")
# matplotlib.rc('xticks', color= "#FFFFFF")


sns.color_palette("Spectral", as_cmap=True)
fig, ax = plt.subplots(ncols=1, nrows=2, figsize=(20,20))
fig.set_facecolor("#000000")
ax[0].hist(perf['f1_score'], bins=10, edgecolor='white', alpha=0.6)
# ax[0].set_facecolor("#000000")
ax[0].set_title("F1 score object classification")
ax[0].axvline(x=np.mean(perf['f1_score']), color="blue")
# ax[0].plot(perf['f1_score'], color='blue', alpha=0.6)

ax[1].hist(perf['recall_boxes'], bins=10, edgecolor='white', alpha=0.6)
ax[1].axvline(x=np.mean(perf['recall_boxes']), color="blue")

# ax[1].set_facecolor("#000000")
ax[0].xaxis.label.set_color("#FFFFFF")
ax[0].tick_params(axis='x', colors="#FFFFFF")
ax[1].tick_params(axis='x', colors="#FFFFFF")
ax[0].tick_params(axis='y', colors="#FFFFFF")
ax[1].tick_params(axis='y', colors="#FFFFFF")
ax[0].grid(axis='y',color="#000000", lw=2)
ax[1].grid(axis='y',color="#000000", lw=2)

plt.savefig('eval1.pdf')<jupyter_output><empty_output><jupyter_text>Image analysis<jupyter_code>perf
def analyse_batch_detections(image_folder, json_folder, mask_folder,):
  """" function to analys images AFTER detection. 
  Returns dictionary with data about prediction batch  
  Parameters:
  json_folder:  folder with the output of the model as produced by <detect_flower_fruits> function

  OPTIONAL parameters:
  mask_folder
  plot
  """
  # load image and data
  
  # flowers

  # image classification

  # flowers per image mean and stdev

  # fruits per image mean and standard dev

  pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10317000096646371, 'fraction_numerical': 0.015076833864888374, 'mean_word_length': 3.3422156945027277, 'pattern_counts': {'":': 3, '<': 41, '<?xml version=': 0, '>': 45, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '527770', 'license': 'MIT', 'n_tokens_mistral': 6858, 'n_tokens_neox': 6534, 'n_words': 1703, 'path': 'notebooks/3__Detect_flowers_and_fruits.ipynb', 'repo_name': 'LVParkinson/herbaria--plant-labeling', 'size': '20694'}