Performance Benchmark of Quantized Detection Model

This directory is built for comparison of our "quantized / quantization aware trained" detection algorithm to one of the SOTA compact detection algorithms, EfficientDet-d0, which has comparable complexity and structure with our quantized model.

Our person tracking algorithm uses MobileNet-v2 as backbone mechanism and combines it with 2 SSD heads using total of 9 anchor boxes. Overall model consists of 60 convolution layers.

We quantized the layers of this model and applied "quantization aware training" methods to recover its accuracy drop due to quantization of layers and output clamping. We have re-scaled and center cropped the images in Wider Person Dataset, also we resized its annotations and converted in to COCO annotation format to use them in our training/evaluation tasks. Then we applied smart training approaches which consider the effects of quantization and output clamping of the layers during optimization, which we call "quantization aware training".

Our main motivation of quantizing networks and applying quantization aware training methods is to reduce the overall network size, inference time and training effort while keeping accuracy drop in an acceptable level. We aim to develop quantized compact detection algorithms executable on low power and low cost accelerator chips.

Dependencies

Evaluating EfficientDet with Wider Person Validation Dataset

In this section, steps to reproduce the evaluation of EfficientDet model from Yet-Another-EfficientDet-Pytorch Repository with d0 coefficients is explained. For evaluation, aforementioned Wider Person Validation Dataset in COCO format is used.

1. Clone EfficientDet to Your Local

Open a terminal and go to directory in your local where you want to clone , then type:

git clone --depth 1 https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch

2. Prepare EfficientDet-d0 Coefficients

Go to main directory
```
cd Yet-Another-EfficientDet-Pytorch/
```
Create weights folder
```
mkdir weights
```

Download EfficientDet d0 coefficients

wget https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/releases/download/1.0/efficientdet-d0.pth -O weights/efficientdet-d0.pth

3. Prepare Wider Person Dataset

Download original Wider Person Dataset
- Sign up Codalab and participate to WIDER Face & Person Challenge 2019
- Under "Participate" tab click "Train & Validation Data in Google Drive" and download
  - val_data.tar.gz
  - Annotations/val_bbox.txt
- Extract val_data.tar.gz as val_data and move val_data folder under ./data/original_wider/val_data
- Move "val_bbox.txt" under ./data/original_wider/
Move our "wider2coco.py" script in "efficientdet_comparison" folder to main folder of your local "Yet-Another-EfficientDet-Pytorch" repository. Following code will produce resized images and annotations.
```
python wider2coco.py -ip ./data/original_wider/val_data -af ./data/original_wider/val_bbox.txt 
```

Script will automatically convert Wider Dataset in to COCO format and create following repository structure:

  ./Yet-Another-EfficientDet-Pytorch/datasets/wider/val
       image001.jpg
       image002.jpg
       ...
  ./Yet-Another-EfficientDet-Pytorch/datasets/wider/annotations
      instances_val.json

4. Manually Set Project's Specific Parameters

Create a yml file "wider.yml" under "projects"
```
touch projects/wider.yml
```

Copy following content in to "wider.yml" file

project_name: wider
train_set: train
val_set: val
num_gpus: 1  # 0 means using cpu, 1-N means using gpus 
 
# Wider validation dataset mean and std in RGB order
mean: [0.416, 0.413, 0.406]
std: [0.308, 0.306, 0.310]
 
# this is coco anchors, change it if necessary
anchors_scales: '[2 ** 0, 2 ** (1.0 / 3.0), 2 ** (2.0 / 3.0)]'
anchors_ratios: '[(1.0, 1.0), (1.4, 0.7), (0.7, 1.4)]'
 
# objects from all labels from your dataset with the order from your annotations.
# its index must match your dataset's category_id.
# category_id is one_indexed,
# for example, index of 'car' here is 2, while category_id of is 3
obj_list: ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', '', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', '', 'backpack', 'umbrella', '', '', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', '', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', '', 'dining table', '', '', 'toilet', '', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', '', 'book', 'clock', 'vase', 'scissors','teddy bear', 'hair drier', 'toothbrush']

5. Evaluate EfficientDet model performance

Setup "person only evaluation"
- Open "coco_eval.py" under /Yet-Another-EfficientDet-Pytorch
- Paste following code after line 132, "coco_eval.params.imgIds = image_ids", to evaluate mAP results only for person category
```
coco_eval.params.catIds = 1
```

For evaluation on cuda enabled platform

python coco_eval.py -p wider -c 0 -w ./weights/efficientdet-d0.pth

For evaluation on cuda disabled platform

python coco_eval.py -p wider -c 0 -w ./weights/efficientdet-d0.pth --cuda False

Evaluating Our Quantized MobilenetSSDLite model with Wider Person Validation Dataset

1. Clone Quantized Mobilenet Model to Your Local

Open a terminal and go to directory in your local where you want to clone our Quantization Aware Training - Person Tracking repository, then type:

git clone --depth 1 https://github.com/sai-tr/persontracking_qat.git

2. Prepare Wider Person Dataset

Download original Wider Person Dataset
- Sign up Codalab and participate to WIDER Face & Person Challenge 2019
- Under "Participate" tab click "Train & Validation Data in Google Drive" and download
  - val_data.tar.gz
  - Annotations/val_bbox.txt
- Extract val_data.tar.gz as val_data and move val_data folder under ./data/original_wider/val_data
- Move "val_bbox.txt" under ./data/original_wider/
Move our "wider2coco.py" script in "efficientdet_comparison" folder to main folder of your local "persontracking_qat" repository. Following code will produce resized images and annotations.
```
python wider2coco.py -ip ./data/original_wider/val_data -af ./data/original_wider/val_bbox.txt 
```

Script will automatically convert Wider Dataset in to COCO format and create following repository structure:

  ./persontracking_qat/datasets/wider/val
       image001.jpg
       image002.jpg
       ...
  ./persontracking_qat/datasets/wider/annotations
      instances_val.json

3. Evaluate Quantized Mobilenet Model Performance

Note that model mode should match with the loaded model parameter dictionary. Selectable model modes are:

Full Precision Unconstrained(fpt_unc): All layers are in full precision and no output clamping
Full Precision Constrained(fpt): All layers are in full precision and layer output are clamped to +-1
Quantized(qat): All layers are quantized layer outputs are clamped to +-1
Move our "coco_eval.py" script in "efficientdet_comparison" folder to "persontracking_qat" folder and use following command for evaluation:
```
python coco_eval.py -m qat -dp ./datasets/wider/val -ap ./datasets/wider/annotations/all_val_prep.json -wp ./efficientdet_comparison/training_experiment_best.pth.tar
```
Note that: Code evaluates quantized model with weights "training_experiment_best.pth.tar", using images and annotations in paths "./datasets/wider/val" "./datasets/wider/annotations/instances_val.json" respectively.

mAP Comparisons

EfficientDet-d0

### Wider Validation Dataset mAP scores ###
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.292
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.543
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.275
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.109
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.409
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.532
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.106
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.369
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.435
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.270
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.546
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.678

Quantized Mobilenet

### Wider Validation Dataset mAP scores ###
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.281
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.457
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.310
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.075
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.406
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.582
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.107
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.324
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.331
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.110
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.481
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.637