Saccade Mechanisms for Image Classification, Object Detection and Tracking
We examine how the saccade mechanism from biological vision can be used to
make deep neural networks more efficient for classification and object
detection problems. Our proposed approach is based on the ideas of
attention-driven visual processing and saccades, miniature eye movements
influenced by attention. We conduct experiments by analyzing: i) the robustness
of different deep neural network (DNN) feature extractors to partially-sensed
images for image classification and object detection, and ii) the utility of
saccades in masking image patches for image classification and object tracking.
Experiments with convolutional nets (ResNet-18) and transformer-based models
(ViT, DETR, TransTrack) are conducted on several datasets (CIFAR-10, DAVSOD,
MSCOCO, and MOT17). Our experiments show intelligent data reduction via
learning to mimic human saccades when used in conjunction with state-of-the-art
DNNs for classification, detection, and tracking tasks. We observed minimal
drop in performance for the classification and detection tasks while only using
about 30\% of the original sensor data. We discuss how the saccade mechanism
can inform hardware design via ``in-pixel'' processing.