@inproceedings{
harrington2023evaluating,
title={Evaluating Peripheral Vision as an Input Transformation to Understand Object Detection Model Behavior},
author={Anne Harrington and Vasha DuTell and Mark Hamilton and Ayush Tewari and Simon Stent and William T. Freeman and Ruth Rosenholtz},
booktitle={NeuRIPS 2023 Workshop on Gaze Meets ML},
year={2023},
url={https://openreview.net/forum?id=y5ihmWxYWx}
}
Incorporating aspects of human gaze into deep neural networks (DNNs) has been used to both improve and understand the representational properties of models. We extend this work by simulating peripheral vision -- a key component of human gaze -- in object detection DNNs. To do so, we modify a well-tested model of human peripheral vision (the Texture Tiling Model, TTM) to transform a subset of the MS-COCO dataset to mimic the information loss from peripheral vision. This transformed dataset enables us to (1) evaluate the performance of a variety of pre-trained DNNs on object detection in the periphery, (2) train a Faster-RCNN with peripheral vision input, and (3) test trained DNNs for corruption robustness. Our results show that stimulating peripheral vision helps us understand how different DNNs perform under constrained viewing conditions. In addition, we show that one benefit of training with peripheral vision is increased robustness to geometric and high severity image corruptions, but decreased robustness to noise-like corruptions. Altogether, our work makes it easier to model human peripheral vision in DNNs to understand both the role of peripheral vision in guiding gaze behavior and the benefits of human gaze in machine learning. Data and code will be released at https://github.com/RosenholtzLab/coco-periph-gaze
Evaluating Peripheral Vision as an Input Transformation to Understand Object Detection Model Behavior
Anne Harrington Vasha DuTell Mark Hamilton Ayush Tewari Simon Stent William T. Freeman Ruth Rosenholtz