SATBench: Benchmarking the speed-accuracy tradeoff in object recognition by humans and dynamic neural networks
The core of everyday tasks like reading and driving is active object
recognition. Attempts to model such tasks are currently stymied by the
inability to incorporate time. People show a flexible tradeoff between speed
and accuracy and this tradeoff is a crucial human skill. Deep neural networks
have emerged as promising candidates for predicting peak human object
recognition performance and neural activity. However, modeling the temporal
dimension i.e., the speed-accuracy tradeoff (SAT), is essential for them to
serve as useful computational models for how humans recognize objects. To this
end, we here present the first large-scale (148 observers, 4 neural networks, 8
tasks) dataset of the speed-accuracy tradeoff (SAT) in recognizing ImageNet
images. In each human trial, a beep, indicating the desired reaction time,
sounds at a fixed delay after the image is presented, and observer's response
counts only if it occurs near the time of the beep. In a series of blocks, we
test many beep latencies, i.e., reaction times. We observe that human accuracy
increases with reaction time and proceed to compare its characteristics with
the behavior of several dynamic neural networks that are capable of
inference-time adaptive computation. Using FLOPs as an analog for reaction
time, we compare networks with humans on curve-fit error, category-wise
correlation, and curve steepness, and conclude that cascaded dynamic neural
networks are a promising model of human reaction time in object recognition
tasks.