Neuroscience theory posits that the brain's visual system coarsely identifies broad object categories via neural activation patterns, with similar objects producing similar neural responses. Artificial neural networks also have internal activation behavior in response to stimuli. We hypothesize that networks exhibiting brain-like activation behavior will demonstrate brain-like characteristics, e.g., stronger generalization capabilities. In this paper we introduce a human-model similarity (HMS) metric, which quantifies the similarity of human fMRI and network activation behavior. To calculate HMS, representational dissimilarity matrices (RDMs) are created as abstractions of activation behavior, measured by the correlations of activations to stimulus pairs. HMS is then the correlation between the fMRI RDM and the neural network RDM across all stimulus pairs. We test the metric on unsupervised predictive coding networks, which specifically model visual perception, and assess the metric for statistical significance over a large range of hyperparameters. Our experiments show that networks with increased human-model similarity are correlated with better performance on two computer vision tasks: next frame prediction and object matching accuracy. Further, HMS identifies networks with high performance on both tasks. An unexpected secondary finding is that the metric can be employed during training as an early-stopping mechanism.
How to best evaluate machine learning algorithms is an ongoing discussion. Traditional evaluations focus on external performance on a dataset, but there are no guarantees against overfitting or unpredictable network performance on real-world data @cite_27 . One alternative evaluation regime is visual psychophysics, which monitors neural network performance while increasingly perturbing stimuli @cite_55 @cite_1 @cite_50 . This evaluation centers on the observation that a network which inconsistently recognizes perturbed stimuli cannot be trustworthy. However, these evaluations are still focused on creating variability within a dataset, offering no guarantee that the network is not simply overfit to it. Moving beyond datasets, our proposed evaluation metric HMS quantifies consistency in a network's internal behavior by directly comparing against the internal behavior of one of the most generalizable vision systems in the world: the biological brain @cite_14 @cite_20 @cite_47 .
@cite_14 @cite_55 @cite_1 @cite_27 @cite_50 @cite_47 @cite_20
2889084315 2784570041 2552164092 2031342017 2130000240 2888892231
Abstract Human occipitotemporal cortex contains neural representations for a variety of perceptual and conceptual features. We report a study examining neural representations of real-world size along the visual ventral stream, while carefully accounting for taxonomic categories that typically co-vary with size. We recorded brain activity during a functional Magnetic Resonance Imaging (fMRI) scan from eighteen participants as they were presented with images of twelve animal species. The animals were selected to vary on a number of dimensions, including taxonomic group, real-world size and prior familiarity. We apply multivariate analysis methods, including representational similarity analysis (RSA) and machine learning classifiers, to probe the distributed patterns of neural activity evoked by these presentations. We find that the real-world size of visually presented animate items is represented in posterior, but not anterior, regions of the ventral stream. A significant linear relationship is present for real-world size representation along the ventral stream. These representations remain after controlling for factors such as taxonomic category, familiarity and models of visual similarity, and even after restricting examinations to within-taxonomic category comparisons, suggesting that size information is found for within, as well as between, taxonomic categories. These findings are consistent with real-world size having an influence on activity patterns in early regions of the visual system. Psychlab is a simulated psychology laboratory inside the first-person 3D game world of DeepMind Lab ( 2016). Psychlab enables implementations of classical laboratory psychological experiments so that they work with both human and artificial agents. Psychlab has a simple and flexible API that enables users to easily create their own tasks. As examples, we are releasing Psychlab implementations of several classical experimental paradigms including visual search, change detection, random dot motion discrimination, and multiple object tracking. We also contribute a study of the visual psychophysics of a specific state-of-the-art deep reinforcement learning agent: UNREAL ( 2016). This study leads to the surprising conclusion that UNREAL learns more quickly about larger target stimuli than it does about smaller stimuli. In turn, this insight motivates a specific improvement in the form of a simple model of foveal vision that turns out to significantly boost UNREAL's performance, both on Psychlab tasks, and on standard DeepMind Lab tasks. By open-sourcing Psychlab we hope to facilitate a range of future such studies that simultaneously advance deep reinforcement learning and improve its links with cognitive science. By providing substantial amounts of data and standardized evaluation protocols, datasets in computer vision have helped fuel advances across all areas of visual recognition. But even in light of breakthrough results on recent benchmarks, it is still fair to ask if our recognition algorithms are doing as well as we think they are. The vision sciences at large make use of a very different evaluation regime known as Visual Psychophysics to study visual perception. Psychophysics is the quantitative examination of the relationships between controlled stimuli and the behavioral responses they elicit in experimental test subjects. Instead of using summary statistics to gauge performance, psychophysics directs us to construct item-response curves made up of individual stimulus responses to find perceptual thresholds, thus allowing one to identify the exact point at which a subject can no longer reliably recognize the stimulus class. In this article, we introduce a comprehensive evaluation framework for visual recognition models that is underpinned by this methodology. Over millions of procedurally rendered 3D scenes and 2D images, we compare the performance of well-known convolutional neural networks. Our results bring into question recent claims of human-like performance, and provide a path forward for correcting newly surfaced algorithmic deficiencies. Datasets are an integral part of contemporary object recognition research. They have been the chief reason for the considerable progress in the field, not just as source of large amounts of training data, but also as means of measuring and comparing performance of competing algorithms. At the same time, datasets have often been blamed for narrowing the focus of object recognition research, reducing it to a single benchmark performance number. Indeed, some datasets, that started out as data capture efforts aimed at representing the visual world, have become closed worlds unto themselves (e.g. the Corel world, the Caltech-101 world, the PASCAL VOC world). With the focus on beating the latest benchmark numbers on the latest dataset, have we perhaps lost sight of the original purpose? The goal of this paper is to take stock of the current state of recognition datasets. We present a comparison study using a set of popular datasets, evaluated based on a number of criteria including: relative data bias, cross-dataset generalization, effects of closed-world assumption, and sample value. The experimental results, some rather surprising, suggest directions that can improve dataset collection as well as algorithm evaluation protocols. But more broadly, the hope is to stimulate discussion in the community regarding this very important, but largely neglected issue. Patients with damage to left anteromedial temporal cortex often show a striking deficit: they fail to recognize animals and other living things. This failure of recognition presents an important challenge to theories of the neural representation of conceptual knowledge. Here we propose that this lesion--behaviour association arises because polymodal neurons in anteromedial temporal cortex integrate simple features into complex feature conjunctions, providing the neural infrastructure for differentiating among objects.