article: Observers can rapidly perform a variety of visual tasks such as categorizing a scene as open, as outdoor, or as a beach. Although we know that different tasks are typically associated with systematic differences in behavioral responses, to date, little is known about the underlying mechanisms. Here, we implemented a single integrated paradigm that links perceptual processes with categorization processes. Using a large image database of natural scenes, we trained machine-learning classifiers to derive quantitative measures of task-specific perceptual discriminability based on the distance between individual images and different categorization boundaries. We showed that the resulting discriminability measure accurately predicts variations in behavioral responses across categorization tasks and stimulus sets. We further used the model to design an experiment, which challenged previous interpretations of the so-called “superordinate advantage. ” Overall, our study suggests that observed differences in behavioral responses across rapid categorization tasks reflect natural variations in perceptual discriminability.
  Categorization is perhaps one of our most critical visual functions as it allowed our ancestors to distinguish friend from foe and the edible from the inedible. Observers can rapidly extract meaning from brief presentations of complex visual scenes [1]—far exceeding the best existing engineered artificial systems [2]. Observers can reliably perform a variety of categorization tasks [3] such as categorizing a scene as open, as outdoor, or as a beach. However, it has also been shown that there exist systematic differences in participants’ behavioral responses across categorization tasks. In particular, categorizing a scene as open or navigable (i. e. , attribute level) necessitates shorter presentation times than categorizing a scene as a lake or a beach (i. e. , basic level, see [4]). (Note that our definition of basic-levelness follows the common usage in vision science (see [5–13]) and reflects a logical [14] rather than functional definition of the basic level.) Similarly, participants appear to be faster and more accurate when categorizing a scene as outdoor (i. e. , superordinate level) compared to categorizing a scene at a basic level [7,9, 11,13]. A very recent study further suggests that subordinate scene categorization is less sensitive and slower than basic level categorization [15]. Beyond the categorization of natural scenes, there exist systematic differences in behavioral responses for object categories across taxonomic levels with observers’ subordinate-level categorization (e. g. , pigeons vs. other birds) being slower and less accurate than basic-level categorization (e. g. , birds vs. non-birds, see [5]. Similarly, basic-level categorization (e. g. , birds vs. dogs) has been shown to be slower than superordinate categorization (e. g. , animals vs. non-animals, see [16]). Participants tend to be faster and more accurate at categorizing faces at the superordinate level (i. e. categorizing faces vs. non-faces) compared with categorizing faces at the familiarity level (famous vs. non-famous, see [6]). However, for both familiar faces and other individually-known familiar objects, categorization at the subordinate level is faster than at the basic level [17]. Similarly, there exist systematic differences in behavioral responses for different social inference tasks [12]: For instance, categorization at the level of intentionality is faster than categorization at the level of belief and personality. Such systematic behavioral differences across categorization tasks are often taken as suggestive evidence for an underlying hierarchical organization of categorization processes with some categorization tasks taking precedence over others [5–13], but see also [18–20]. Overall, the past decade of research on visual categorization has produced a significant and rapidly increasing amount of data and, while systematic differences across categorization tasks have been well-characterized to date, little is known about the underlying mechanisms. In this study, we describe a computational model to account for variations in participants’ behavioral responses (both accuracy and reaction time) across tasks and stimuli for the rapid categorization of natural scenes. Previous work has proceeded along two seemingly parallel paths (see [21,22] for discussions) with a nearly exclusive focus on modeling either visual representations (see [23] for review) or categorization and decision-making (see [24] for review). Here, we implemented a single integrated paradigm that links perception with categorization processes. Formally, visual categorization corresponds to the process of associating visual stimuli xi = 1…m to category labels yi = 1…m to form (xi, yi) exemplar-label pairs. xi may be parametrized by a feature vector in a N-dimensional perceptual space x i = (x i 1, … x i k, … x i N). Fig 1A illustrates such a feature space for an hypothetical population of N = 2 feature detectors (in practice, we expect N to be much larger). Learning to categorize visual stimuli requires learning a categorization boundary that best represents the relation between input images xi and their corresponding category labels yi. Once a categorization boundary had been learned, the classification of a stimulus depends on its position relative to the categorization boundary: One side of the categorization boundary will be associated with a target set of stimuli while the other side will be associated with the distractor set. An illustration for hypothetical decision boundaries corresponding to different taxonomic levels is shown in Fig 1B–1D. According to this computational framework, different categorization tasks correspond to different decision boundaries, which carve the same perceptual space, an idea that has motivated the development of most existing computational models of perceptual categorization (see [22] for review). We used a rudimentary visual representation based on the “gist” algorithm [25] but other visual representations are possible (see [23] for review; see also Discussion). We further used a large image database [26] to train and test machine learning classifiers (regularized logistic regression) and estimate the decision boundaries associated with many different scene categorization tasks. A task-dependent measure of perceptual discriminability can then be derived for a particular categorization task by considering the distance between individual stimuli and the categorization boundary (Fig 2A). The basic intuition for this measure is that, for a particular categorization task, images that are closer to the categorization boundary will be harder to categorize than those that are further away leading to behavioral responses that are slower and less accurate. Furthermore, these values can be aggregated to yield estimates of accuracy for arbitrary sets of target and distractor stimuli (Fig 2B). The goal of the present study was to test the hypothesis that the perceptual discriminability of individual stimuli for a particular task is one of the main factors driving behavioral responses. While this hypothesis is built-in for many categorization models (see [24] for review), it had so far only been tested with simple artificial stimuli where participants were trained to learn a new object category parametrized by two dimensions (e. g. , [27], but see also [28] for alternative models.) However, this hypothesis has not yet been tested for well-learned, natural categories. We first found that model-derived discriminability values predicted well behavioral responses for different categorization tasks as reported in two published studies [4,11]. In addition, in experiment 1, we were further able to show that the model accurately predicted variations in accuracy and reaction time at the level of individual stimuli within the context of a scene categorization task. We then used the model to test the hypothesis that the so-called “superordinate advantage” [7,9, 11,13], whereby superordinate categorization is faster and more accurate than basic categorization, may reflect the greater perceptual discriminability of scenes at the superordinate vs. basic level. Consistent with this hypothesis, we first found that the model was consistent with the reported results of a published study on the superordinate advantage [9]. In experiment 2, we further showed that it is possible to use model-derived discriminability values to sample stimuli and to effectively reverse the superordinate advantage, making participants superordinate categorization slower and less accurate than basic categorization, thus offering a possible perceptual explanation of the phenomenon. Overall, our results provide a computational-level explanation for the systematic variations in rapid categorization behavioral responses across taxonomic levels, suggesting that these differences may simply reflect natural variations in perceptual discriminability. Our study thus challenges several existing theories of visual processing and offers a vivid example of how computational models can help summarize existing data as well as plan and interpret novel experiments.
  We have described an integrated paradigm that links perceptual processes with categorization processes. We used a large natural scene database to train and test machine learning classifiers in order to derive task-dependent perceptual discriminability values for individual images based on their distance to different categorization boundaries. We showed that the resulting model is consistent with a host of published results [4,9, 11]. In addition, we also designed two experiments to demonstrate that it is possible to use the model to sample stimuli in order to manipulate participants’ behavioral responses (both accuracy and reaction times). In experiment 1, we showed that sampling stimuli with increasing discriminability values (i. e. with increasing distance to the category boundary) yields behavioral responses that are increasingly fast and accurate. This suggests that the perceptual discriminability of individual stimuli for a particular task is one of the main factors driving behavioral responses. A few recent studies have hinted at the contribution of perceptual discriminability to categorization using isolated objects [37], objects in clutter [23] and scenes [38,39]. It has been shown that the perceptual dissimilarity between categories directly affects the speed of superordinate-level vs. basic-level categorization in pigeons [40]. Early work on scene and face processing already hinted at this contribution by showing, for instance, that the stimulus content across spatial scales affects scene categorization performance [41]. Subsequent work has also shown that the manipulation of the phase and amplitude spectra of an image affects behavioral responses during scene superordinate categorization [42,43]. More recently, it has been shown that a low-level perceptual similarity measure based on stimulus contrast predicts the ease of categorization judgments for both artificial stimuli [44] and natural scenes [45]. Our study further demonstrates that it is possible to use modern machine learning tools and computer vision databases to predict human behavioral responses for many categorization tasks across taxonomic levels. In experiment 2, we further showed that it is possible to use the model to sample stimuli in order to reverse the “superordinate advantage” rendering participants’ superordinate categorization arbitrarily slower and less accurate than basic categorization. Previous work has shown that it is possible to manipulate level-of-categorization effects by controlling the similarity between face stimuli [46] and the typicality of objects [47]. Here, we used the model to sample stimuli based on computed discriminability values, possibly making a superordinate categorization task harder compared to a basic level categorization task simply by sampling the right stimuli. Our results suggest that the superordinate advantage is at least in part driven by the perceptual discriminability of target and distractor stimulus sets. Simply put, superordinate-level categorization tasks tend to be easier than basic-level categorization tasks leading to observers’ behavioral responses that are faster and more accurate. This is consistent with the somewhat higher accuracy of both connectionist models [48] and modern computer vision systems for categorization at the superordinate vs. basic level [49] and is consistent with the fact that children learn to categorize natural object categories at the superordinate level first [48,50]. Our results are consistent with the differentiation theory [51] and the Parallel Distributed Processing (PDP) theory [52] in that level-of-categorization effects as reported in multiple studies [5–7,9, 11–13] arise, not because of privileged processing at particular taxonomic levels, but because of differences in perceptual discriminability across tasks. In addition, this perceptual explanation rules out an interpretation of level-of-categorization effects based on the “global-to-specific” theory of categorization, whereby categorization at more global (coarser) categorization stages need to be completed before categorization at more specific (finer) levels can begin. Hence, one would expect a basic advantage over subordinate categorization (e. g. , detection preceding identification [53]) as well as a superordinate and attribute advantage over basic and subordinate categorizations [4,5, 11]. Our results demonstrate that observed differences in timing across categorization tasks do not necessarily reflect the fact that some categorization tasks take precedence over others (see also [19,20]). While our results point to perceptual discriminability as playing a fundamental role in level-of-categorization effects, additional memory-related factors such as typicality are likely to affect rapid categorization. More generally, a complete model should also take into account known semantic contributions to visual categorization. One proposal is that mental representations of categories across taxonomic levels occupy nodes in a semantic network [54]. The rapid perceptual categorization mechanisms studied here may determine which nodes get activated first before activation spreads to other nodes enabling the slower retrieval of information at other levels of categorization [52]. The present study also has implications for models of category learning and models on the development of visual expertise. It is known that experts can override the supremacy of one level of categorization found in novices with their own level of expertise (e. g. the subordinate level becomes faster for bird experts that are over-trained at the subordinate level, the basic-level becomes faster for Chinese character experts that are over-trained in discriminating characters at the basic level (irrespective of font and writing style, see [22] for review). One simple explanation consistent with our results is that practice for a task leads to long-term perceptual learning that increases the discriminability between targets and distractors, making participants faster and more accurate. Despite its ability to account for behavioral responses, the proposed model remains relatively simple. We used a rudimentary visual representation based on the “gist” algorithm [25] and off-the-shelf machine learning classifiers (see [39] for a similar model used to explain the scene categorization advantage when scenes contained consistent vs. inconsistent objects). However, the fact that a relatively simple (V1-like) model of feature computation, seems sufficient to account for behavioral responses does not necessary imply that rapid scene categorization is based on low-level visual processing. We have tested alternative visual representations based on common features used in computer vision and found all these models to be relatively correlated. This could possibly reflect a limitation inherent to the ever limited size of natural image databases [55] as well as possible inherent biases such as photographers selecting vantage points [56]. Note that such image bias is quite different from the “natural bias” reported here in terms of differences in perceptual discriminability across categorization tasks, which is likely to reflect physical properties of our visual environment as opposed to biases in the image dataset per se. In addition, while the superordinate advantage has been described for other classes of stimuli beyond scenes such as animals [16] or faces [5,6], we have here only considered the relevance of the model for scene categorization. The use of a similar framework for other type of classifications would be likely to require more elaborated visual representations. In theory, it should be relatively straightforward to test additional perceptual representations—possibly reflecting higher level visual processes (see [23] for a review). A possible neural correlate for decision boundaries includes neurons with category-like tuning found throughout the cortex such as within the ventral stream, the prefrontal cortex (PFC) and the parietal cortex [57] and/or attentional processes that would differentially modulate individual feature dimensions according to their task diagnosticity [58]. Perceptual spaces in practice tend to be more flexible than assumed in the model as novel features can be learned (i. e. , the meaning of some of the dimensions may change and/or dimensions may be added as a result of learning and plasticity) and perceptual spaces can be reshaped by task history and other cognitive factors [59]. Alternative categorization algorithms to the proposed decision boundary have been described based on either the distance to category prototypes [60] or the distance to individual exemplars [61]. The proposed discriminability measures based on the distance between stimuli and decision boundaries could be easily extended to distances to exemplars or prototypes [62]. While it is expected that a better model of the categorization process should improve the fit to behavioral data, it is unlikely to change any of our conclusions, since categorization models tend to produce similar behavioral predictions. Overall, our study provides a computational level explanation for systematic variations found in behavioral responses for rapid categorization tasks across taxonomic levels, challenging several existing theories of visual processing and suggesting, instead, that observed differences in behavioral responses may simply reflect natural variations in perceptual discriminability.
summary: The speed of sight has fascinated scientists and philosophers for centuries. In the blink of an eye, observers can rapidly and effortlessly perform a variety of categorization tasks such as categorizing a scene as open, as natural, or as a beach. The past decade of work has shown that there exist systematic differences in behavioral responses across different categorization tasks: For instance, participants appear to be faster and more accurate at categorizing a scene as outdoor (i. e. , superordinate level) compared to categorizing a scene as a beach (i. e. , basic level). Here, we describe a computational model combined with human psychophysics experiments, which help shed light on the underlying mechanisms. Using a large natural scene database, we trained machine learning algorithms for different categorization tasks and showed that it is possible to derive confidence measures that accurately predict variations in participants’ behavioral responses across categorization tasks and stimulus sets. Using the computational model to sample stimuli for a human experiment, we demonstrated that it is possible to reverse the superordinate advantage, rendering human observers superordinate categorization slower and less accurate than basic categorization—effectively challenging previous interpretations of the phenomenon. The study further offers a vivid example on how computational models can help summarize and organize existing experimental data as well as plan and interpret new experiments.
title: Explaining the Timing of Natural Scene Understanding with a Computational Model of Perceptual Categorization