Visual pattern detection and discrimination are essential first steps for scene analysis. Numerous human psychophysical studies have modeled visual pattern detection and discrimination by estimating linear templates for classifying noisy stimuli defined by spatial variations in pixel intensities. However, such methods are poorly suited to understanding sensory processing mechanisms for complex visual stimuli such as second-order boundaries defined by spatial differences in contrast or texture. We introduce a novel machine learning framework for modeling human perception of second-order visual stimuli, using image-computable hierarchical neural network models fit directly to psychophysical trial data. This framework is applied to modeling visual processing of boundaries defined by differences in the contrast of a carrier texture pattern, in two different psychophysical tasks: (1) boundary orientation identification, and (2) fine orientation discrimination. Cross-validation analysis is employed to optimize model hyper-parameters, and demonstrate that these models are able to accurately predict human performance on novel stimulus sets not used for fitting model parameters. We find that, like the ideal observer, human observers take a region-based approach to the orientation identification task, while taking an edge-based approach to the fine orientation discrimination task. How observers integrate contrast modulation across orientation channels is investigated by fitting psychophysical data with two models representing competing hypotheses, revealing a preference for a model which combines multiple orientations at the earliest possible stage. Our results suggest that this machine learning approach has much potential to advance the study of second-order visual processing, and we outline future steps towards generalizing the method to modeling visual segmentation of natural texture boundaries. This study demonstrates how machine learning methodology can be fruitfully applied to psychophysical studies of second-order visual processing.
Many of the most common functions of sensory systems involve detection, identification or discrimination of particular stimuli. For example, a critically important ability of early visual processing is to segment images into distinct regions, which may correspond to cohesive surfaces or objects [1]. To better understand the underlying mechanisms, it would be useful to fit biologically plausible models of surface segmentation to psychophysical data, and assess how well such models can account for independent datasets. A widely used approach to modeling human psychophysics has been to assume that performance can be understood in terms of how well a visual stimulus matches an internal “template”, typically modeled as a linear spatial filter. This approach uses high-dimensional regression models fit to data from tasks in which a subject detects or classifies a sensory stimulus presented with superimposed white noise [2–5], to recover an estimate of the linear filter that best accounts for the data. This system-identification (SI) approach to psychophysics, often called “psychophysical reverse correlation” or “classification image analysis”, parallels neurophysiology studies that fit linear filter models to neural responses elicited by white noise stimuli [6,7]. Psychophysical SI methods have been applied in varied problem domains and yielded valuable insight into perceptual mechanisms [4,8–13]. One limitation of most previous psychophysical system identification studies has been the use of stimuli that are typically defined by spatial or temporal variations in first-order statistics like luminance or color. However, many natural visual stimuli such as occlusion boundaries (e. g. , the top image patch in Fig 1a, showing animal fur occluding a forest floor) are defined by spatial variations in higher-order image statistics, e. g. texture or contrast, as well as simple luminance differences [14–18]. Such “second-order” boundaries (Fig 1a, bottom) have been of particular interest because their detection cannot be explained by simple linear filters [19,20], and consequently human performance cannot be readily modeled using conventional psychophysical SI methods. More generally, there is a need for a psychophysical modeling approach that can handle a broader range of stimuli and tasks. Here we present a novel machine learning method to extend psychophysical SI to more complex naturalistic stimuli such as second-order boundaries. Our general approach is to fit image-computable hierarchical neural network models of second-order vision (Fig 2) directly to psychophysical trial data, similar to recent studies using multi-stage models to characterize nonlinear responses in sensory neurons [21–24]. Our application of this framework generalizes a class of biologically plausible “Filter-Rectify-Filter” (FRF) models that have often been invoked to explain second-order processing [20]. However, in contrast to previous general schemes for second-order vision [25–27] and quantitative models [28,29], we take a machine learning approach in which we estimate the defining model parameters by direct fitting to psychophysical trial data. This permits more incisive tests of predictions on novel datasets in order to validate our models, and to evaluate competing models. We focus on contrast-defined second-order boundaries (Fig 1a and 1b) since they are well studied psychophysically [30–34], and there are neurons in the early mammalian visual cortex that give selective responses to contrast modulation stimuli [35–38]. Performance on two tasks making use of contrast-defined boundaries is examined: (1) identification of boundary orientation, and (2) discrimination between two slightly off-vertical boundaries (Fig 1c). The results demonstrate that subjects utilize spatial information in markedly different ways for these two tasks, employing “region-based” processing for the orientation identification task and “edge-based” processing for the orientation discrimination task, in both cases consistent with an ideal observer. We further demonstrate the power of this methodology for evaluating competing hypotheses regarding the nature of channel summation within the FRF scheme [29,39–41] by comparing the ability of two different model architectures to fit human performance on a contrast edge detection task in which the carrier pattern contains elements with two orientations. Post-hoc Bayesian model comparison [42–44] demonstrates a preference for a model in which carrier pattern orientation is integrated across channels by the second-stage filters, consistent with previous psychophysical studies [40]. Finally, we discuss potential applications of this methodology to future quantitative studies of first- and second-order cue integration [45–47] and natural boundary detection [18,48]. We suggest that the application of machine learning methodology similar to that presented here can potentially help to better quantify visual processing of natural occlusion boundaries and complex natural visual stimuli more generally.
The model used to fit psychophysical trial data (Fig 2) employed a first stage comprised of a bank of fine-scale Gabor filters that compute local oriented energy in a manner similar to V1 complex cells [49]. The second stage consisted of a pair of filters, representing the two task alternatives (L/R), each instantiated by a weighted sum of the normalized outputs of the first-stage filters. In contrast to the first-stage filters, these second-stage filters analyze the image at a more coarse spatial scale, integrating variations in contrast across the boundary. Finally, the outputs of these two second-stage filters are passed through a static non-linearity h (u) = |u|α and then compared to decide on a trial-by-trial basis if a given stimulus boundary appeared more likely to be tilted left or right (L/R), i. e. the probability 0 < P (R) < 1 of the stimulus being classified by the observer as R. The main goal of Experiments 1 and 2 was to estimate the shapes of the second-stage filters. The filter shapes are not specified beforehand, but rather are learned by fitting actual psychophysical data, with the goal of revealing how the observer spatially integrates texture information. To reduce model parameter space dimensionality, we performed downsampling between the first- and second-stage filters, i. e. pooling of spatially adjacent first-stage filter responses. Here we compared three ways of implementing downsampling within each locality: (1) taking the simple average (AVG), (2) taking the largest value (MAX) [50], and (3) taking just one of the values, e. g. the center one (sub-sampling, SUB) [26]. In Experiment 1 we considered all three pooling rules; most analyses utilized 16x16 downsampling, but 3 downsampling sizes (22x22,16x16 and 8x8) were compared. In Experiments 2 and 3 we considered only AVG pooling and a single downsampling size (Experiment 2: 22x22, Experiment 3: 12x12). We implement the model’s final (L/R) classification in two alternative ways: In the deterministic model (DET), if the output of the R filter is larger than that of the L filter (plus the bias term) so that P (R) > 0. 5, the stimulus will be classified as R, otherwise it is classified as L. In the stochastic model (STO), the output P (R) specifies only the probability of classifying the stimulus as R or L, thus introducing randomness to the model without requiring an additional free parameter. A common approach to make the estimation problem more tractable is to utilize reasonable prior constraints on the model parameter space [51–53]. Here we use two kinds of Bayesian priors: (1) ridge-regression [44,54] to penalize large weights (ridge), and (2) ridge-regression combined with an additional smoothness prior [51] to penalize excessively “rough” second-stage filter weight profiles (ridge + smooth). Model performance was evaluated by using different randomly selected partitions (called folds) of the data for training and testing, a standard practice in machine learning (i. e. , k-fold cross-validation [51]). Additional details of the model are given in Methods. To better assess the meaningfulness of the above model-fitting results, it is important to make quantitative measures of the fitted models’ accuracy in predicting observers’ responses to novel stimuli. Such measures can also address the relative merit of different variants of the estimation procedure (ridge vs ridge + smooth prior) and of the model details, such as downsampling method (AVG, MAX, SUB) and classification rule (DET, STO). In this section we describe some of our analyses of model performance, and refer to other assessments in the Supplementary Material. The perceptual filters obtained in Experiment 1 (Fig 3) seem consistent with the idea that observers are monitoring the entire stimulus region for each of the two possible boundary orientations. However, to rule out the possibility that such filters could also be observed if observers were utilizing some different kind of spatial processing, we implemented four alternative varieties of spatial summation as ideal observer models fit to category labels. We then fit the model architecture in Fig 2 to data from these ground truth ideal observers in order to assess whether our methodology is consistent in the sense that it recovers the actual set of filters generating the data. One possible alternative spatial summation that could mediate our task would be to simply monitor two adjacent “pizza slice” shaped quadrants defined by the four compass directions (2-slice). For instance, if one is monitoring the North and East slices, a difference in contrast between these slices is diagnostic for a right-oblique boundary. Implementing an ideal observer that utilizes such a subset of the spatial information (Fig 10a, top) and fitting our model architecture shown in Fig 2 to this data reveals consistent results: A single filter which monitors two slices only. Implementing an observer who monitors three adjacent slices (" 3-slice" ), for instance N+E and N+W (Fig 10a, second row) also reveals consistent filters. Another possible spatial processing approach would be to observe all slices but to only monitor for one of the two potential boundary orientations (" 1-filter" ) —simulating this model yields a consistent estimate of a single filter tuned to the monitored boundary (Fig 10a, third row). One can in principle observe filters somewhat similar to those we recover in Fig 3 by using a sub-optimal spatial processing model which only monitors a single pair of potentially informative slices, chosen at random on each trial. However, such a model leads to perceptual filters which are extremely “noisy” compared to those we find (Fig 10a, bottom row), so it is quite unlikely that our observers are actually using such spatial processing. The simulated ideal observer that produces filters providing the best match to those we obtain from human observers (Fig 3) makes use of two filters, each of which monitors the entire stimulus region for each of the two stimulus possibilities (Fig 10b). The first experiment demonstrated that human observers employ region-based processing for the orientation identification task. To see if our modeling method could also reveal situations where observers employ edge-based processing, we utilized a fine orientation discrimination task (Fig 1c, right). The orientation of a second-order boundary was perturbed slightly (JND at 32% contrast modulation—see Methods) from vertical, and observers judged the direction of perturbation (i. e. slightly left- vs right-oblique). In one variant of this task (Experiment 2-VAR), contrast modulation was varied in 9 logarithmic steps from 16% to 64% centered at 32% (plus zero modulation) to vary the task difficulty. In a second variant (Experiment 2-FIX), contrast modulation was fixed at 32%. In both variants of Experiment 2, the perturbation of orientation was fixed near its JND for each observer at 32% contrast modulation (CJD: 7 deg. , JJF: 6 deg. , VHB: 7 deg.). In the data analysis, a higher downsampling factor (22 x 22) was used to better reveal fine spatial details, and only AVG pooling was considered, since Experiment 1 demonstrated substantial robustness of spatial summation to pooling rules (S8 and S9 Figs). Fig 11 shows the second-stage filter weights from an observer (JJF) tested in both Experiment 2-VAR (Fig 11a) and Experiment 2-FIX (Fig 11b). Unlike in Experiment 1, here observers employ an “edge-based” summation, assigning substantially more weight to texture regions near the boundary. Bootstrapped significance plots for individual weights are shown in S15 Fig, and consistent results from other observers are shown in S16 Fig. The edge-based processing employed by these observers is very similar to that obtained from an ideal observer (S17 Fig). As with Experiment 1, we find a reasonably strong agreement between model predictions and observed performance on sets of novel data not used for model parameter estimation (S18 Fig), and find an expansive second-stage filter nonlinearity for Experiment 2 (S7 Table). Taken together, Experiments 1 and 2 show that our modeling methodology can reveal different kinds of spatial processing that observers may employ for different kinds of psychophysical tasks. We also considered the possibility that our results in Experiment 2 (Fig 11) could also be explained by two off-orientation filters defined on the spatial scale of the whole stimulus [61]. We defined an ideal observer with two off-orientation filters (+/- 20 deg.) defined on the scale of the stimulus and simulated responses of this observer to the stimuli used in Experiment 2. This analysis yielded very different filters from those we observe in Fig 11, as we simply recovered the original off-orientation filters (S22 Fig), again demonstrating the consistency of our method (as in Fig 10). Based on these results, we suggest that it is more likely that the relevant neural mechanisms employed in Experiment 2 are contrast-modulation tuned neurons with smaller receptive fields localized to the boundary (see Discussion). In Experiments 1 and 2, we considered a particular FRF model architecture and focused on characterizing the spatial processing that observers employ in different second-order vision tasks. However, one of the most powerful applications of this methodology is model comparison—i. e. comparing the ability of alternative models, embodying different assumptions about an FRF model architecture, to account for psychophysical data. Indeed, a major focus of much of the second-order vision literature has been to design experiments whose express goal is to test competing hypotheses of FRF architecture [29,39–41,62]. We applied our method to ask how second-stage filters for contrast modulation integrate orientation modulations across multiple kinds of first-stage filters or channels. In many previous computational studies of second-order processing, the FRF models were comprised of second-stage filters, each of which only analyzes a single orientation/spatial frequency channel [28,63,64], with subsequent “late summation” of their signals. However, other psychophysical studies have suggested “early summation” FRF models, in which second-stage filters integrate contrast modulation across texture channels [39,40,62]. To address this question systematically, we used a boundary orientation identification task as in Experiment 1, but with the carrier pattern containing two orthogonal orientations of micropatterns (Fig 1a, bottom), which would be processed by distinct first-stage orientation-selective channels. Since the nature of spatial summation is not critical to this question, we used a coarser downsampling factor (12 x 12), and as above, only AVG pooling was used. We considered two models of how information is integrated by the second-stage filters. In a “late summation” Model 1 (Fig 12a, top), each first-stage channel is analyzed by its own pair of second-stage filters (L/R oblique). Alternatively, in an “early summation” Model 2 (Fig 12a, bottom), each second-stage filter receives inputs from both first-order channels. As in Experiments 1 and 2, we find that both the models are in reasonably good agreement with observer performance on the task (S20 Fig). Also there was reassuring consistency in that both the fitted models exhibited similar region-based weight maps, with an expansive second-order nonlinearity (S8 Table). When comparing Models 1 and 2 using standard model comparison techniques such as the Bayes Information Criterion, or BIC [42,44], there is no need to correct for overfitting since both models have the same number of parameters. This is because when parameter space dimensionality is equal, the complexity-penalizing terms of the BIC cancel, so that ΔBIC = ln p2 (D) − ln p1 (D), where pi (D) denotes the likelihood of the data (D) under model i evaluated at the MAP estimate of the model parameters. That is, when two models are of equal complexity and equally likely a priori, we simply choose the one with higher log-likelihood of the data. The difference in log-likelihoods (ΔBIC) is identical to the natural log of the Bayes factor B21=p2 (D) p1 (D). Standard effect size conventions state that when 2 ln B21 > 10 (corresponding to > 150-fold higher probability of Model 2), the preference is considered to be “very strong” [43]. We see in Fig 12b that for all three observers tested in Experiment 3, the evidence (measured by 2 ln B21) in favor of the orientation opponent model (Model 2) exceeds the standard criterion (10) for a very strong preference. Ideal observer analysis also revealed a very strong preference for the orientation-opponent model fit to stimulus category labels (2 ln B21 = 478. 4). With a large number of trials, a strong preference for one binomial response model does not require a huge difference in predicting observer performance (Appendix A). We see in Fig 12d that although Model 2 very consistently does a better job of matching observer performance than Model 1, that the differences between predicted proportions correct are fairly small. We also wanted to compare the predictive abilities of the two models using a cross-validation approach, by evaluating their agreement with observer performance on data not used for training. Therefore we performed a bootstrap analysis in which both models were fit on randomly selected samples of 4000 stimuli and tested on the remaining 1000 (50 bootstraps). To optimize the generalization performance, each model was trained using the value of the ridge regression hyperparameter λ which yielded the best generalization (S19 and S21 Figs). The results in Fig 12c show a consistent preference across observers for Model 2 (magenta line). Performing a 3 x 2 factorial ANOVA (3 observers, 2 models, 50 observations each) on the test set log-likelihoods revealed significant main effects of both observer (F2,294 = 201. 085, p < 0. 001) and model (F1,294 = 39. 811, p < 0. 001), but no significant interaction (F2,294 = 3. 001, p = 0. 051). In summary, both analysis approaches (Fig 12b and 12c) show a strong and consistent preference for the early summation version (Model 2).
Psychophysical system identification is critically limited by the modest amount of data that can be collected in an experiment. Although constraints or priors can help alleviate the demand for data [51,67,68], it is also often necessary to fit simplified models requiring fewer parameters, which omit many known biological details. Here we discuss several modeling simplifications which may need to be addressed when extending this methodology to more complicated second-order stimuli and tasks. Despite the simplifications made in the present study, it represents a substantial methodological advance over previous efforts to model second-order visual perception. Although previous studies have developed image-computable FRF models, they have generally fixed not only the model architecture but also the shapes of the second-stage filters. The present study demonstrates how the second-stage filter shapes can be “learned” directly from experimental data in order to reveal the nature of spatial integration used in different tasks (Experiments 1 and 2). Most previous FRF models in human psychophysics studies were conceptual or qualitative, without quantitative fitting of model parameters to data. Two recent studies [28,90] estimated free parameters of an FRF model by fitting threshold data as functions of stimulus parameters. Since thresholds are typically determined at only a few levels of a one-dimensional independent variable, this yields relatively few data points, greatly limiting the number of model parameters one can estimate. In our work, we directly estimate hundreds of model parameters by fitting stimulus-response data from thousands of psychophysical trials, enabling us to characterize the shapes of the second-stage filters. In addition to extending the FRF modeling literature, our work also complements and extends previous classification image studies, which have generally been applied in the context of first-order vision [4,5]. Although classification images yield image-computable predictive models of observer responses, relatively few studies validate these models by testing their predictive performance on a set of stimuli not used for model estimation. We tested model performance on novel stimuli not used for model estimation using both standard measures, e. g. percentage correct, as well as more advanced methods such as double-pass analysis [58,59] and decision variable correlation [60]. Finally, in contrast to most classification image studies, here we systematically address the issue of model comparison. Psychophysical system identification can be formulated in the framework of estimating a generalized linear model [51,67,68], which is unique up to choice of link function. Therefore, issues of model comparison (beyond coefficient shrinkage) do not naturally arise in this context. However, the FRF models relevant to second-order vision can vary greatly in architectural details (e. g. , number of filtering layers, nonlinearities, connectivity, etc.), and model comparison is of great interest. In Experiment 3, we compare two competing FRF architectures making different assumptions about how the second-stage filters integrate across orientation channels, finding a preference for a model with early summation of first-order channels by the second-stage filters, consistent with previous psychophysical work [39,40,76]. We applied our methodology to the question of whether or not observers take a region-based or an edge-based approach to segmenting a contrast boundary (Experiment 1). To the best of our knowledge, our experiment is the first time the issue of edge/region-based processing has been addressed for boundaries defined by differences in contrast, as it has for other second-order boundaries [55–57]. This question is of particular interest in light of several studies that have revealed possible neurophysiological mechanisms for contrast-boundary detection [36–38,91]. This work has demonstrated a population of contrast-modulation (CM) tuned neurons in early visual cortex, which exhibit band-pass spatial frequency and orientation tuning to second-order modulation envelope frequency, consistent with a Gabor-shaped second-stage filter applied to the outputs of a bank of first-stage filters selectively tuned to the carrier pattern [19]. If such CM-responsive neurons were the mechanism responsible for psychophysical detection of CM boundaries, one might expect the 1-D weighting profiles to have a Gabor-like shape, placing greater weight on regions near the boundary. However, we find that this was not the case and that all areas of the image are weighted equally for the orientation-identification task (Experiment 1, Figs 3 and 4). Furthermore, in an additional control manipulation where we increased the carrier frequency in order to make sure CM neurons with smaller RFs would be relatively more engaged (Experiment 1-HFC), we still did not see any change in the type of spatial integration (Fig 5). These findings are consistent with at least three distinct possibilities (not mutually exclusive) for the underlying neural mechanisms. One possibility is that task performance reflects CM-responsive neurons, like those described in V2 with large receptive fields (> 4 dva) encompassing the entire stimulus used in our experiments. Such neurons would prefer a carrier frequency much higher than their envelope frequency, which is plausible since primate V2 CM-responsive neurons can have preferred carrier: envelope ratios of up to 40: 1 [38]. A second possibility is that psychophysical performance is mediated by a range of Gabor-like second-order mechanisms covering a range of spatial scales, and that our measured 1-d profiles reflect an ensemble of their contributions. Another possibility is that an important role is played by other neurons sensitive to texture patterns, such as those described in ventral stream areas [81–83,92]. Clearly, the fact that we can readily distinguish non-juxtaposed textures implies that there must be mechanisms for texture representation that operate in the absence of a boundary. However, in some cases texture segmentation can benefit from boundary information, as in the textural Craik-Cornsweet illusion [55]. For example, [57] demonstrated that direct juxtaposition of textures having different orientations gives rise to local junction or corner cues that can support segmentation. Such enhanced segmentation might be mediated by neurons such as those described by [93,94], which are selective for such cues. However, other work on texture segmentation has shown a minimal contribution of information along the boundary. A study using natural image patches in a task of discriminating occlusion boundaries from uniform textures found that removing the boundary region had little effect on performance [18]. However, performance on natural occlusion edge and junction detection tasks can be seriously degraded when textural information, which is by its nature defined over a region, is removed [18,48]. Similarly, segmentation of complex textures defined by letters shows little effect of boundary information [56]. In general, the answer to this question is most likely dependent on the particular stimulus and the nature of the psychophysical task, as indicated by the very different results of our first two experiments. In our view, the question of “edge” versus “region”-based processing should be treated with some caution, since clearly for a very large stimulus extending into the far periphery there will be reduced weighting at the largest eccentricities, since visual acuity falls off sharply outside the fovea. In such a case the real question of interest would be the shape of the fall-off in weight as a function of stimulus eccentricity, and how this fall-off may relate to putative neural codes. Our method is well suited to address such issues for many different kinds of second-order boundaries and tasks, much as standard classification images can be similarly used for first-order boundaries. We further demonstrate that our modeling methodology is consistent, meaning that it can accurately recover the actual perceptual filters employed in a task, provided the fitted and generating models have corresponding architectures. Simulation of ideal observers implementing sub-optimal spatial processing, e. g. utilizing only limited subsets of the texture, revealed filters which do not match those we observed experimentally (Fig 10a). In contrast, a strong agreement was found with results from simulating an observer monitoring the whole stimulus for each of two possible boundary orientations (Fig 10b), strengthening our interpretation that this is the kind of spatial summation actually being employed. In Experiment 1 our results showed that human observers use every part of the image to perform the task (Fig 3), consistent with an ideal observer that integrates information from all image regions (Fig 10b). In terms of possible mechanisms, we might therefore expect CM-tuned neurons with larger receptive fields that integrate over the expanse of the stimulus image to be engaged in Experiment 1. The results in Experiment 2 showed that only the texture information near the boundary was being utilized by humans (Fig 11), again consistent with an ideal observer that is indifferent to the stimulus elsewhere. This suggests that the most useful neural substrate in Experiment 2 might be CM-responsive neurons with smaller receptive fields localized near the boundary, that integrate texture only over small extents to either side of the boundary. We demonstrate using simulation that one cannot explain the filters we observe in Experiment 2 by assuming two large off-orientation receptive fields (S22 Fig), since otherwise we would have recovered filters with slightly non-vertical orientation rather than filters localized to the vertical (Fig 11). We conjecture that our results are more consistent with the possibility that observers are making use of contrast-modulation tuned neurons having smaller receptive fields localized to the boundary. In both Experiments 1 and 2 we found that humans behaved as ideal observers with regard to their spatial integration of texture. Such a result should not be regarded as inevitable or having little relevance for questions of neural mechanism. Previous psychophysical studies (including several classification image studies) have demonstrated that humans often do behave ideally in perceptual tasks [12,95–98] and furthermore such ideal behavior can be suggestive of neuronal mechanisms underlying the behavior [99–101]. However in many other cases, human psychophysical performance does not match that of an ideal observer [2,3, 8,96,98,102,103]. Interestingly, [96] demonstrated that even with the same stimulus, changes in the psychophysical task could alter whether or not human observers behaved ideally. We feel that the demonstration of ideal behavior in Experiments 1 and 2 is significant, and strongly suggests possible neural mechanisms, for example suggesting a greater involvement of neurons with larger CM-responsive receptive fields in the task of Experiment 1, and those with smaller receptive fields in Experiment 2. In general, the relationship between neural response properties and the perceptual filters obtained in classification image studies is complex [4,104]. Perceptual filters do not generally correspond to fixed," hard-wired" neural populations, but are most likely actively constructed from available neuronal populations through perceptual learning. Consistent with this idea, previous work has demonstrated that observers’ perceptual summation gets successively closer to that of the ideal observer with practice [10,105]. It would be of great interest for future research to explore the dynamics of the perceptual filtering strategies that observers employ in our tasks. A common ingredient to the various FRF models proposed to account for psychophysical texture segmentation data is a set of fine-scale first-stage filters summed by one or more second-stage filters defined on a much larger spatial scale. However one aspect in which such models differ is whether the individual second-stage filters analyze first-stage filter responses at a single orientation/spatial frequency, or whether they integrate across heterogeneous first-stage channels. In one standard model of FRF processing, each second-stage filter only analyzes the outputs of a single first-stage orientation/spatial frequency channel [25,63,64]. However a number of psychophysical results have called this model into question. One study using a sub-threshold summation paradigm found support for a model utilizing linear integration across carrier orientation for detecting contrast modulations [40]. Another study making use of energy-frequency analysis revealed the existence of orientation-opponent mechanisms in second-order vision [62]. Other work has suggested at least partly independent mechanisms for detecting second-order modulations defined by contrast and spatial/frequency orientation, with the suggestion that contrast modulations may be pooled across first-stage channels [39]. In Experiment 3 we find support for a model with second-stage filters that integrate across first-stage orientation channels, consistent with the latter psychophysical studies. Although CM-responsive neurons described so far exhibit narrow tuning for carrier orientation [38], this discrepancy could be due to the use of grating carriers rather than multi-orientation micropatterns. This suggests that that possible effects of carrier orientation bandwidth might be worth future investigation, both in human psychophysics and for CM-responsive neurons. In our opinion, Experiment 3 best demonstrates the power of this methodology compared to traditional linear classification image approaches: The ability to explore a much larger space of non-linear perceptual models embodying various assumptions about neural mechanisms. Rather than simply comparing recovered filter shapes to ideal behavior as in most previous classification image studies [5], here we directly compare two biologically plausible hypotheses of how contrast modulation is integrated across orientation channels. Our finding of clear support for an early summation model [40] suggests that an interesting direction for future neurophysiological investigation would be to examine neural responses to contrast modulations of carrier patterns containing multiple orientations. Such a study would be quite novel, as previous neurophysiological investigations of contrast modulation tuning have only considered a single carrier orientation [35,38]. Statistical machine learning has in recent years become a powerful and widely applied methodology in vision and in computational neuroscience. Our work demonstrates that the same machine learning methodology used to characterize neural coding [21,52,88,106,107] and first-order visual processing [5,51] can be fruitfully applied to extend psychophysical system identification approaches to complex visual stimuli, including second-order boundaries. We anticipate that future applications of this approach may lead to a better understanding of how multiple first- and second-order cues are combined by human observers to detect natural boundaries, and provide insights into neural and computational mechanisms of image region segmentation.
Here we derive a simple expression for the Bayes factor for two binomial models. We show using our formula that over large numbers of trials, a small difference in the accuracy of the two models can lead to a very strong preference for the more accurate model, as measured by the Bayes Factor. Let M1, M2 be two binomial models, each predicting for a fixed stimulus, positive response (r = 1) probabilities of p1, p2 and negative response (r = −1) probabilities 1 − p1,1 − p2. Assume that we perform n experimental trials and observe proportion p* positive responses and (1 − p*) negative responses. Let D = r1, …, rn denote the observer responses. The Bayes Factor is B21=p (D|M2) p (D|M1), (A. 1) and taking the natural log we obtain lnB21=lnL2−lnL1, (A. 2) where Lk = p (D|Mk) for (k = 1,2). Expanding the log-likelihoods, we obtain lnLk=∑ri=1lnpk+∑ri=0ln (1−pk), (A. 3) where pk = p (r = 1|Mk). Since there are np* trials with positive responses and n (1 − p*) trials with negative responses, we may simplify (A. 3) to lnLk=n (p*lnpk+ (1−p*) ln (1−pk) ). (A. 4) Noting that the bracketed term in (A. 4) is simply the negative cross-entropy between two binomial models, and using the fact that the cross-entropy C (p, q) relates to the entropy H (p) and the KL divergence DKL (p||q) by the relation C (p, q) = H (p) + DKL (p||q), we can re-write (A. 4) as lnLk=−n (H (p*) +DKL (p*||pk) ). (A. 5) Plugging (A. 5) into (A. 2), the natural log of the Bayes factor becomes lnB21=n (DKL (p*||p1) −DKL (p*||p2) ), (A. 6) and we obtain the final expression B21=en (DKL (p*||p1) −DKL (p*||p2) ). (A. 7) We see from the final result (A. 7) that the Bayes factor magnitude is exponential in the number of trials (n), meaning that small differences in model performance (measured by the difference in the KL divergences) over a large number of trials can lead to very large values for the Bayes factor. Similarly, we see from (A. 6) that the standard model selection criterion 2ln B21 [43] grows linearly as n, meaning that a small difference in performance may accumulate over trials to produce large values of this criterion. Concretely, consider the case where p* = 0. 75, p1 = 0. 73, p2 = 0. 74 for n = 5000 trials. Qualitatively, we see that both models are slightly inaccurate, with Model 2 agreeing with the observer more than Model 1, but with a difference of only 0. 01. Applying formula (A. 6) gives us ln B21 = 3. 8458 and therefore 2ln B21 = 7. 6916, corresponding to a “strong” preference for Model 2 according to standard criteria [43]. Re-working this example with a slightly larger difference in performance (0. 02) by setting p1 = 0. 72 yields a “very strong” preference for Model 2 (2ln B21 = 20. 2224), and a 2. 46 × 104 fold preference for Model 2. This effect is even more pronounced for predictions of very high (or low) success rates: setting p* = 0. 95, p1 = 0. 92, p2 = 0. 94 yields 2ln B21 = 60. 4679, greatly exceeding the standard criterion (10) for “very strong” evidence in favor of Model 2.
Many naturally occurring visual boundaries are defined by spatial differences in features other than luminance, for example by differences in texture or contrast. Quantitative models of such “second-order” boundary perception cannot be estimated using the standard regression techniques (known as “classification images”) commonly applied to “first-order”, luminance-defined stimuli. Here we present a novel machine learning approach to modeling second-order boundary perception using hierarchical neural networks. In contrast to previous quantitative studies of second-order boundary perception, we directly estimate network model parameters using psychophysical trial data. We demonstrate that our method can reveal different spatial summation strategies that human observers utilize for different kinds of second-order boundary perception tasks, and can be used to compare competing hypotheses of how contrast modulation is integrated across orientation channels. We outline extensions of the methodology to other kinds of second-order boundaries, including those in natural images.
neural networks
social sciences
neuroscience
human performance
artificial intelligence
vision
neuronal tuning
computer and information sciences
animal cells
behavior
psychology
cellular neuroscience
cell biology
neurons
psychophysics
biology and life sciences
cellular types
sensory perception
machine learning
Modeling second-order boundary perception: A machine learning approach