A central goal in the study of the primate visual cortex and hierarchical models for object recognition is understanding how and why single units trade off invariance versus sensitivity to image transformations. For example, in both deep networks and visual cortex there is substantial variation from layer-to-layer and unit-to-unit in the degree of translation invariance. Here, we provide theoretical insight into this variation and its consequences for encoding in a deep network. Our critical insight comes from the fact that rectification simultaneously decreases response variance and correlation across responses to transformed stimuli, naturally inducing a positive relationship between invariance and dynamic range. Invariant input units then tend to drive the network more than those sensitive to small image transformations. We discuss consequences of this relationship for AI: deep nets naturally weight invariant units over sensitive units, and this can be strengthened with training, perhaps contributing to generalization performance. Our results predict a signature relationship between invariance and dynamic range that can now be tested in future neurophysiological studies. Invariances to image transformations, such as translation and scaling, have been reported in single units in visual cortex, but just as often sensitivity to these transformations has been found (El-Shamayleh and Pasupathy, 2016 , Sharpee et al. 2013 , Rust and DiCarlo, 2012 . Similarly, in deep networks there is variation in translation invariance both within and across layers (Pospisil et al., 2018 , Shen et al., 2016 , Shang et al., 2016 , Goodfellow et al., 2009 . Notionally, information about the position of the features composing objects may be important to category selectivity. For example, the detection of eyes, nose, and lips are not sufficient for face recognition, the relative positions of these parts must also be encoded. Thus it is reasonable to expect some balance between invariance and sensitivity to position. We empirically observe that in a popular deep network, in both its trained and untrained state, invariant units tend to have higher dynamic range than sensitive units (Figure 1B and C) . This raises the possibility that the effective gain on invariant units into the subsequent layer is stronger than that of sensitive units. Here we provide theoretical insight into how rectification in a deep network could naturally biase networks to this difference between invariant and sensitive units. We do this by examining how co-variance of a multivariate normal distribution is influenced by rectification, and we then test these insights in a deep neural network. We have documented an empirical relationship between the dynamic range of unrectified units in a deep network and their invariance. We provided a simple 1st order statistical model to explain this effect in which rectification caused the population representation to primarily vary in dimensions that were invariant to small image perturbations, whereas small perturbations were represented in directions of lower variance. Further work can investigate whether this imbalance improves generalization because of the emphasis placed on invariant over sensitive units. We note this relationship is weaker in the trained then untrained network further work can udnerstand this difference. Our approximations assumed low covariance between input units and homoegenous input variance while this may be expected in a random network it may not be true in a trained network. More crucially further theoretical work should consider the influence of co-variance between input units and invariance of output units as a function of weights. To extend insights from simplified, artificial networks to neurobiology, it will first of all be important to test whether cortical neurons showing more invariance also tend to have a higher dynamic range. If they do, this will establish a fundamental theoretical connection between computations of deep networks and the brain.
Rectification in deep neural networks naturally leads them to favor an invariant representation.
{"src": "scitldr-AIC_train", "paper_id": "r1lwNXKU8r"}