Sparse code formation in the primary visual cortex (V1) has been inspiration for many state-of-the-art visual recognition systems. To stimulate this behavior, networks are trained networks under mathematical constraint of sparsity or selectivity. In this paper, the authors exploit another approach which uses lateral interconnections in feature learning networks. However, instead of adding direct lateral interconnections among neurons, we introduce an inhibitory layer placed right after normal encoding layer. This idea overcomes the challenge of computational cost and complexity on lateral networks while preserving crucial objective of sparse code formation. To demonstrate this idea, we use sparse autoencoder as normal encoding layer and apply inhibitory layer. Early experiments in visual recognition show relative improvements over traditional approach on CIFAR-10 dataset. Moreover, simple installment and training process using Hebbian rule allow inhibitory layer to be integrated into existing networks, which enables further analysis in the future.
Lateral connections have been widely used in models in neuroscience. Our work is loosely based on E-I net @cite_4 . In E-I net, simple cells are divided into two types of neurons, excitatory neurons and inhibitory neurons. One inhibitory cell sends an amount of inhibitory current directly to all excitotary simple cells and other inhibitory cells. Inhibitory cells predict the redundant part of the network activity, thus decorrelate the activity of the excitatory cells by suppressing redundant spiking activity. In our networks, relationship between encoding cells is stored in connections between two layers. Inhibitory cells only do the computation based on encoding signals and the corresponding weights. In one recent work that shares the idea of local computation @cite_0 , spiking signal is chosen using winter-take-all rule while the rest is grounded to 0. Moreover, the competition range is predefined. In other works aiming at learning structural features such as structural sparse coding @cite_10 and topographical ICA @cite_1 , structural order emerges through regularization. Meanwhile, with inhibitory layer, neighborhood rather than order is obtained through ad hoc optimization.
@cite_0 @cite_10 @cite_1 @cite_4
2099049980 2161977692 2025539936
Local competition among neighboring neurons is common in biological neural networks (NNs). In this paper, we apply the concept to gradient-based, backprop-trained artificial multilayer NNs. NNs with competing linear units tend to outperform those with non-competing nonlinear units, and avoid catastrophic forgetting when training sets change over time. This work describes a conceptually simple method for structured sparse coding and dictionary design. Supposing a dictionary with K atoms, we introduce a structure as a set of penalties or interactions between every pair of atoms. We describe modifications of standard sparse coding algorithms for inference in this setting, and describe experiments showing that these algorithms are efficient. We show that interesting dictionaries can be learned for interactions that encode tree structures or locally connected structures. Finally, we show that our framework allows us to learn the values of the interactions from the data, rather than having them pre-specified. Sparse coding models of natural scenes can account for several physiological properties of primary visual cortex (V1), including the shapes of simple cell receptive fields (RFs) and the highly kurtotic firing rates of V1 neurons. Current spiking network models of pattern learning and sparse coding require direct inhibitory connections between the excitatory simple cells, in conflict with the physiological distinction between excitatory (glutamatergic) and inhibitory (GABAergic) neurons (Dale9s Law). At the same time, the computational role of inhibitory neurons in cortical microcircuit function has yet to be fully explained. Here we show that adding a separate population of inhibitory neurons to a spiking model of V1 provides conformance to Dale9s Law, proposes a computational role for at least one class of interneurons, and accounts for certain observed physiological properties in V1. When trained on natural images, this excitatory–inhibitory spiking circuit learns a sparse code with Gabor-like RFs as found in V1 using only local synaptic plasticity rules. The inhibitory neurons enable sparse code formation by suppressing predictable spikes, which actively decorrelates the excitatory population. The model predicts that only a small number of inhibitory cells is required relative to excitatory cells and that excitatory and inhibitory input should be correlated, in agreement with experimental findings in visual cortex. We also introduce a novel local learning rule that measures stimulus-dependent correlations between neurons to support “explaining away” mechanisms in neural coding.