Inspired by the adaptation phenomenon of biological neuronal firing, we propose regularity normalization: a reparameterization of the activation in the neural network that take into account the statistical regularity in the implicit space. By considering the neural network optimization process as a model selection problem, the implicit space is constrained by the normalizing factor, the minimum description length of the optimal universal code. We introduce an incremental version of computing this universal code as normalized maximum likelihood and demonstrated its flexibility to include data prior such as top-down attention and other oracle information and its compatibility to be incorporated into batch normalization and layer normalization. The preliminary results showed that the proposed method outperforms existing normalization methods in tackling the limited and imbalanced data from a non-stationary distribution benchmarked on computer vision task. As an unsupervised attention mechanism given input data, this biologically plausible normalization has the potential to deal with other complicated real-world scenarios as well as reinforcement learning setting where the rewards are sparse and non-uniform. Further research is proposed to discover these scenarios and explore the behaviors among different variants. The Minimum Description Length (MDL) principle asserts that the best model given some data minimizes the combined cost of describing the model and describing the misfit between the model and data BID9 with a goal to maximize regularity extraction for optimal data compression, prediction and communication BID4 . Most unsupervised learning algorithms can be understood using the MDL principle BID10 , treating the neural network as a system communicating the input to a receiver. If we consider the neural network training as the optimization process of a communication system, each input at each layers of the system can be described as a point in a low-dimensional continuous constraint space BID13 . If we consider the neural networks as population codes, the constraint space can be subdivided into the input-vector space, the hidden-vector space, and the implicit space, which represents the underlying dimensions of variability in the other two spaces, i.e., a reduced representation of the constraint space. For instance, given a image of an object, the rotated or scaled version still refers to the same object, thus each image instance of the same object can be represented by a position on a 2D implicit space with one dimension as orientation and the other as size BID13 . The relevant information about the implicit space can be constrained to ensure a minimized description length of the system. This type of constraint can also be found in biological brains of primates: high-level brain areas are known to send top-down feedback connections to lower-level areas to select of the most relevant information in the current input given the current task BID2 , a process similar to the communication system. This type of modulation is performed by collecting statistical regularity in a hierarchical encoding process among brain areas. One feature of the neural coding during the hierarchical processing is the adaptation: in vision neuroscience, vertical orientation reduce their firing rates to that orientaiton after the adaptation BID1 , while the cell responses to other orientations may increase BID3 . These behaviors well match the information theoretical point-of-view that the most relevant information (saliency), which depends on the statistical regularity, have higher "information", just as the firing of the neurons. The more regular the input features are, the lower it should yield the activation. We introduce the minimum description length (MDL), such that the activation of neurons can be analogous to the code length of the model (a specific neuron or neuronal population) -a shorter code length would be assigned to a more regular input (such as after adaptation), and a longer code length to a more rare input or event.In this paper, we adopt the similar definition of implicit space as in BID13 , but extend it beyond unsupervised learning, into a generic neural network optimization problem in both supervised and unsupervised setting. Given the neuroscience inspiration described above, we consider the formulation and computation of description length differently. Instead of considering neural networks as population codes, we formulate each layer of neural networks during training a state of module selection. In our setup, the description length is computed not in the scale of the entire neural networks, but by the unit of each layer of the network. In addition, the optimization objective is not to minimize the description length, but instead, to take into account the minimum description length as part of the normalization procedure to reparameterize the activation of each neurons in each layer. The computation of the description length (or model cost as in BID13 ) aims to minimize it, while we directly compute the minimum description length in each layer not to minimize anything, but to reassign the weights based on statistical regularities. Finally, we compute the description length by an optimal universal code obtained by the batch input distribution in an online incremental fashion.We begin our presentation in section 2, formulating the problem setting in neural network training as a layer-specific model selection process under MDL principle. We then introduce the proposed regularity normalization (RN) method, its formulation and the incremental implementation. We also present several variants of the regularity normalization by incorporating batch and layer normalizations, termed regularity batch normalization (RBN) and regularity layer normalization (RLN), as well as including the data prior as a top-down attention mechanism during the training process, termed saliency normalization (SN). In appendix A, we present the preliminary results on the imbalanced MNIST dataset and demonstrated that our approach is advantageous over existing normalization methods in different imbalanced scenarios. In the last section, we conclude our methods and point out several future work directions as the next step of this research. DISPLAYFORM0 Figure 1: Normalized maximal likelihood. Data sample xi are drawn from the data distribution X and model θi is the optimal model that describes data xi with the shortest code length. θj is an arbitrary model that is notθ3, so P (x3|θj) is not considered when computing optimal universal code according to NML formulation.Given a model class Θ consisting of a finite number of models parameterized by the parameter set θ. Given a data sample x, each model in the model class describes a probability P (x|θ) with the code length computed as − log P (x|θ). The minimum code length given any arbitrary θ would be given by L(x|θ(x)) = − log P (x|θ(x)) with model θ(x) which compresses data x most efficiently and offers the maximum likelihood P (x|θ(x)) BID4 . However, the compressibility of the model will be unattainable for multiple inputs, as the probability distributions are different. The solution relies on a universal code,P (x) defined for a model class Θ such that for any data sample x, the shortest code for x is always L(x|θ(x)) BID11 ). Inspired by the neural code adaptation of biological brains, we propose a biologically plausible normalization method taking into account the regularity (or saliency) of the activation distribution in the implicit space, and normalize it to upweight activation for rarely seen scenario and downweight activation for commonly seen ones. We introduce the concept from MDL principle and proposed to consider neural network training process as a model selection problem. We compute the optimal universal code length by normalized maximum likelihood in an incremental fashion, and showed this implementation can be easily incorporated with established methods like batch normalization and layer normalization. In addition, we proposed saliency normalization, which can introduce topdown attention and data prior to facilitate representation learning. Fundamentally, we implemented with an incremental update of normalized maximum likelihood, constraining the implicit space to have a low model complexity and short universal code length.Preliminary results offered a proof of concept to the proposed method. Given the limited experiments at the current state, our approach empirically outperforms existing normalization methods its advantage in the imbalanced or limited data scenario as hypothesized. Next steps of this research include experiments with variants of the regularity normalization (SN, RLN, RBN etc. ), as well as the inclusion of top-down attention given by data prior (such as feature extracted from signal processing, or task-dependent information). In concept, regularity-based normalization can also be considered as an unsupervised attention mechanism imposed on the input data. As the next step, we are currently exploring this method to convolutional and recurrent neural networks, and applying to popular state-of-the-art neural network architectures in multiple modalities of datasets, as well as the reinforcement learning setting where the rewards can be very sparse and non-uniform. Table 1 : Test errors of the imbalanced permutation-invariant MNIST 784-1000-1000-10 task "Balanced" "Rare minority" "Highly imbalanced" "Dominant oligarchy" n = 0 n = 1 n = 2 n = 3 n = 4 n = 5 n = 6 n = 7 n = 8 n = 9 baseline 4.80 ± 0.34 14.
Considering neural network optimization process as a model selection problem, we introduce a biological plausible normalization method that extracts statistical regularity under MDL principle to tackle imbalanced and limited data issue.
{"src": "scitldr-AIC_train", "paper_id": "HklJQ1JEDE"}