Many approaches to transform classification problems from non-linear to linear by feature transformation have been recently presented in the literature. These notably include sparse coding methods and deep neural networks. However, many of these approaches require the repeated application of a learning process upon the presentation of unseen data input vectors, or else involve the use of large numbers of parameters and hyper-parameters, which must be chosen through cross-validation, thus increasing running time dramatically. In this paper, we propose and experimentally investigate a new approach for the purpose of overcoming limitations of both kinds. The proposed approach makes use of a linear auto-associative network (called SCNN) with just one hidden layer. The combination of this architecture with a specific error function to be minimized enables one to learn a linear encoder computing a sparse code which turns out to be as similar as possible to the sparse coding that one obtains by re-training the neural network. Importantly, the linearity of SCNN and the choice of the error function allow one to achieve reduced running time in the learning phase. The proposed architecture is evaluated on the basis of two standard machine learning tasks. Its performances are compared with those of recently proposed non-linear auto-associative neural networks. The overall results suggest that linear encoders can be profitably used to obtain sparse data representations in the context of machine learning problems, provided that an appropriate error function is used during the learning phase.
Both networks, ASCNN and SAANN, are trained by means of the mini batch stochastic gradient descent learning algorithm because the error function is differentiable as the penalization term is differentiable. The possibility of identifying valid alternatives to non-linear approaches to sparse coding, in the context of non-linear autoassociative networks with a single hidden layer, is suggested by some classification which were successfully addressed on the basis of linear network approaches. Notably, linear approaches were successfully used to model the early stage responses of the visual system. In the early stage visual information is a small number of simultaneously active neurons among the much larger number of available neurons. The first attempt to model this behaviour of the visual system is due to Olshausen and Field ( @cite_17 ). The authors built a simple single weight layer feed forward neural network (Sparsenet) where the observed data @math are a linear combination of top-bottom basis vectors @math and top-layer sparse responses @math . The sparsity of the solution is obtained by minimizing an error function composed of the standard sum-of-squares error and a regularization term, which can be expressed as follows:
@cite_17
2145889472
THE receptive fields of simple cells in mammalian primary visual cortex can be characterized as being spatially localized, oriented1–4 and bandpass (selective to structure at different spatial scales), comparable to the basis functions of wavelet transforms5,6. One approach to understanding such response properties of visual neurons has been to consider their relationship to the statistical structure of natural images in terms of efficient coding7–12. Along these lines, a number of studies have attempted to train unsupervised learning algorithms on natural images in the hope of developing receptive fields with similar properties13–18, but none has succeeded in producing a full set that spans the image space and contains all three of the above properties. Here we investigate the proposal8,12 that a coding strategy that maximizes sparseness is sufficient to account for these properties. We show that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex. The resulting sparse image code provides a more efficient representation for later stages of processing because it possesses a higher degree of statistical independence among its outputs.