Whereas CNNs have demonstrated immense progress in many vision problems, they suffer from a dependence on monumental amounts of labeled training data. On the other hand, dictionary learning does not scale to the size of problems that CNNs can handle, despite being very effective at low-level vision tasks such as denoising and inpainting. Recently, interest has grown in adapting dictionary learning methods for supervised tasks such as classification and inverse problems. We propose two new network layers that are based on dictionary learning: a sparse factorization layer and a convolutional sparse factorization layer, analogous to fully-connected and convolutional layers, respectively. Using our derivations, these layers can be dropped in to existing CNNs, trained together in an end-to-end fashion with back-propagation, and leverage semisupervision in ways classical CNNs cannot. We experimentally compare networks with these two new layers against a baseline CNN. Our results demonstrate that networks with either of the sparse factorization layers are able to outperform classical CNNs when supervised data are few. They also show performance improvements in certain tasks when compared to the CNN with no sparse factorization layers with the same exact number of parameters.
We first focus our discussion on methods seeking to bridge between artificial neural networks and sparse representation modeling. Greedy deep dictionary learning @cite_35 involves a layered, generative architecture that seeks to encode inputs through a series of sparse factorizations, which are trained individually to reconstruct the input faithfully. It, however, has no mechanism for end-to-end training and is unsupervised, limiting its potential in various tasks like classification, given modern performance statistics. Sparse autoencoders @cite_7 @cite_11 do the same using linear operations and sparsifying transforms in place of sparse factorization. Although these models can be trained in an end-to-end manner, we are not aware of any work that incorporates them into supervised or semisupervised tasks. In contrast, the SF layer that we propose subsumes these capabilities, can be trained end-to-end and can be embedded in supervised, semisupervised and unsupervised networks.
@cite_35 @cite_7 @cite_11
2254140767 44815768
In this work we propose a new deep learning tool called deep dictionary learning. Multi-level dictionaries are learnt in a greedy fashion, one layer at a time. This requires solving a simple (shallow) dictionary learning problem, the solution to this is well known. We apply the proposed technique on some benchmark deep learning datasets. We compare our results with other deep learning tools like stacked autoencoder and deep belief network; and state of the art supervised dictionary learning tools like discriminative KSVD and label consistent KSVD. Our method yields better results than all. Restricted Boltzmann machines (RBMs) have been used as generative models of many different types of data. RBMs are usually trained using the contrastive divergence learning procedure. This requires a certain amount of practical experience to decide how to set the values of numerical meta-parameters. Over the last few years, the machine learning group at the University of Toronto has acquired considerable expertise at training RBMs and this guide is an attempt to share this expertise with other machine learning researchers.