Representation Learning: A Review and New Perspectives <s> Patch-Based Training <s> Previous work on action recognition has focused on adapting hand-designed local features, such as SIFT or HOG, from static images to the video domain. In this paper, we propose using unsupervised feature learning as a way to learn features directly from video data. More specifically, we present an extension of the Independent Subspace Analysis algorithm to learn invariant spatio-temporal features from unlabeled video data. We discovered that, despite its simplicity, this method performs surprisingly well when combined with deep learning techniques such as stacking and convolution to learn hierarchical representations. By replacing hand-designed features with our learned features, we achieve classification results superior to all previous published results on the Hollywood2, UCF, KTH and YouTube action recognition datasets. On the challenging Hollywood2 and YouTube action datasets we obtain 53.3% and 75.8% respectively, which are approximately 5% better than the current best published results. Further benefits of this method, such as the ease of training and the efficiency of training and prediction, will also be discussed. You can download our code and learned spatio-temporal features here: http://ai.stanford.edu/∼wzou/ <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Patch-Based Training <s> While vector quantization (VQ) has been applied widely to generate features for visual recognition problems, much recent work has focused on more powerful methods. In particular, sparse coding has emerged as a strong alternative to traditional VQ approaches and has been shown to achieve consistently higher performance on benchmark datasets. Both approaches can be split into a training phase, where the system learns a dictionary of basis functions, and an encoding phase, where the dictionary is used to extract features from new inputs. In this work, we investigate the reasons for the success of sparse coding over VQ by decoupling these phases, allowing us to separate out the contributions of training and encoding in a controlled way. Through extensive experiments on CIFAR, NORB and Caltech 101 datasets, we compare several training and encoding schemes, including sparse coding and a form of VQ with a soft threshold activation function. Our results show not only that we can use fast VQ algorithms for training, but that we can just as well use randomly chosen exemplars from the training set. Rather than spend resources on training, we find it is more important to choose a good encoder—which can often be a simple feed forward non-linearity. Our results include state-of-the-art performance on both CIFAR and NORB. <s> BIB002
The simplest approach for learning a convolutional layer in an unsupervised fashion is patch-based training: simply feeding a generic unsupervised feature learning algorithm with local patches extracted at random positions of the inputs. The resulting feature extractor can then be swiped over the input to produce the convolutional feature maps. That map may be used as a new input for the next layer, and the operation repeated to thus learn and stack several layers. Such an approach was recently used with independent subspace analysis BIB001 on 3D video blocks, reaching the state of the art on the Hollywood2, UCF, KTH, and YouTube action recognition datasets. Similarly, Coates and Ng BIB002 compared several feature learners with patch-based training and reached state-of-the-art results on several classification benchmarks. Interestingly, in this work, performance was almost as good with very simple k-means clustering as with more sophisticated feature learners. We, however, conjecture that this is the case only because patches are rather low-dimensional (compared to the dimension of a whole image). A large dataset might provide sufficient coverage of the space of, for example, edges prevalent in 6 Â 6 patches so that a distributed representation is not absolutely necessary. Another plausible explanation for this success is that the clusters identified in each image patch are then pooled into a histogram of cluster counts associated with a larger subimage. Whereas the output of a regular clustering is a one-hot nondistributed code, this histogram is itself a distributed representation, and the "soft" k-means BIB002 representation allows not only the nearest filter, but also its neighbors to be active.