Restricted Boltzmann Machine (RBM) is an importan- t generative model modeling vectorial data. While applying an RBM in practice to images, the data have to be vec- torized. This results in high-dimensional data and valu- able spatial information has got lost in vectorization. In this paper, a Matrix-Variate Restricted Boltzmann Machine (MVRBM) model is proposed by generalizing the classic RBM to explicitly model matrix data. In the new RBM model, both input and hidden variables are in matrix forms which are connected by bilinear transforms. The MVRBM has much less model parameters, resulting in a faster train- ing algorithm while retaining comparable performance as the classic RBM. The advantages of the MVRBM have been demonstrated on two real-world applications: Image super- resolution and handwritten digit recognition.
@cite_28 propose a tensor-product model based representation of neural networks in which a neural network structure for mutual interaction of variable components is introduced. It conceptually restructures the tensor product model transformation to a generalized form for fuzzy modeling and to propose new features and several new variants of the tensor product model transformation. However this type of neural networks is actually defined for vectorial data rather than tensorial variates. The similar idea can be seen in the most recent paper @cite_2 . The key characterization for these networks is the connection weights (neural networks parameters) are in tensor format, rather than the data variables in the networks. Thus except for the nonlinearity introduced by the activation function, the neural networks offer the capacity of encoding nonlinear interaction among the hidden variable components. The so-called tensor analyzer @cite_12 also serves as such an example.
@cite_28 @cite_12 @cite_2
2010639055 2108597378
The approximation methods of mathematics are widely used in theory and practice for several problems. In the framework of the paper a novel tensor-product based approach for representation of neural networks (NNs) is proposed. The NNs in this case stand for local models based on which a more complex parameter varying model can numerically be reconstructed and reduced using the higher order singular value decomposition (HOSVD). The HOSVD as well as the tensor-product based representation of NNs will be discussed in detail. A novel deep architecture, the tensor deep stacking network (T-DSN), is presented. The T-DSN consists of multiple, stacked blocks, where each block contains a bilinear mapping from two hidden layers to the output layer, using a weight tensor to incorporate higher order statistics of the hidden binary (([0,1])) features. A learning algorithm for the T-DSN's weight matrices and tensors is developed and described in which the main parameter estimation burden is shifted to a convex subproblem with a closed-form solution. Using an efficient and scalable parallel implementation for CPU clusters, we train sets of T-DSNs in three popular tasks in increasing order of the data size: handwritten digit recognition using MNIST (60k), isolated state phone classification and continuous phone recognition using TIMIT (1.1 m), and isolated phone classification using WSJ0 (5.2 m). Experimental results in all three tasks demonstrate the effectiveness of the T-DSN and the associated learning methods in a consistent manner. In particular, a sufficient depth of the T-DSN, a symmetry in the two hidden layers structure in each T-DSN block, our model parameter learning algorithm, and a softmax layer on top of T-DSN are shown to have all contributed to the low error rates observed in the experiments for all three tasks.