A Survey on the New Generation of Deep Learning in Image Processing <s> 2) H-ELM <s> Extreme learning machine (ELM) is an emerging learning algorithm for the generalized single hidden layer feedforward neural networks, of which the hidden node parameters are randomly generated and the output weights are analytically computed. However, due to its shallow architecture, feature learning using ELM may not be effective for natural signals (e.g., images/videos), even with a large number of hidden nodes. To address this issue, in this paper, a new ELM-based hierarchical learning framework is proposed for multilayer perceptron. The proposed architecture is divided into two main components: 1) self-taught feature extraction followed by supervised feature classification and 2) they are bridged by random initialized hidden weights. The novelties of this paper are as follows: 1) unsupervised multilayer encoding is conducted for feature extraction, and an ELM-based sparse autoencoder is developed via l 1 constraint. By doing so, it achieves more compact and meaningful feature representations than the original ELM; 2) by exploiting the advantages of ELM random feature mapping, the hierarchically encoded outputs are randomly projected before final decision making, which leads to a better generalization with faster learning speed; and 3) unlike the greedy layerwise training of deep learning (DL), the hidden layers of the proposed framework are trained in a forward manner. Once the previous layer is established, the weights of the current layer are fixed without fine-tuning. Therefore, it has much better learning efficiency than the DL. Extensive experiments on various widely used classification data sets show that the proposed algorithm achieves better and faster convergence than the existing state-of-the-art hierarchical learning methods. Furthermore, multiple applications in computer vision further confirm the generality and capability of the proposed learning scheme. <s> BIB001 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> 2) H-ELM <s> Single-hidden layer feedforward neural networks with randomly fixed hidden neurons (RHN-SLFNs) have been shown, both theoretically and experimentally, to be fast and accurate. Besides, it is well known that deep architectures can find higher-level representations, thus can potentially capture relevant higher-level abstractions. But most of current deep learning methods require a long time to solve a non-convex optimization problem. In this paper, we propose a stacked deep neural network, St-URHN-SLFNs, via unsupervised RHN-SLFNs according to stacked generalization philosophy to deal with unsupervised problems. Empirical study on a wide range of data sets demonstrates that the proposed algorithm outperforms the state-of-the-art unsupervised algorithms in terms of accuracy. On the computational effectiveness, the proposed algorithm runs much faster than other deep learning methods, i.e. deep autoencoder (DA) and stacked autoencoder (SAE), and little slower than other methods. <s> BIB002
For pattern recognition tasks, feature learning is often required before classification conducted in many applications. However, feature learning using ELM may not be effective for natural signals. To address this issue, inspired by the multilayer perceptron theories and deep stacked autoencoder networks, H-ELM is proposed BIB001 , BIB002 . The H-ELM learning framework consists of two main components, one is ELM-based sparse auto-encoder for unsupervised multilayer feature encoding, and the other is that the original ELM applied for final decision making. First, for ELM sparse auto-encoder (see in Fig.15 ), in order to generate more sparse and compact features of the inputs, we have the following equation, where W and b are randomly choose the input weight and the complex bias, and X = [X , 1], and · F is the Frobenius norm, · 1 is the sum of absolute values of of a matrix's all components or elements. If θ [W ; b], then we have H = σ ( X θ). Once β can be obtained, we can replace random parameter θ with β T . Further, we have the hidden mapping matrix with non-randomness, that is, where H β can be viewed as an non-randomness alternative to H . Its main advantage lies in an effective feature learning for x. It should be noted thatx is a approximation for x in Fig.15 . Second, H-ELM can be constructed by using the following hierarchical stack way in Fig.16 . where L is the number of the hidden layer for multi-layer perceptron. Finally, for the decision making, we have obtained the parameter β by original ELM, that is, where Y is the target label correspond to input training samples X . And H † is the Moore-Penrose generalized pseudoinverse of the matrix H . Extensive experiments show that the ELM sparse auto-encoder of H-ELM helps to generate a more excellent performance by providing more robust features. 