6477031
Universal approximation using incremental constructive feedforward networks with random hidden nodes
According to conventional neural network theories, single-hidden-layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes are universal approximators when all the parameters of the networks are allowed adjustable. However, as observed in most neural network implementations, tuning all the parameters of the networks may cause learning complicated and inefficient, and it may be difficult to train networks with nondifferential activation functions such as threshold networks. Unlike conventional neural network theories, this paper proves in an incremental constructive method that in order to let SLFNs work as universal approximators, one may simply randomly choose hidden nodes and then only need to adjust the output weights linking the hidden layer and the output layer. In such SLFNs implementations, the activation functions for additive nodes can be any bounded nonconstant piecewise continuous functions g : R --> R and the activation functions for RBF nodes can be any integrable piecewise continuous functions g : R --> R and integral of R g(x)dx not equal to 0. The proposed incremental method is efficient not only for SFLNs with continuous (including nondifferentiable) activation functions but also for SLFNs with piecewise continuous (such as threshold) activation functions. Compared to other popular methods such a new network is fully automatic and users need not intervene the learning process by manually tuning control parameters.
22510443
Universal Approximation and QoS Violation Application of Extreme Learning Machine
Neural networks have been successfully applied to many applications due to their approximation capability. However, complicated network structures and algorithms will lead to computational and time-consuming burdens. In order to satisfy demanding real-time requirements, many fast learning algorithms were explored in the past. Recently, a fast algorithm, Extreme Learning Machine (ELM) (Huang et al. 70:489–501, 2006) was proposed. Unlike conventional algorithms whose neurons need to be tuned, the input-to-hidden neurons of ELM are randomly generated. Though a large number of experimental results have shown that input-to-hidden neurons need not be tuned, there lacks a rigorous proof whether ELM possesses the universal approximation capability. In this paper, based on the universal approximation property of an orthonormal method, we firstly illustrate the equivalent relationship between ELM and the orthonormal method, and further prove that neural networks with ELM are also universal approximations. We also successfully apply ELM to the identification of QoS violation in the multimedia transmission.
31409147
Systemical convergence rate analysis of convex incremental feedforward neural networks
In this paper, we systemically investigate several convex incremental feedforward neural networks. Firstly, we prove the universal approximation and the convergence rate of a generalized convex incremental (GCI) structure, which provides us a wider parameter selection. Second, according to the convergence rate proof of GCI, we further prove the convergence rate of a best convex incremental (BCI) structure, moreover its proof also illustrates that BCI can achieve a better generalization performance than GCI. But we should note that the hidden neurons of BCI and GCI both are constructed on the maximum principle (not random). Next, we introduce the random neuron conception based on CI-ELM (convex incremental extreme learning machines), and further propose an alternative algorithm (improved CI-ELM, ICI-ELM) between CI-ELM and BCI, which removes the ''useless'' neurons in CI-ELM and improves the efficiency of neural networks. ICI-ELM randomly generates a group of parameters, among which we determine the best parameters leading to the smallest residual error. Therefore ICI-ELM can achieve a faster convergence rate than CI-ELM, meanwhile it still retains the same convergence rate as BCI. On the other hand, ICI-ELM also provides an alternative scheme to replace conventional gradient methods, which are only suitable for differential functions and often achieves local minima. The experimental results based on several benchmark regression problems support our claims.
2677048
Convergence analysis of convex incremental neural networks
Recently, a convex incremental algorithm (CI-ELM) has been proposed in Huang and Chen (Neurocomputing 70:3056–3062, 2007), which randomly chooses hidden neurons and then analytically determines the output weights connecting with the hidden layer and the output layer. Though hidden neurons are generated randomly, the network constructed by CI-ELM is still based on the principle of universal approximation. The random approximation theory breaks through the limitation of most conventional theories, eliminating the need for tuning hidden neurons. However, due to the random characteristic, some of the neurons contribute little to decrease the residual error, which eventually increase the complexity and computation of neural networks. Thus, CI-ELM cannot precisely give out its convergence rate. Based on Lee’s results (Lee et al., IEEE Trans Inf Theory 42(6):2118–2132, 1996), we first show the convergence rate of a maximum CI-ELM, and then systematically analyze the convergence rate of an enhanced CI-ELM. Different from CI-ELM, the hidden neurons of the two algorithms are chosen by following the maximum or optimality principle under the same structure as CI-ELM. Further, the proof process also demonstrates that our algorithms achieve smaller residual errors than CI-ELM. Since the proposed neural networks remove these “useless” neurons, they improve the efficiency of neural networks. The experimental results on benchmark regression problems will support our conclusions.
2891159
Convex incremental extreme learning machine
Unlike the conventional neural network theories and implementations, Huang et al. [Universal approximation using incremental constructive feedforward networks with random hidden nodes, IEEE Transactions on Neural Networks 17(4) (2006) 879-892] have recently proposed a new theory to show that single-hidden-layer feedforward networks (SLFNs) with randomly generated additive or radial basis function (RBF) hidden nodes (according to any continuous sampling distribution) can work as universal approximators and the resulting incremental extreme learning machine (I-ELM) outperforms many popular learning algorithms. I-ELM randomly generates the hidden nodes and analytically calculates the output weights of SLFNs, however, I-ELM does not recalculate the output weights of all the existing nodes when a new node is added. This paper shows that while retaining the same simplicity, the convergence rate of I-ELM can be further improved by recalculating the output weights of the existing nodes based on a convex optimization method when a new hidden node is randomly added. Furthermore, we show that given a type of piecewise continuous computational hidden nodes (possibly not neural alike nodes), if SLFNs f"n(x)=@?i=1n@b"iG(x,a"i,b"i) can work as universal approximators with adjustable hidden node parameters, from a function approximation point of view the hidden node parameters of such ''generalized'' SLFNs (including sigmoid networks, RBF networks, trigonometric networks, threshold networks, fuzzy inference systems, fully complex neural networks, high-order networks, ridge polynomial networks, wavelet networks, etc.) can actually be randomly generated according to any continuous sampling distribution. In theory, the parameters of these SLFNs can be analytically determined by ELM instead of being tuned.
14619623
Incremental extreme learning machine with fully complex hidden nodes
Huang et al. [Universal approximation using incremental constructive feedforward networks with random hidden nodes, IEEE Trans. Neural Networks 17(4) (2006) 879-892] has recently proposed an incremental extreme learning machine (I-ELM), which randomly adds hidden nodes incrementally and analytically determines the output weights. Although hidden nodes are generated randomly, the network constructed by I-ELM remains as a universal approximator. This paper extends I-ELM from the real domain to the complex domain. We show that, as long as the hidden layer activation function is complex continuous discriminatory or complex bounded nonlinear piecewise continuous, I-ELM can still approximate any target functions in the complex domain. The universal capability of the I-ELM in the complex domain is further verified by two function approximations and one channel equalization problems.
32539682
A combination of linear and nonlinear activation functions in neural networks for modeling a de-superheater
Abstract This paper deals with modeling a power plant component with mild nonlinear characteristics using a modified neural network structure. The hidden layer of the proposed neural network has a combination of neurons with linear and nonlinear activation functions. This approach is particularly suitable for nonlinear system with a low grade of nonlinearity, which can not be modeled satisfactorily by neural networks with purely nonlinear hidden layers or by the method of least square of errors (the ideal modeling method of linear systems). In this approach, two channels are installed in a hidden layer of the neural network to cover both linear and nonlinear behavior of systems. If the nonlinear characteristics of the system (i.e. de-superheater) are not negligible, then the nonlinear channel of the neural network is activated; that is, after training, the connections in nonlinear channel get considerable weights. The approach was applied to a de-superheater of a 325 MW power generating plant. The actual plant response, obtained from field experiments, is compared with the response of the proposed model and the responses of linear and neuro-fuzzy models as well as a neural network with purely nonlinear hidden layer. A better accuracy is observed using the proposed approach.
190958
Reinforcement learning and aggregation
Reinforcement learning (RL) is a learning technique that provides a means for learning an optimal control policy when the dynamics of the environment under consideration is unavailable [L.P. Kaelbling et al., 1996, R.S. Sutton and A.G. Barto, 1998]. While RL has been successfully applied in many single or multiple agents systems [S. Arai et al., 2000, H.R. Berenji and D.A. Vengerov, 2000, M. Tan, 1993, Y. Nagayuki et al., 2000], the learning quality is greatly influenced by learning algorithms and their parameters. Setting of the parameters of RL algorithms is something of a black art, and small differences in these parameters can lead to large differences in learning qualities. Determining the best algorithm and the optimal parameters can be costly in terms of time and computation. Even if the cost is acceptable, the robustness of learning is still a question. In order to address the difficulty, an aggregated multiagent reinforcement learning system (AMRLS) is proposed to deal with the RL environment as a multiagent environment. A maze world environment is used to validate the AMRLS. Experimental results illustrate that compared with normal Q(/spl lambda/)-learning and SARSA(/spl lambda/) algorithms, the AMRLS increases both the learning speed and the rate of reaching the shortest path.
39616456
Advances in Web and Network Technologies, and Information Management, APWeb/WAIM 2009 International Workshops: WCMT 2009, RTBI 2009, DBIR-ENQOIR 2009, PAIS 2009, Suzhou, China, April 2-4, 2009, Revised Selected Papers
16858369
Efficient Processing of Metric Skyline Queries
Skyline query is of great importance in many applications, such as multi-criteria decision making and business planning. In particular, a skyline point is a data object in the database whose attribute vector is not dominated by that of any other objects. Previous methods to retrieve skyline points usually assume static data objects in the database (i.e. their attribute vectors are fixed), whereas several recent work focus on skyline queries with dynamic attributes. In this paper, we propose a novel variant of skyline queries, namely metric skyline, whose dynamic attributes are defined in the metric space (i.e. not limited to the Euclidean space). We illustrate an efficient and effective pruning mechanism to answer metric skyline queries through a metric index. Most importantly, we formalize the query performance of the metric skyline query in terms of the pruning power, by a cost model, in light of which we construct an optimized metric index aiming to maximize the pruning power of metric skyline queries. Extensive experiments have demonstrated the efficiency and effectiveness of our proposed pruning techniques as well as the constructed index in answering metric skyline queries.
728817
Study of Coupling Based on Genetic Neural Network Multivariable Nonlinear Complex System
In this paper a method of decoupling has been put forward. Combine genetic neural network with multivariable nonlinear system adaptive decoupling control algorithm. Establish a control model of multivariable parameter in order to cancel the couplings of various variables. This method improves the stability and precision. According to the METLAB emulator proving, genetic neural network multivariable nonlinear decoupling model can cancel multivariable coupling problem preferably, make the system optimized.