We present a greedy-based approach to construct an efficient single hidden layer neural network with the ReLU activation that approximates a target function. In our approach we obtain a shallow network by utilizing a greedy algorithm with the prescribed dictionary provided by the available training data and a set of possible inner weights. To facilitate the greedy selection process we employ an integral representation of the network, based on the ridgelet transform, that significantly reduces the cardinality of the dictionary and hence promotes feasibility of the greedy selection. Our approach allows for the construction of efficient architectures which can be treated either as improved initializations to be used in place of random-based alternatives, or as fully-trained networks in certain cases, thus potentially nullifying the need for backpropagation training. Numerical experiments demonstrate the tenability of the proposed concept and its advantages compared to the conventional techniques for selecting architectures and initializations for neural networks.
Another related approach to ours is the Random Vector Functional-Link (or RVFL) network @cite_2 where the inner weights @math are generated randomly and the outer weights @math are found via a least square regularization. The random selection can be interpreted as a Monte-Carlo sampling for an integral representation. However the authors do not use any integral representation explicitly, only the fact of existence of such representation to justify their approach.
@cite_2
1987618706
Abstract We present a strategy for learning optimal control. The approach uses functional-link neural network implementations which have several beneficial properties giving advantages over the more common generalized delta rule implementations. The learning task is decomposed into three parts: identification and monitoring, one-step-ahead control generation, and control path optimization. Each of these parts is accomplished with its own functional-link net and these are coordinated to provide the real-time learning of the optimal control path.