Gradient calculation for dynamic recurrent neural networks: a survey <s> A. Why Recurrent Networks <s> Perhaps one of the most striking differences between a brain and today’s computers is the amount of “wiring.” In a digital computer the ratio of connections to components is about 3, whereas for the mammalian cortex it lies between 10 and 10,000 (1). <s> BIB001 </s> Gradient calculation for dynamic recurrent neural networks: a survey <s> A. Why Recurrent Networks <s> This chapter contains sections titled: Parallel Distributed Processing, Examples Of PDP Models, Representation and Learning in PDP Models, Origins of Parallel Distributed Processing, Acknowledgments <s> BIB002 </s> Gradient calculation for dynamic recurrent neural networks: a survey <s> A. Why Recurrent Networks <s> The ability of learning networks to generalize can be greatly enhanced by providing constraints from the task domain. This paper demonstrates how such constraints can be integrated into a backpropagation network through the architecture of the network. This approach has been successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service. A single network learns the entire recognition operation, going from the normalized image of the character to the final classification. <s> BIB003 </s> Gradient calculation for dynamic recurrent neural networks: a survey <s> A. Why Recurrent Networks <s> Marr and Poggio (1976) recently described a cooperative algorithm that solves the correspondence problem for stereopsis. This article uses a probabilistic technique to analyze the convergence of that algorithm, and derives the conditions governing the stability of the solution state. The actual results of applying the algorithm to random-dot stereograms are compared with the probabilistic analysis. A satisfactory mathematical analysis of the asymptotic behaviour of the algorithm is possible for a suitable choice of the parameter values and loading rules, and again the actual performance of the algorithm under these conditions is compared with the theoretical predictions. Finally, some problems raised by the analysis of this type of "cooperative" algorithm are briefly discussed. <s> BIB004
The motivation for exploring recurrent architectures is their potential for dealing with two sorts of temporal behavior. First, recurrent networks are capable of settling to a solution that satis es many constraints 1], as in a vision system which relaxes to an interpretation of an image which maximally satis es a complex set of con icting constraints 2], 3], 4], 5], 6], a system which relaxes to nd a posture for a robot satisfying many criteria 7], and models of language parsing 8]. Although algorithms suitable for building systems of this type are reviewed to some extent below, such as the algorithm used in 9], the bulk of this paper is concerned with the problem of causing networks to exhibit particular desired detailed temporal behavior, which has found application in signal processing 10 BIB003 . For this reason, if one is interested in solving a particular problem, it would be only prudent t o t r y a v ariety of non-recurrent a r c hitectures before resorting to the more powerful and general recurrent n e t works. This paper is concerned with learning algorithms for recurrent n e t works themselves, and not with recurrent networks as elements of larger systems, such as specialized architectures for control  We will restrict our attention to training procedures for networks which m a y include hidden units, units which h a ve no particular desired behavior and are not directly involved in the input or output of the network. For the biologically inclined, they can be thought o f a s i n terneurons. With the practical successes of backpropagation, it seems gratuitous to expound the virtues of hidden units and internal representations. Hidden units make it possible for networks to discover and exploit regularities of the task at hand, such as symmetries or replicated structure   , can be used to train recurrent n e t works without hidden units, so recurrent networks without hidden units reduce to nonrecurrent networks without hidden units, and therefore do not need special learning algorithms. Consider a neural network governed by the equations dy dt = f(y(t) w I (t)) where y is the time-varying state vector, w the parameters to be modi ed by the learning, and I a time-varying vector of external input. Given some error metric E 0 (y t), our task is to modify w to reduce E = R E 0 (y t)dt. Our strategy will be gradient descent, so the main portion of our work will be nding algorithms to calculate the gradient r w E, the vector whose elements are @E=@w i . The above formulation is for a continuous time system. The alternative t o t h i s i s a c l o c ked system, which obeys an equation of the form y(t + t) = f(y(t) w I (t)): Without loss of generality, for clocked systems we will use t = 1 , giving y(t + 1 ) = f(y(t) w I (t)) (2) with t an integer. Certainly, barring high-frequency components in I, t h e behavior of (1) can be precisely duplicated by BIB001 with suitable choice of f in the latter. For this reason, in order to determine the practical tradeo s of one against the other, we m ust consider particular functional forms for f:We will consider the most common neural network formulation, dy i dt = ;y i + (x i ) + I i BIB004 where y i is the state or activation level of unit i, is the total input to unit i, w ij is the strength of the connection from unit i to unit j, and is a di erentiable function. BIB002 The initial conditions y i (t 0 ) and driving functions I i (t) are the inputs to the system. This de nes a rather general dynamic system. Even assuming that the external input terms I i (t) are held constant, it is possible for the system to exhibit a wide range of asymptotic behaviors. The simplest is that the system reaches a stable xedpoint in the next section, we will discuss two di erent t e c hniques for modifying the xedpoints of networks that exhibit them. More complicated possible asymptotic behaviors include limit cycles and even chaos. Later, we will describe a number of gradient based training procedures that can be applied to training networks to exhibit desired limit cycles, or particular detailed temporal behavior. We will not discuss specialized non-gradient methods for learning limit Later, we will describe a number of training procedures that, for a price in space or time, do not rely on such r estrictions and can be applied to training networks to exhibit desired limit cycles, or particular detailed temporal behavior. C. Continuous vs. Discrete Time We will be concerned predominantly with continuous time networks, as in BIB004 . However, all of the learning procedures we will discuss can be equally well applied to discrete time systems, which obey equations like BIB001 . Continuous time has advantages for expository purposes, in that the derivative of the state of a unit with respect to time is well de ned, allowing calculus to be used instead of tedious explicit temporal indexing, making for simpler derivations and exposition. When a continuous time system is simulated on a digital computer, it is usually converted into a set of simple rst order di erence equations, which is formally identical to a discrete time network. However, regarding the discrete time network running on the computer as a simulation of a continuous time network has a number of advantages. First, more sophisticated and faster simulation techniques than simple rst order di erence equations can be used 82]. Second, even if simple rst order equations are used, the size of the time step can be varied to suit changing circumstances for instance, if the network is being used for a signal processing application and faster sensors and computers become available, the size of the time step could be decreased without retraining the network. Third, because continuous time units are sti in time, they tend to retain information better through time. Another way o f putting this is that their bias in the learning theory sense is towards temporally continuous tasks, which is certainly advantageous if the task being performed is in fact temporally continuous. Another advantage of continuous time networks is somewhat more subtle. Even for tasks which themselves have n o temporal content, such as constraint satisfaction, the natural way for a recurrent network to perform the required computation is for each unit to represent nearly the same thing at nearby points in time. Using continuous time units makes this the default behavior in the absence other forces, units will tend to retain their state through time. In contrast, in discrete time networks, there is no a priori reason for a unit's state at one point in time to have a n y special relationship to its state at the next point in time. A pleasant added bene t of units tending to maintain their states through time is that it helps make information about the past decay more slowly, speeding up learning about the relationship between temporally distant e v ents.