Learning long-term dependencies still remains difficult for recurrent neural networks (RNNs) despite their success in sequence modeling recently. In this paper, we propose a novel gated RNN structure, which contains only one gate. Hidden states in the proposed grouped distributor unit (GDU) are partitioned into groups. For each group, the proportion of memory to be overwritten in each state transition is limited to a constant and is adaptively distributed to each group member. In other word, every separate group has a fixed overall update rate, yet all units are allowed to have different paces. Information is therefore forced to be latched in a flexible way, which helps the model to capture long-term dependencies in data. Besides having a simpler structure, GDU is demonstrated experimentally to outperform LSTM and GRU on tasks including both pathological problems and natural data set.
An RNN is able to encode sequences of arbitrary length into a fixed-length representation by folding a new observation @math into its hidden state @math using a transition operator @math at each time step @math : Simple recurrent networks (SRN, @cite_22 ), known as one of the earliest variants, make @math as the composition of an element-wise nonlinearity with an affine transformation of both @math and @math : W _s x _t + U _s s _ t-1 + b _s where @math is the input-to-state weight matrix, @math is the state-to-state recurrent weight matrix, @math is the bias and @math is the nonlinear activation function. For the convenience of the following descriptions, we denote this kind of operators as @math , and a subscript can be added to distinguish different network components. Thus in SRN, @math .
@cite_22
2110485445
Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory. In this approach, hidden unit patterns are fed back to themselves; the internal representations which develop thus reflect task demands in the context of prior internal states. A set of simulations is reported which range from relatively simple problems (temporal version of XOR) to discovering syntactic semantic features for words. The networks are able to learn interesting internal representations which incorporate task demands with memory demands; indeed, in this approach the notion of memory is inextricably bound up with task processing. These representations reveal a rich structure, which allows them to be highly context-dependent while also expressing generalizations across classes of items. These representations suggest a method for representing lexical categories and the type token distinction.