Sequence prediction and classification are ubiquitous and challenging problems in machine learning that can require identifying complex dependencies between temporally distant inputs. Recurrent Neural Networks (RNNs) have the ability, in theory, to cope with these temporal dependencies by virtue of the short-term memory implemented by their recurrent (feedback) connections. However, in practice they are difficult to train successfully when the long-term memory is required. This paper introduces a simple, yet powerful modification to the standard RNN architecture, the Clockwork RNN (CW-RNN), in which the hidden layer is partitioned into separate modules, each processing inputs at its own temporal granularity, making computations only at its prescribed clock rate. Rather than making the standard RNN models more complex, CW-RNN reduces the number of RNN parameters, improves the performance significantly in the tasks tested, and speeds up the network evaluation. The network is demonstrated in preliminary experiments involving two tasks: audio signal generation and TIMIT spoken word classification, where it outperforms both RNN and LSTM networks.
Temporal Transition Hierarchy (TTH; Ring:93 ) incrementally adds high-order neurons in order to build a memory that is used to disambiguate an input at the current time step. This approach can, in principle, bridge time intervals of any length, but with proportionally growing network size. The model was recently improved by adding recurrent connections @cite_16 that prevent it from bloating by reusing the high-level nodes through the recurrent connections.
@cite_16
2183208319
Continual learning is the unending process of learning new things on top of what has already been learned (Ring 1994). Temporal Transition Hierarchies (TTHs) were developed to allow prediction of Markov-k sequences in a way that was consistent with the needs of a continual-learning agent (Ring 1993). However, the algorithm could not learn arbitrary temporal contingencies. This paper describes Recurrent Transition Hierarchies (RTH), a learning method that combines several properties desirable for agents that must learn as they go. In particular, it learns online and incrementally, autonomously discovering new features as learning progresses. It requires no reset or episodes. It has a simple learning rule with update complexity linear in the number of parameters.