Orthogonal recurrent neural networks address the vanishing gradient problem by parameterizing the recurrent connections using an orthogonal matrix. This class of models is particularly effective to solve tasks that require the memorization of long sequences. We propose an alternative solution based on explicit memorization using linear autoencoders for sequences. We show how a recently proposed recurrent architecture, the Linear Memory Network, composed of a nonlinear feedforward layer and a separate linear recurrence, can be used to solve hard memorization tasks. We propose an initialization schema that sets the weights of a recurrent architecture to approximate a linear autoencoder of the input sequences, which can be found with a closed-form solution. The initialization schema can be easily adapted to any recurrent architecture.
     We argue that this approach is superior to a random orthogonal initialization due to the autoencoder, which allows the memorization of long sequences even before training. The empirical analysis show that our approach achieves competitive results against alternative orthogonal models, and the LSTM, on sequential MNIST, permuted MNIST and TIMIT. Several sequential problems require the memorization of long sequences of patterns. As an example, a generative model for music should be able to memorize long sequences of notes and be able to repeat them, as it is typically done in musical pieces. RNNs and LSTMs struggle to solve even simple memorization tasks (Arjovsky et al., 2015; Graves et al., 2014) . Therefore, it is important to study alternative solutions to this problem. Orthogonal RNNs are a class of recurrent architectures that solve the vanishing gradient problem by constraining the recurrent connections to be an orthogonal or unitary matrix (Arjovsky et al., 2015) . They are particularly effective to solve long memorization tasks Henaff et al. (2016) . In this paper, we address the problem of memorization with orthogonal RNNs and linear autoencoders. Our objective is to find a solution to the problem of memorization of long sequences. The memorization of long sequences with orthogonal models can require a large number of hidden units, increasing the hidden state size and the cost in time and space. If we assume that the input sequences in the training data lay in a low-dimensional manifold, as is typically believed for real-world data, then we can train an autoencoder with a small number of hidden units, sufficient to encode the entire sequence. If we restrict ourselves to the linear case, we can compute the optimal autoencoder with a closedform solution (Sperduti, 2013) . This result can be exploited to initialize recurrent architectures to approximate the linear autoencoder for the input sequences. In our experiments we use the RNN (Elman, 1990 ) and the Linear Memory Network (LMN) (Bacciu et al., 2019) . The LMN with the autoencoder initialization is a recurrent architecture equivalent to the Elman RNN but able to solve the vanishing gradient problem and memorize input sequences with a minimal hidden state. We test our approach on classic benchmarks for orthogonal RNNs, showing that our proposed approach behaves similarly to orthogonal architectures on pure memorization tasks, and even improving the performance on real-world datasets. Finally, we show that the model can also be used in situations where a strict orthogonal parameterization struggles (Vorontsov et al., 2017) , like the TIMIT benchmark (Garofolo et al., 1993) . Work on orthogonal models typically focuses on the properties of an orthogonal parameterization at the backward step, to address the vanishing gradient. In our work instead, we focus on the forward step, by investigating the effect of an orthogonal parameterization against an autoencoder-based solution. For these reasons, we are not particularly interested in enforcing exact orthogonality constraints, but in the study of the effectiveness of an autoencoder-based memorization mechanism. Our proposed approach requires the memorization of the entire sequence within the hidden state activations. It is necessary to note that this approach can be quite inefficient whenever the underlying task does not require complete memorization of the input sequences. In this situation, the hidden state size necessary to encode the entire sequence could be much larger than the minimal hidden state size necessary to solve the problem. Therefore, it is fundamental to allow the model to diverge from the orthogonality or the autoencoder solution by only imposing soft orthogonality constraints. Nonetheless, we believe that even when complete full memorization is not necessary, the autoencoder initialization can help to speed up training convergence by allowing the model to remember long sequences, which can then be gradually forgotten during training if deemed uninformative. In summary, the main contributions of the paper are: • the proposal of a novel initialization schema designed for the explicit memorization of long sequences; • highlighting the connection between orthogonal models and linear autoencoders for sequences; • an empirical analysis that shows the effectiveness of our proposed initialization schema to solve tasks that require long memorization. In this work, we studied the problem of building an autoencoder-based memorization mechanism. This system has the ability to encode and decode the input sequences to compute the desired target. Using results for the linear autoencoder for sequences, we showed how to initialize a recurrent neural network by approximating a linear autoencoder of the input sequences. The architecture exploits a linear recurrence to obtain a better approximation of the autoencoder. The results show that an autoencoder-based initialization can be effective for learning memorization tasks. In the future, we plan to extend this work by studying the effect of the autoencoder during training, possibly by enforcing the encoding of the input sequence even during the learning phase. Another possible avenue of research is the study of better optimization algorithms for the parameters of the linear component, where the linearity could be exploited to speed up the training process through dedicated learning algorithms.
We show how to initialize recurrent architectures with the closed-form solution of a linear autoencoder for sequences. We show the advantages of this approach compared to orthogonal RNNs.
{"src": "scitldr-AIC_train", "paper_id": "BkgM7xHYwH"}