Long Short-Term Memory (LSTM) units have the ability to memorise and use long-term dependencies between inputs to generate predictions on time series data. We introduce the concept of modifying the cell state (memory) of LSTMs using rotation matrices parametrised by a new set of trainable weights. This addition shows significant increases of performance on some of the tasks from the bAbI dataset. In the recent years, Recurrent Neural Networks (RNNs) have been successfully used to tackle problems with data that can be represented in the shape of time series. Application domains include Natural Language Processing (NLP) (translation BID12 , summarisation BID9 , question answering and more), speech recogition BID5 BID3 ), text to speech systems BID0 , computer vision tasks BID13 BID16 , and differentiable programming language interpreters BID10 BID11 ).An intuitive explanation for the success of RNNs in fields such as natural language understanding is that they allow words at the beginning of a sentence or paragraph to be memorised. This can be crucial to understanding the semantic content. Thus in the phrase "The cat ate the fish" it is important to memorise the subject (cat). However , often later words can change the meaning of a senstence in subtle ways. For example , "The cat ate the fish, didn't it" changes a simple statement into a question. In this paper , we study a mechanism to enhance a standard RNN to enable it to modify its memory, with the hope that this will allow it to capture in the memory cells sequence information using a shorter and more robust representation.One of the most used RNN units is the Long Short-Term Memory (LSTM) BID7 . The core of the LSTM is that each unit has a cell state that is modified in a gated fashion at every time step. At a high level , the cell state has the role of providing the neural network with memory to hold long-term relationships between inputs. There are many small variations of LSTM units in the literature and most of them yield similar performance BID4 .The memory (cell state) is expected to encode information necessary to make the next prediction. Currently the ability of the LSTMs to rotate and swap memory positions is limited to what can be achieved using the available gates. In this work we introduce a new operation on the memory that explicitly enables rotations and swaps of pairwise memory elements. Our preliminary tests show performance improvements on some of the bAbI tasks compared with LSTM based architectures. A limitation of the models in our experiments is only applying pairwise 2D rotations. Representations of past input can be larger groups of the cell state vector, thus 2D rotations might not fully exploit the benefits of transformations. In the future we hope to explore rotating groups of elements and multi-dimensional rotations. Rotating groups of elements of the cell state could potentially also force the models to learn a more structured representation of the world, similar to how forcing a model to learn specific representations of scenes, as presented in BID6 , yields semantic representations of the scene.Rotations also need not be fully flexible. Introducing hard constraints on the rotations and what groups of parameters can be rotated might lead the model to learn richer memory representations. Future work could explore how adding such constraints impacts learning times and final performance on different datasets, but also look at what constraints can qualitatively improve the representation of long-term dependencies.In this work we presented prelimiary tests for adding rotations to simple models but we only used a toy dataset. The bAbI dataset has certain advantages such as being small thus easy to train many models on a single machine, not having noise as it is generated from a simulation, and having a wide range of tasks of various difficulties. However it is a toy dataset that has a very limited vocabulary and lacks the complexity of real world datasets (noise, inconsistencies, larger vocabularies, more complex language constructs, and so on). Another limitation of our evaluation is only using text, specifically question answering. To fully evaluate the idea of adding rotations to memory cells, in the future, we aim to look into incorporating our rotations on different domains and tasks including speech to text, translation, language generation, stock prices, and other common problems using real world datasets.Tuning the hyperparameters of the rotation models might give better insights and performance increases and is something we aim to incorporate in our training pipeline in the future.A brief exploration of the angles produced by u and the weight matrix W rot show that u does not saturate, thus rotations are in fact applied to our cell states and do not converge to 0 (or 360 degress). A more in-depth qualitative analysis of the rotation gate is planned for future work. Peeking into the activations of our rotation gates could help understand the behaviour of rotations and to what extent they help better represent long-term memory.A very successful and popular mutation of the LSTM is the Gated Recurrent Unit (GRU) unit BID1 . The GRU only has an output as opposed to both a cell state and an output and uses fewer gates. In the future we hope to explore adding rotations to GRU units and whether we can obtain similar results. We have introduced a novel gating mechanism for RNN units that enables applying a parametrised transformation matrix to the cell state. We picked pairwise 2D rotations as the transformation and shown how this can be added to the popular LSTM units to create what we call RotLSTM. Figure 3: Accuracy comparison on training, validation (val) and test sets over 40 epochs for LSTM and RotLSTM models. The models were trained 10 times and shown is the average accuracy and in faded colour is the standard deviation. Test set accuracy was computed every 10 epochs.We trained a simple model using RotLSTM units and compared them with the same model based on LSTM units. We show that for the LSTM-based architetures adding rotations has a positive impact on most bAbI tasks, making the training require fewer epochs to achieve similar or higher accuracy. On some tasks the RotLSTM model can use a lower dimensional cell state vector and maintain its performance. Significant accracy improvements of approximatively 20% for the RotLSTM model over the LSTM model are visible on bAbI tasks 5 (three argument relations) and 18 (reasoning about size).
Adding a new set of weights to the LSTM that rotate the cell memory improves performance on some bAbI tasks.
{"src": "scitldr-AIC_train", "paper_id": "ByUEelW0-"}