In this work, we first conduct mathematical analysis on the memory, which is
 defined as a function that maps an element in a sequence to the current output,
 of three RNN cells; namely, the simple recurrent neural network (SRN), the long
 short-term memory (LSTM) and the gated recurrent unit (GRU). Based on the
 analysis, we propose a new design, called the extended-long short-term memory
 (ELSTM), to extend the memory length of a cell. Next, we present a multi-task
 RNN model that is robust to previous erroneous predictions, called the dependent
 bidirectional recurrent neural network (DBRNN), for the sequence-in-sequenceout
 (SISO) problem. Finally, the performance of the DBRNN model with the
 ELSTM cell is demonstrated by experimental results. The recurrent neural network (RNN) has proved to be an effective solution for natural language processing (NLP) through the advancement in the last three decades BID8 BID11 BID2 BID1 . At the cell level of a RNN, the long short-term memory (LSTM) BID10 and the gated recurrent unit (GRU) are often adopted by a RNN as its low-level building cell. Being built upon these cells, various RNN models have been proposed to solve the sequence-in-sequence-out (SISO) problem. To name a few, there are the bidirectional RNN (BRNN) BID14 , the encoder-decoder model BID15 BID16 BID0 and the deep RNN BID12 . Although the LSTM and the GRU were designed to enhance the memory length of RNNs and avoid the gradient vanishing/exploding issue BID10 BID13 BID3 , a good understanding of their memory length is still lacking. Here, we define the memory of a RNN model as a function that maps an element in a sequence to current output. The first objective of this research is to analyze the memory length of three RNN cells -the simple RNN (SRN) BID8 BID11 , the long short-term memory (LSTM) and the gated recurrent unit (GRU). This will be conducted in Sec. 2. Such analysis is different to the investigation of gradient vanishing/exploding problem in a sense that gradient vanishing/exploding problem happens during the training process, the memory analysis is, however, done on a trained RNN model. Based on the understanding from the memory analysis, we propose a new design, called the extended-long short-term memory (ELSTM), to extend the memory length of a cell in Sec.3.As to the macro RNN model, one popular choice is the BRNN. Since the elements in BRNN output sequences should be independent of each other BID14 , the BRNN cannot be used to solve dependent output sequence problem alone. Nevertheless, most language tasks do involve dependent output sequences. The second choice is the encoder-decoder system, where the attention mechanism has been introduced BID16 BID0 to improve its performance furthermore. As shown later in this work, the encoder-decoder system is not an efficient learner. Here, to take advantages of both the encoder-decoder and the BRNN and overcome their drawbacks, we propose a new multitask model called the dependent bidirectional recurrent neural network (DBRNN), which will be elaborated in Sec. 4. Furthermore, we conduct a series of experiments on the part of speech (POS) tagging and the dependency parsing (DP) problems in Sec. 5 to demonstrate the performance of the DBRNN model with the ELSTM cell. Finally, concluding remarks are given and future research direction is pointed out in Sec. 6. The memory decay behavior of the LSTM and the GRU was investigated and explained by mathematical analysis. Although the memory of the LSTM and the GRU fades slower than that of the SRN, it may not be long enough for complicated language tasks such as dependency parsing. To enhance the memory length, two cells called the ELSTM-I and the ELSTM-II were proposed. Furthermore, we introduced a new RNN model called the DBRNN that has the merits of both the BRNN and the encoder-decoder. It was shown by experimental results that the ELSTM-I and ELSTM-II outperforms other designs by a significant margin for complex language tasks. The DBRNN design is superior to BRNN as well as sequence-to-sequence models for both simple and complex language tasks. There are interesting issues to be further explored. For example, is the ELSTM cell also helpful in more sophisticated RNN models such as the deep RNN? Is it possible to make the DBRNN deeper and better? They are left for future study.
A recurrent neural network cell with extended-long short-term memory and a multi-task RNN model for sequence-in-sequence-out problems
{"src": "scitldr-AIC_train", "paper_id": "r1AMITFaW"}