Similarities between HMMs and RNNs have long been observed. Bourlard and Wellenkens [1]} have shown that the outputs of RNNs approximate the maximum a posteriori (MAP) output probabilities of HMMs trained via the Viterbi algorithm [2]}. Bridle [3]} also showed that the alpha part of the forward-backward algorithm can be simulated by a recurrent network, and that the beta part bears similarities with the backward computation of derivatives in the training of neural networks. Bidirectional RNNs were subsequently defined by Schuster and Paliwal [4]} to explicitly allow networks to take into account future observations. This approach was later applied to gated RNNs [5]} and is now the standard approach for LSTMs [6]}, GRUs [7]} and Li-GRUs [8]}. In a bidirectional recurrent layer, the size of all feedforward weight matrices (i.e., matrices that are applied to the layer inputs) is doubled compared to the unidirectional case, which represents a significant increase in the amount of trainable parameters. Recently, Garner and Tong [9]} have used a Bayesian interpretation of gated RNNs to derive a backward recursion through the input sequences. Their probabilistic approach, analogous to a Kalman smoother, allows the consideration of future observations without requiring any additional trainable parameters.
