Recurrent Neural Networks (RNNs) are powerful, stateful models that capture temporal dependencies in sequential data. They have been widely used in solving complex tasks, such as machine translation [1]}, [2]}, [3]}, language modeling [4]}, [5]}, [6]}, multimodal language modelling [7]}, [8]} and speech recognition [9]}. In theory, first order RNNs with the desired weights (arranged as 2D weight matrices) and infinite precision have been shown to be as powerful as a pushdown automaton (PDA) [10]}, [11]}, [12]} and can be considered to be computationally universal models [13]}, [14]}. However, a higher order model (such as second order) offers further representational capability – [15]}, [16]}, [17]} proved that any finite state machine can be stably constructed in a 2nd order, or tensor, RNN.
