Low-Rank, Stabilized, and Quantized Recurrent Models:
Although recurrent neural networks (RNN) are lightweight by design, they suffer from exploding and vanishing gradient problem (EVGP) for long time-series sequences. Widely-used solutions to EVGP, namely long short-term memory (LSTM) [1]}, gated recurrent units [2]}, and unitary RNN [3]} either cause loss in accuracy, or increase memory and compute overhead. FastRNN [4]} solves EVGP by adding a weighted residual connection with two scalars between RNN layers to stabilize gradients during training without adding significant compute overhead. The scalars control the hidden state update extent based on inputs. FastGRNN [4]} then converts the residual connection to a gate while enforcing the RNN matrices to be low-rank, sparse, and quantized. The resulting RNN is 35\(\times \)  smaller than gated or unitary RNN. Kronecker recurrent units [6]}[7]} use Kronecker products to stabilize RNN training and decompose large RNN matrices into rank-preserving smaller matrices with fewer parameters, compressing RNN by 16-50\(\times \)  without significant accuracy loss. Doping, co-matrix adaptation and co-matrix regularization can further compress Kronecker recurrent units by 1.3-2.4\(\times \)  [8]}. Legendre memory units (LMU) [9]}, derived to orthogonalize its continuous-time history, have 10,000\(\times \)  more capacity while being 100\(\times \)  smaller than LSTM.
<TABLE>