As the distribution of weight matrices' eigenvalues can be different across different LSTM layers, we follow the practice of <cite>[11]</cite> to set the same threshold τ across layers as the fraction of retained singular values, defined as τ = Table 1 summarizes the results of low-rank matrix factorization compared to our baseline 3-layer LSTM.