We focus on two widely used and effective methods for deep models: low-rank matrix factorization and and quantization. Singular value decomposition (SVD) is a common factorization technique and has been explored in feedforward networks [9, 10, 21, 22] and recurrent neural networks (RNN) <cite>[11]</cite> .