Parameter Reduction. Network pruning [1]}, [2]} trains a large model and then reduces its size while maintaining performance. Reducing the size of CNNs for mobile applications is an active research area [3]}, [4]}, [5]}, [6]}, [7]}. Our work is closely related to the work that shares parameters across layers in deep neural networks. Trellis network [8]} is a temporal convolutional architecture with weight-tying across time and depth. Similar to ours, Universal Transformer [9]}, RSNMT [10]}, DEQ [11]}, ALBERT [12]} share weights across layers in Transformers. We combine this idea with our novel cross-Transformer weight sharing, which decomposes weight matrices with low-rank approximation.
