Optimizing our model is challenging due to the large memory requirement. The most expensive part is the Transformers, which take up 76% of model parameters. One could reduce the model size by making the Transformers shallower, but the depth of Transformers has shown to be crucial to get good performance [1]}. We propose to reduce the model size by aggressively sharing parts of weights across Transformers as well as layers within each Transformer (see Figure REF  (d)).
