Sharing across Layers. Recently, [1]} showed that sharing parameters across layers in deep neural networks does not hurt the representational power of the network. Furthermore, [2]} demonstrated that cross-layer parameter sharing in the Transformer leads to a lighter and faster-to-train model without sacrificing the performance on various language understanding benchmarks. Motivated by this, we let each Transformer share parameters across different layers.
