In our work, we analyze the contribution of differant layers and heads through attention visualization, as depicted in Fig 1. We consider that the diversity of information between blocks is useful. We study how to take advantage of the output information of each block. We believe that the output information of each block in the encoder and decoder is not completely inclusive, and perhaps complementary. Combine the output of each block efficiently to make full use of complementary information.
Motivated by this, we introduce a novel model, named Blockformer, achieves state-of-the-art results on Aishell-1. The Blockformer adds a block ensemble module, which exploits the output information of each block in a parameter-efficient way.
We have implemented two block ensemble methods: the base Weighted Sum of the Blocks Output (Base-WSBO), and the Squeeze-and-Excitation module[1]} to Weighted Sum of the Blocks Output (SE-WSBO).
Our experiments are mainly conducted on a public Mandarin Chinese dataset AISHELL-1. Results show that a CER of 4.35% without using a language model and 4.10% with an external language model on the testset.
