ProbSparse self-attention uses the dominant dot-product pairs by randomly selecting \(\log L\)  top query vectors and setting the rest of the query vector values to zero. The computational complexity and memory usage reduce with \(L\)  value vectors to \(\mathcal {O}(L\log L)\) . With a stack of \(J\)  encoders, the total memory usage reduces to \(\mathcal {O}(J L\log L)\) . Furthermore, the self-attention distilling operation removes redundant combinations of value vectors. This operation is inspired by dilated convolution as proposed in [1]} and [2]}. The output of the multi-head self-attention is fed into 1-D convolution filters with kernel size \(= 3\) . Later, an exponential linear unit (ELU) activation function is applied, followed by a max-pooling operation with a stride of 2. These operations reduce the size in half and thus, form a pyramid, as shown in Figure REF . Effectively, the total space complexity reduces considerably. Stacked replicas are also built, with an input length half of the previous stack. Figure REF  shows only one replica stack. The output of the main stack and the replica stacks have the same dimension and are concatenated to form the final output of the encoder. These replica stacks enhance the robustness of the distilling operation.
