The encoder consists of a 10 layer CNN (CNN10) and the encoder of the transformer, and takes in log-mel spectrograms as input. CNN10 is adapted from different CNN architectures availables in the Pretrained Audio Neural Networks collection [1]}. The output of CNN10 is passed to a transformer encoder. The transformer encoder [2]} consists of two layers with a hidden size of 192. We hypothesize that the multihead attention mechanism in the transformer encoder is instrumental in learning higher-level representations from the features provided by the CNN. In our experiments, we show that using the transformer encoder in transfer learning provides a huge boost in performance. Where transfer learning is applied, only the CNN10 encoder is preloaded with pretrained weights from, while the transformer encoder is always trained from scratch.
