The encoder is a CNN-LSTM, with four 2-dimensional convolution layers with (3,3) kernals, with two of the convolution layers downsampling to a total of \(\frac{1}{4}\)  the time frames.
The CNN is followed by four bidirectional LSTM layers to produce a 320-dimensional embedding at each downsampled time-step.
The decoder consists of three unidirectional LSTM layers with Bahdanau attention [1]}.
All the supervised models use 320 dimension LSTMs, while the semisupervised models vary across languages between 320 and 512 for each LSTM, though the layer sizes are consistent within a given model.
We also add an additional layer in the encoder for the semisupervised models.
The output targets used for all languages are 1000 subword units generated using the SentencePiece implementation of [2]}.
