All our models are built using the ESPNet[1]} toolkit, and will be publicly released to encourage further research.
We use HUBERT-large [2]} embeddings as the input features and a 4 layer conformer encoder with 64 hidden units for our models. We freeze the HUBERT-large frontend, and train the conformer and pooling decoder parameters. We employ separate self-attentive pooling[3]} layers for continuous and discrete emotion prediction so the model can learn which frames to focus on in order to make emotion predictions. Our decoders consist of linear projections that map from the encoded output dimension of 768 to 64, 32 and consequently the output sizes of 3 for the continuous prediction and 5 for the discrete prediction. We use a ReLU activation to ensure that the predicted continuous attributes are strictly positive and use LeakyReLU elsewhere. We use a dropout of 0.2 in the decoder. Our models are trained with the Adam optimizer, and a peak learning rate of 1e-3 for 15,000 warmup steps.
