All trainable parameters in the model are optimized with a gradient-based stochastic Adam solver [1]}. Mini-batch size and learning rate are set to 64 and 0.001 respectively. The learning rate is decreased exponentially with the factor of 0.1 for every 40K iteration. Also, we set \(\beta _1\)  value in [1]} to \(0.7\)  to reduce the oscillations in learning phase. Number of iterations varies from 50K to 200K depending on the loss changes between the training and validation sets for each emotion type.
