With the current success of machine learning techniques, speech recognition architectures can be trained in an end-to-end fashion, by exploiting auto-differentiation inside deep learning frameworks like PyTorch [1]}. Here, recurrence is also an important concept and recurrent neural networks (RNNs) are commonly trained via the error Back-Propagation Through Time (BPTT) algorithm [2]}, [3]}, which is a generalization of gradient descent to process sequential data. During the forward pass, a batch of input examples is passed through the network, and a loss function is applied to the final outputs. During the subsequent backward computation of derivatives, the network trainable parameters are updated to minimize the loss.
