In practice, it is also possible that only a subset of the languages is available at first, and the data for new languages can be added after the training process. Moreover, the data for previously trained languages might be discarded for storage or privacy reasons. In such case, the typical batch-training scenario often resorts to two options, either to fine-tune the models on the new datasets to obtain new models that are capable of transcribing new languages, or to combine the old and new datasets to construct new models that fit all languages. Naturally, both of these options are not optimal. Fine-tuning on new languages poses a threat for the previously learned languages to be forgotten, known as catastrophic forgetting [1]} that happened when the parameters of the neural networks are shifted towards optimizing the loss function for the new dataset, and far away from the optimal points with respect to the old ones. On the other hand, training all languages together can potentially obtain the best performance for all languages, but is costly since the training time of neural networks can scale depending on the amount of training data. Furthermore it is not possible when the previous languages are no longer available.
