The combination of all three techniques is applied in our experiments with 27 languages involved with the Transformer based model [1]}, [2]}. In the process of learning these languages in different iterations, we were able to minimize the effect of catastrophic forgetting for previously learned languages, while the performance of the new languages are still comparable or potentially competitive with training all languages from scratch. Our contribution in continual learning, to the best of our knowledge, the first application in learning to transcribe languages.
