Learning new languages without forgetting falls into the intersection between automatic speech recognition and continual learning/incremental learning areas. In this paper, we presented a model for this specific task with the combination of Elastic Weight Consolidation and Weight factorization. This model exhibits an interesting property when learning languages sequentially, that it can learn the new languages very effectively compared to other regularized baselines, while maintaining the performance for the previously learned languages to an acceptable extent.
We provided an analysis to empirically explain the effectiveness (and ineffectiveness) of EWC and aim at improving the weaknesses of the model to improve further on this task. One of the missing ingredients is to use distillation from the previous models as a compression of the data that is promising, which is shown in many concurrent work in image generation [1]}.
