In order to realize the desiderata, the key idea is to combine three different techniques: transfer learning, weight factorization and elastic weight consolidation, by the following reasons. On the one hand, neural sequence-to-sequence models rely on a fixed vocabulary output, which is likely to be changed when the models are exposed with new languages. Multilingual pretrained language models [1]} can relax this problem, by providing the syntax knowledge of the new languages, so that the output layer can be easily applied for new languages. On the other hand, Weight factorization [2]} facilitates multilingual models by decomposing each matrix in the neural network into a shared component and per-language additive and multiplicative factors. Motivated by progressive neural networks [3]}, this modeling scheme can prevent catastrophic forgetting by offloading information in the language specific weights. Meanwhile, elastic weight consolidation [4]} can be used to find the “empty space” in the shared components of the networks to accommodate new languages.
