The recent advances in neural networks have opened the way to the design of architecturally simple multilingual models for various NLP tasks, such as language modeling or next word prediction (Tsvetkov et al., 2016; Östling and Tiedemann, 2017; Malaviya et al., 2017; Tiedemann, 2018) , translation (Dong et al., 2015; Zoph et al., 2016; Firat et al., 2016; <cite>Johnson et al., 2017)</cite> , morphological reinflection (Kann et al., 2017) and more (Bjerva, 2017) .