To avoid complexities of asynchronous parallel training with shared parameter server (Dean et al., 2012) , the architecture in Fig.2 and Fig. 3 instead can be trained using the alternating training approach proposed in <cite>(Luong et al., 2016)</cite> , where each task is optimized for a fixed number of parameter updates (or mini-batches) before switching to the next task (which is a different language pair).