As discussed in Section 4.3, for our transfer learning optimization experiments, we segment the optimization of different parameters of our model with different learning rates, in order to limit catastrophic forgetting and interference among the tasks, as proposed by<cite> [Howard and Ruder, 2018]</cite> .