A common approach to deal with domain scaling is transfer learning (TF).We do not differentiate between domain adaptation and transfer learning in this paper. In the TF setup we have \(K\)  source domains \(\mathcal {D}^1_{S}, \mathcal {D}^{2}_{S}, \ldots , \mathcal {D}^{K}_{S}\) 
and a target domain \(\mathcal {D}^{K+1}_{T}\) , and we assume abundance of data in \(\mathcal {D_S}\)  and limited data in \(\mathcal {D_T}\) . Instead of
training a target model \(\mathcal {M}_T\)  for \(\mathcal {D}_T\)  from scratch, TF aims to adapt the learned model \(\mathcal {M}_S\)  from \(\mathcal {D}_S\)  to produce a model \(\mathcal {M}_T\)  trained on \(\mathcal {D}_T\) . TF is typically applied with various parameter sharing and training mechanisms. For SF and IC two approaches are proposed, namely data-driven and model- driven. As for data-driven techniques, typically we combine data from \(\mathcal {D}_S\)  and \(\mathcal {D}_T\)  and we partition the parameters in the model into parts that are task-specific and parameters that are shared across tasks.
Some studies [1]}, [2]}, [3]} apply this technique using multi-task learning (MTL) [4]} and the models are trained simultaneously on \(\mathcal {D}_S\)  and \(\mathcal {D}_T\)  (Figure REF  Left). Results have shown that MTL is particularly effective relative to single-task learning (STL) when the data in \(\mathcal {D_T}\)  is scarce and the benefits over STL diminish as more data is available. Another technique that is typically used in data-driven approaches is based on pre-train and fine-tune mechanisms. DBLP:conf/naacl/GoyalMM18 train a joint model of SF and IC, \(\mathcal {M}_S\) , on large \(\mathcal {D_S}\) , then fine-tune \(\mathcal {M}_S\)  by replacing the output layer corresponding with the label space from \(\mathcal {D}_T\)  and train the model further on \(\mathcal {D}_T\) . DBLP:conf/aaai/SiddhantGM19 also uses fine-tuning mechanism, but the main difference with DBLP:conf/naacl/GoyalMM18 is they leverage large unlabeled data to learn contextual embedding, ELMo [5]}, before fine-tuning on \(\mathcal {D}_T\) .
