However, the PT/FT approach has a number of limitations. First, it is unclear how
to initialize a model given learned weights from a sequence of related tasks
[1]}. Second, when a model is
fine-tuned using initial weights learned from a source task, the end-model loses
its ability to solve the source task, a phenomenon termed the “forgetting effect” [2]}.
Finally, transferring learned parameters
between networks can be challenging if the networks have inconsistent architectures.
In this work, we use the PT/FT approach as our baseline for
transfer learning.
