In the sense that training a language model as a processing pipeline for solving NLU tasks, our study relates to ongoing research on models for pre-training generalizable language representations.
In [1]}, the authors suggested to use contextualized embeddings obtained from a pre-trained LM as input to another task, demonstrating it on challenging NLP tasks.
ULMfit [2]} and BERT [3]} employ an approach that replaces the final layer of a pre-trained LM according to the form of a target task and fine-tunes the network, and in particular the latter [3]} showed significantly improved performance on multiple benchmark datasets.
Since our method focuses on SPE, the above fine-tuning methods can be applied equally to ours, which may lead to further improvement.
