reStructured Pre-training
In this work, we try to decipher the internal connection of NLP technology
development in the past decades, searching for essence, which rewards us with a
(potential) new learning paradigm for NLP tasks, dubbed as reStructured
Pre-training (RST). In such a paradigm, the role of data will be re-emphasized,
and model pre-training and fine-tuning of downstream tasks are viewed as a
process of data storing and accessing. Based on that, we operationalize the
simple principle that a good storage mechanism should not only have the ability
to cache a large amount of data but also consider the ease of access. We
achieve this by pre-training models over restructured data that consist of a
variety of valuable information instead of raw data after overcoming several
engineering challenges. Experimentally, RST models not only surpass strong
competitors (e.g., T0) on 52/55 popular datasets from a variety of NLP tasks,
but also achieve superior performance in National College Entrance Examination
- English (Gaokao-English),the most authoritative examination in China.
Specifically, the proposed system Qin achieves 40 points higher than the
average scores made by students and 15 points higher than GPT3 with 1/16
parameters. In particular, Qin gets a high score of 138.5 (the full mark is
150) in the 2018 English exam (national paper III). We have released the Gaokao
Benchmark with an online submission platform.
  In addition, we test our model in the 2022 College Entrance Examination
English that happened a few days ago (2022.06.08), and it gets a total score of
134 (v.s. GPT3's 108).