In this work, we try to decipher the internal connection of NLP technology development in the past decades, searching for essence, which rewards us with a (potential)  new learning paradigm for NLP tasks, dubbed asreStructured Pre-training(RST) .
In such a paradigm, the role ofdatawill be re-emphasized, and model pre-training and fine-tuning of downstream tasks are viewed as a process of data storing and accessing.
Based on that, we operationalize the simple principle thata good storage mechanism should not only have the ability to cache a large amount of data but also consider the ease of access. We achieve this by pre-training models overrestructureddata that consist of a variety of valuable information instead of raw data after overcoming several engineering challenges.
Experimentally, RST models not onlysurpass strong competitors (e.g., T0)  on 52/55 popular datasetsfrom a variety of NLP tasks (e.g., classification, information extraction, fact retrieval, text generation, etc.)  without fine-tuning on downstream tasks, but also achieve superior performance in National College Entrance Examination - English (Gaokao-English) , the most authoritative examination in China, which millions of students will attend every year.
Specifically, the proposed system Qin () achieves 40 points higher than the average
scores made by students and 15 points higher than GPT3 with 1/16 parameters. In particular, Qin gets a high score of 138.5 (the full mark is 150)  in the 2018 English exam (national paper III) .
We have released the Gaokao Benchmark with an online submission platform that contains ten annotated English papers from 2018-2021 so far (and will be expanded annually) , which allows more AI models to attend Gaokao, establishing a relatively fair test bed for human and AI competition and helping us better understandwhere we are. We test our model in the 2022 College Entrance Examination English that happened a few days ago (2022.06.08) , andit gets a total score of 134 (v.s. GPT3’s 108) .We released alldata111https://github.com/ExpressAI/reStructured-Pretrainingandmodels.222https://huggingface.co/XLab/rst-all-11b