In the Chinese Language test of Gaokao, approximately half of the reading comprehension questions instruct students to select a statement (i.e., an option from four choices) that is the most consistent or contradicting with the given passage.
These questions are designed to evaluate students' ability in extracting and integrating information from long passages, and analyzing certain linguistic phenomena or semantic relations among several similar sentencesAccording to the Syllabus of Chinese Language in China by the Ministry of Education..
According to the target language skills, we call these questions verification style questions and convert them into the task of verifying given statements against the articles. Our task is similar in spirit to the fact checking task, FEVER [1]}. But, unlike the claims tagged with a definite label in FEVER (Supported, Refuted or Not-Enough-Info), the statements in Gaokao questions are designed to be not absolutely correct nor absolutely incorrect.
In other words, the options of a question in VGaokao differ in how much they are supported by the passage, which is difficult to be quantified with a value between 0 and 1.
The design of relative correctness is in the purpose of evaluating test-takers' ability to disambiguate very similar options through considering the subtle difference among them.
