Automatic Essay Scoring (AES) has been an active research area as it can greatly reduce the workload of teachers and prevents subjectivity bias . Most recent AES solutions apply deep neural network (DNN)-based models with regression, where the neural neural-based encoder learns an essay representation that helps differentiate among the essays and the corresponding essay score is inferred by a regressor. Such DNN approach usually requires a lot of expert-rated essays as training data in order to learn a good essay representation for accurate scoring. However, such data is usually expensive and thus is sparse. Inspired by the observation that human usually scores an essay by comparing it with some references, we propose a Siamese framework called Referee Network (RefNet) which allows the model to compare the quality of two essays by capturing the relative features that can differentiate the essay pair. The proposed framework can be applied as an extension to regression models as it can capture additional relative features on top of internal information. Moreover, it intrinsically augment the data by pairing thus is ideal for handling data sparsity. Experiment shows that our framework can significantly improve the existing regression models and achieve acceptable performance even when the training data is greatly reduced. Automatic Essay Scoring (AES) is the technique to automatically score an essay over some specific marking scale. AES has been an eye-catching problem in machine learning due to its promising application in education. It can free tremendous amount of repetitive labour, boosting the efficiency of educators. Apart from automation, computers also prevail human beings in consistency, thus eliminate subjectivity and improve fairness in scoring. Attempts in AES started as early as Project Essay Grade (PEG) (Page, 1967; 2003) , when the most prevalent methods relied on hand-crafted features engineered by human experts. Recent advances in neural networks bring new possibilities to AES. Several related works leveraged neural networks and achieved decent results (Dong et al., 2017; Taghipour & Ng, 2016; Tay et al., 2018; Liu et al., 2019) . As is shown in Figure 1 , these approaches generally follow the 'representation + regression' scheme where a neural network reads in the text embeddings and generates a high level representation that will be fed to some regression model for a score. However, such model requires a large amount of expert-rated essays for training. In reality, collecting such dataset is expensive. Therefore, data sparsity remains a knotty problem to be solved. Inspired by the observation that human raters usually score an essay by comparing it to a set of references, we propose to leverage the pairwise comparisons for scoring instead of regression. The goal of the model is shifted from predicting the score directly to comparing two essays, and the final score will be determined by comparing new essays with known samples. In order to achieve this, we designed a Siamese network called Referee Network (RefNet) and corresponding scoring algorithms. RefNet is a framework so that it can use various representation encoders as backbones. What's more, though this model is designed to capture mutual features, it can also benefit from essay internal information via transfer learning. Scoring essays by comparison has various benefits. First, RefNet is incredibly strong in dealing with data sparsity problem. Essays are paired with each other to form the training data for RefNet, which significantly augmented the data size. Experiments show that our model achieve acceptable performance even when the training data is radically reduced, while regression models are subject to drastic performance degeneration. Second, unlike end-to-end black-box models, our system scores an essay by comparing it with a set of labeled anchors, providing transparency to a certain degree during inference process. Last but not least, with information in both internal and mutual perspective, RefNet can have better insight into the quality of essays. Our contributions can be summarized as follows: • We designed Referee Network (RefNet), a simple but effective model to compare two essays, and Majority Probability Voting Algorithm to infer the score from pairwise comparison results. To the best of our knowledge, it is the first time a Siamese neutral network is used in AES. • Our model intrinsically solves the problem of data sparsity. It achieves acceptable performance even when the training data is greatly reduced, while regression models are impaired a lot. Its efficacy in few-shot learning makes it an ideal solution for real applications where labelled data is usually limited. • RefNet exploits a new realm of information, mutual relationship, by pairwise comparison. With transfer learning, it also leverages internal features captured by regression. Moreover, RefNet can be applied as an extension to various regression models and consistently improve the performance. In this paper we present Referee Network, a framework for automatic essay scoring using pairwise comparisons. We demonstrate that RefNet is expert in solving data sparsity problems. It can retain the performance at a high level even when the training data is significantly reduced, which outperforms regression models by a significant margin. We also show that RefNet can improve conventional regression models by leveraging the additional mutual information between representations. With only vanilla backbones, our model is able to obtain state-of-the-art results. Even if the essay representations are fixed and have mediocre quality, our model can still boost up the scoring accuracy. Furthermore, the capacity of RefNet can go far beyond this context as it is an extendable framework that can be used with any kind of representation encoders. Besides the simple backbones we tried in this paper, one can by all means utilize more complicated and better performing models as backbones. In this way, the performance of AES systems can always be pushed to a new record.
Automatically score essays on sparse data by comparing new essays with known samples with Referee Network. 
{"src": "scitldr-AIC_train", "paper_id": "SyePUgBtPr"}