TITLE: MRCLens: an MRC Dataset Bias Detection Toolkit 

ABSTRACT: Many recent neural models have shown remarkable empirical results in Machine
Reading Comprehension, but evidence suggests sometimes the models take
advantage of dataset biases to predict and fail to generalize on out-of-sample
data. While many other approaches have been proposed to address this issue from
the computation perspective such as new architectures or training procedures,
we believe a method that allows researchers to discover biases, and adjust the
data or the models in an earlier stage will be beneficial. Thus, we introduce
MRCLens, a toolkit that detects whether biases exist before users train the
full model. For the convenience of introducing the toolkit, we also provide a
categorization of common biases in MRC.
Many recent neural models have shown remarkable empirical results in Machine
Reading Comprehension, but evidence suggests sometimes the models take
advantage of dataset biases to predict and fail to generalize on out-of-sample
data. While many other approaches have been proposed to address this issue from
the computation perspective such as new architectures or training procedures,
we believe a method that allows researchers to discover biases, and adjust the
data or the models in an earlier stage will be beneficial. Thus, we introduce
MRCLens, a toolkit that detects whether biases exist before users train the
full model. For the convenience of introducing the toolkit, we also provide a
categorization of common biases in MRC.
MRCLens: an MRC Dataset Bias Detection Toolkit
http://arxiv.org/abs/2207.08943v1