Testing the Reasoning Power for NLI Models with Annotated Multi perspective Entailment Dataset
Natural language inference (NLI) is a challenging task to determine the relationship between a pair of sentences. Existing Neural Network based (NN based) models have achieved prominent success. However, rare models are interpretable. In this paper, we propose a Multi perspective Entailment Category Labeling System (METALs) It consists of three categories, ten sub categories. We manually annotate 3,368 entailment items. The annotated data is used to explain the recognition ability of four NN based models at a fine grained level. The experimental results show that all the models have poor performance in the commonsense reasoning than in other entailment categories. The highest accuracy difference is 13.22%
CCL