Logical Reasoning with Span Predictions: Span-level Logical Atoms for Interpretable and Robust NLI Models
Current Natural Language Inference (NLI) models achieve impressive results,
sometimes outperforming humans when evaluating on in-distribution test sets.
However, as these models are known to learn from annotation artefacts and
dataset biases, it is unclear to what extent the models are learning the task
of NLI instead of learning from shallow heuristics in their training data. We
address this issue by introducing a logical reasoning framework for NLI,
creating highly transparent model decisions that are based on logical rules.
Unlike prior work, we show that the improved interpretability can be achieved
without decreasing the predictive accuracy. We almost fully retain performance
on SNLI while identifying the exact hypothesis spans that are responsible for
each model prediction. Using the e-SNLI human explanations, we also verify that
our model makes sensible decisions at a span level, despite not using any
span-level labels during training. We can further improve model performance and
the span-level decisions by using the e-SNLI explanations during training.
Finally, our model outperforms its baseline in a reduced data setting. When
training with only 100 examples, in-distribution performance improves by 18%,
while out-of-distribution performance improves on SNLI-hard, MNLI-mismatched,
MNLI-matched and SICK by 11%, 26%, 22%, and 21% respectively.