TITLE: XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning 

ABSTRACT: In order to simulate human language capacity, natural language processing
systems must be able to reason about the dynamics of everyday situations,
including their possible causes and effects. Moreover, they should be able to
generalise the acquired world knowledge to new languages, modulo cultural
differences. Advances in machine reasoning and cross-lingual transfer depend on
the availability of challenging evaluation benchmarks. Motivated by both
demands, we introduce Cross-lingual Choice of Plausible Alternatives (XCOPA), a
typologically diverse multilingual dataset for causal commonsense reasoning in
11 languages, which includes resource-poor languages like Eastern Apur\'imac
Quechua and Haitian Creole. We evaluate a range of state-of-the-art models on
this novel dataset, revealing that the performance of current methods based on
multilingual pretraining and zero-shot fine-tuning falls short compared to
translation-based transfer. Finally, we propose strategies to adapt
multilingual models to out-of-sample resource-lean languages where only a small
corpus or a bilingual dictionary is available, and report substantial
improvements over the random baseline. The XCOPA dataset is freely available at
github.com/cambridgeltl/xcopa.
In order to simulate human language capacity, natural language processing
systems must be able to reason about the dynamics of everyday situations,
including their possible causes and effects. Moreover, they should be able to
generalise the acquired world knowledge to new languages, modulo cultural
differences. Advances in machine reasoning and cross-lingual transfer depend on
the availability of challenging evaluation benchmarks. Motivated by both
demands, we introduce Cross-lingual Choice of Plausible Alternatives (XCOPA), a
typologically diverse multilingual dataset for causal commonsense reasoning in
11 languages, which includes resource-poor languages like Eastern Apur\'imac
Quechua and Haitian Creole. We evaluate a range of state-of-the-art models on
this novel dataset, revealing that the performance of current methods based on
multilingual pretraining and zero-shot fine-tuning falls short compared to
translation-based transfer. Finally, we propose strategies to adapt
multilingual models to out-of-sample resource-lean languages where only a small
corpus or a bilingual dictionary is available, and report substantial
improvements over the random baseline. The XCOPA dataset is freely available at
github.com/cambridgeltl/xcopa.
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning
http://arxiv.org/abs/2005.00333v2