arXiv:2104.07838v1  [cs.CL]  16 Apr 2021Investigating Failures of Automatic Translation
in the Case of Unambiguous Gender
Adithy
Investigating Failures of Automatic Translation in the Case of Unambiguous Gender
Transformer based models are the modern work horses for neural machine
translation (NMT), reaching state of the art across several benchmarks. Despite
their impressive accuracy, we observe a systemic and rudimentary class of
errors made by transformer based models with regards to translating from a
language that doesn't mark gender on nouns into others that do. We find that
even when the surrounding context provides unambiguous evidence of the
appropriate grammatical gender marking, no transformer based model we tested
was able to accurately gender occupation nouns systematically. We release an
evaluation scheme and dataset for measuring the ability of transformer based
NMT models to translate gender morphology correctly in unambiguous contexts
across syntactically diverse sentences. Our dataset translates from an English
source into 20 languages from several different language families. With the
availability of this dataset, our hope is that the NMT community can iterate on
solutions for this class of especially egregious errors.
Adithya Renduchintala Adina Williams