TITLE: Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translation 

ABSTRACT: Despite being the seventh most widely spoken language in the world, Bengali
has received much less attention in machine translation literature due to being
low in resources. Most publicly available parallel corpora for Bengali are not
large enough; and have rather poor quality, mostly because of incorrect
sentence alignments resulting from erroneous sentence segmentation, and also
because of a high volume of noise present in them. In this work, we build a
customized sentence segmenter for Bengali and propose two novel methods for
parallel corpus creation on low-resource setups: aligner ensembling and batch
filtering. With the segmenter and the two methods combined, we compile a
high-quality Bengali-English parallel corpus comprising of 2.75 million
sentence pairs, more than 2 million of which were not available before.
Training on neural models, we achieve an improvement of more than 9 BLEU score
over previous approaches to Bengali-English machine translation. We also
evaluate on a new test set of 1000 pairs made with extensive quality control.
We release the segmenter, parallel corpus, and the evaluation set, thus
elevating Bengali from its low-resource status. To the best of our knowledge,
this is the first ever large scale study on Bengali-English machine
translation. We believe our study will pave the way for future research on
Bengali-English machine translation as well as other low-resource languages.
Our data and code are available at https://github.com/csebuetnlp/banglanmt.
Despite being the seventh most widely spoken language in the world, Bengali
has received much less attention in machine translation literature due to being
low in resources. Most publicly available parallel corpora for Bengali are not
large enough; and have rather poor quality, mostly because of incorrect
sentence alignments resulting from erroneous sentence segmentation, and also
because of a high volume of noise present in them. In this work, we build a
customized sentence segmenter for Bengali and propose two novel methods for
parallel corpus creation on low-resource setups: aligner ensembling and batch
filtering. With the segmenter and the two methods combined, we compile a
high-quality Bengali-English parallel corpus comprising of 2.75 million
sentence pairs, more than 2 million of which were not available before.
Training on neural models, we achieve an improvement of more than 9 BLEU score
over previous approaches to Bengali-English machine translation. We also
evaluate on a new test set of 1000 pairs made with extensive quality control.
We release the segmenter, parallel corpus, and the evaluation set, thus
elevating Bengali from its low-resource status. To the best of our knowledge,
this is the first ever large scale study on Bengali-English machine
translation. We believe our study will pave the way for future research on
Bengali-English machine translation as well as other low-resource languages.
Our data and code are available at https://github.com/csebuetnlp/banglanmt.
Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translation
http://arxiv.org/abs/2009.09359v2