We describe an Arabic-Hebrew parallel corpus of TED talks built upon WIT3, the Web inventory that repurposes the original content of the TED website in a way which is more convenient for MT researchers. The benchmark consists of about 2,000 talks, whose subtitles in Arabic and Hebrew have been accurately aligned and rearranged in sentences, for a total of about 3.5M tokens per language. Talks have been partitioned in train, development and test sets similarly in all respects to the MT tasks of the IWSLT 2016 evaluation campaign. In addition to describing the benchmark, we list the problems encountered in preparing it and the novel methods designed to solve them. Baseline MT results and some measures on sentence length are provided as an extrinsic evaluation of the quality of the benchmark.
To the best of our knowledge, to date the richest collection of publicly available Arabic-Hebrew parallel corpora is part of the OPUS project; opus.lingfil.uu.se in total, it provides more than 110M tokens per language subdivided into 5 corpora, OpenSubtitles2016 being by far the largest. The OpenSubtitles2016 collection @cite_3 opus.lingfil.uu.se OpenSubtitles2016.php provides parallel subtitles of movies and TV programs made available by the Open multilanguage subtitle database. www.opensubtitles.org The size of this corpus makes it outstandingly valuable; nevertheless, the translation of such kind of subtitles is often less literal than in other domains (even TED), likely affecting the accuracy of the fully automatic processing implemented for parallelizing the Arabic and Hebrew subtitles.
@cite_3
2419539795
We present a new major release of the OpenSubtitles collection of parallel corpora. The release is compiled from a large database of movie and TV subtitles and includes a total of 1689 bitexts spanning 2.6 billion sentences across 60 languages. The release also incorporates a number of enhancements in the preprocessing and alignment of the subtitles, such as the automatic correction of OCR errors and the use of meta-data to estimate the quality of each subtitle and score subtitle pairs.