We present an analysis of the performance of machine learning classifiers on discriminating between similar languages and language varieties. We carried out a number of experiments using the results of the two editions of the Discriminating between Similar Languages (DSL) shared task. We investigate the progress made between the two tasks, estimate an upper bound on possible performance using ensemble and oracle combination, and provide learning curves to help us understand which languages are more challenging. A number of difficult sentences are identified and investigated further with human annotation.
In the closed submission track the best performance was obtained by the NRC-CNRC @cite_8 team, which used a two-step classification approach to predict first the language group of the text, and subsequently the language. Both NRC-CNRC @cite_8 and QMUL @cite_6 , ranked 5 th used linear support vector machines (SVM) classifiers with words and characters as features.
@cite_6 @cite_8
2252123100 2251071163
This work is licensed under a Creative Commons Attribution 4.0 International Licence. Page numbers and proceedings footer are added by the organisers. Licence details: http: creativecommons.org licenses by 4.0 We describe the system built by the National Research Council Canada for the ”Discriminating between similar languages” (DSL) shared task. Our system uses various statistical classifiers and makes predictions based on a two-stage process: we first predict the language group, then discriminate between languages or variants within the group. Language groups are predicted using a generative classifier with 99.99 accuracy on the five target groups. Within each group (except English), we use a voting combination of discriminative classifiers trained on a variety of feature spaces, achieving an average accuracy of 95.71 , with per-group accuracy between 90.95 and 100 depending on the group. This approach turns out to reach the best performance among all systems submitted to the open and closed tasks.