Vanilla Classifiers for Distinguishing between Similar Languages

In this paper we describe the submission of the UniBuc-NLP team for the Discriminating between Similar Languages Shared Task, DSL 2016. We present and analyze the results we obtained in the closed track of sub-task 1 (Similar languages and language varieties) and sub-task 2 (Arabic dialects). For sub-task 1 we used a logistic regression classifier with tf-idf feature weighting and for sub-task 2 a character-based string kernel with an SVM classifier. Our results show that good accuracy scores can be obtained with limited feature and model engineering. While certain limitations are to be acknowledged, our approach worked surprisingly well for out-of-domain, social media data, with 0.898 accuracy (3rd place) for dataset B1 and 0.838 accuracy (4th place) for dataset B2.


Introduction
Automatic language identification is the task of determining the language in which a piece of text is written using computational methods. In today's context of multilingualism, and given the rapid development of the online repositories of cross-language information, language identification is an essential task for many downstream applications (such as cross-language information retrieval or question answering), to route the documents to the appropriate NLP systems, based on their language.
Although language identification has been intensively studied in the recent period, there are still questions to be answered. Language identification is still a challenging research problem for very similar languages and language varieties, for very short pieces of text, such as tweets, or for documents involving code-switching (the practice of mixing more languages within a single communication). The DSL 2016 shared task (Malmasi et al., 2016) tackles two interesting aspects of language identification: similar language and language varieties (with in-domain and out-of-domain -social media datatest sets) and Arabic dialects. In this paper we present the submission of the UniBuc-NLP team for the closed track (using only the training data provided by the organizers) of both sub-tasks.

Related Work
Most approaches to language identification are based on character n-grams. Dunning (1994) was one of the very first who used them. He proposed a statistical method for language identification based on Markov models to compute the likelihood of the character n-grams. Ever since, character n-grams have been employed to discriminate between a wide variety of closely related languages and dialects. Maier and Gómez-Rodríguez (2014) performed language classification on tweets for Spanish varieties, with character n-grams as features and using the country of the speaker to identify the variety. Trieschnigg et al. (2012) discriminated between Dutch dialects (and several other languages) using a large collection of folktales. They compared several approaches to language identification and reported good results when using the method of Cavnar and Trenkle (1994), based on character n-grams. Sadat et al. (2014) performed language identification on Arabic dialects using social media texts. They obtained better results with Naive Bayes and n-gram features (2-grams) than with a character n-gram Markov model for most of the Arabic dialects. Gottron and Lipka (2010) conducted a comparative experiment of classification methods for language identification in short texts, discriminating between languages from various language families and using n-gram features. Their results show that Naive Bayes classifier performs best and that errors occur for languages from the same family, reinforcing the hypothesis that language identification is more difficult for very similar languages.
Word n-grams have also proven effective for discriminating between languages and language varieties. Malmasi and Dras (2015) achieved the best performance in the closed track of the DSL 2015 shared task, experimenting with classifier ensembles trained on character and word n-gram features. Goutte and Leger (2015) obtained a very good performance in the same competition using statistical classifiers and employing a combination of character and word n-grams as features. Zampieri and Gebre (2012) made use of a character n-gram model and a word n-gram language model to discriminate between two varieties of Portuguese. They reported the highest accuracy when using character 4-grams and reached the conclusion that orthographic and lexical differences between the two varieties have more discriminative power than lexico-syntactic differences.

Data
The organizers released two training datasets for the 2016 DSL shared task: a dataset of similar languages and language varieties (for sub-task 1) and a dataset of Arabic dialects (for sub-task 2).
The dataset for sub-task 1 is a new version of the DSL Corpus Collection (Tan et al., 2014). It contains instances written in the following languages and language varieties (organized by groups of similarity):

Language
Lang . Table 1: Statistics for the dataset of similar languages and language varieties (sub-task 1).
The dataset consists of 20,000 instances (18,000 for training and 2,000 for development) in each language or language variety, extracted from journalistic texts. In Table 1 we report several statistics for this dataset. The average sentence length varies from 26.01 (for Malay) to 45.06 (for the Spanish variety used in Spain). We observe a high variance for the average sentence length within some of the language groups (the difference between the average sentence length of Indonesian and Malay is ∼8, and between the average sentence length of the Spanish variety spoken in Spain and the one spoken in Mexico is ∼14). The average word length varies from 4.69 (for both versions of French) to 5.91 (for Malay), with a low variance within groups.
Comparing these statistics with those extracted from the sub-task 1 test sets, we notice that while the average sentence length values for test set A are similar to those of the training set, for test sets B1 and B2 -social media data -sentences are significantly shorter, as expected, ranging from an average of 11.33 for Portuguese (Brazil) to an average of 13.39 for Serbian. The average word length values for B1 and B2 are also smaller than those for test set A and the training set, but the differences are not as significant as the differences regarding the length of the sentences. The dataset for sub-task 2 contains automatic speech recognition transcripts  written in the following Arabic dialects: Egyptian, Gulf, Levantine, North-African, and Modern Standard Arabic. In Table 2 we report several statistics for this dataset. The average sentence length ranges from 35.41 (for North-African) to 60.57 (for Egyptian). All the Arabic dialects have the average word length lower than 4.

Our Approach
In this section we describe and analyze the methods we used for discriminating between similar languages, language varieties and dialects. We used standard linear classifiers with basic n-grams features. 1

Classifiers
Logistic Regression For sub-task 1 we used a logistic regression classifier with word unigrams and bigrams as features. The features are tf-idf (Salton and Buckley, 1988) weighted and we keep only the features that occur at least 3 times in the training set. We use the L 2 distance for term vectors and the default regularization constant C = 1 without performing any grid search for best parameters. We use the wrapper of the scikit learn Python library (Pedregosa et al., 2011) over the Liblinear logistic regression implementation (Fan et al., 2008). The main advantages of this model are its simplicity and training speed.

SVM + String Kernel
On the Arabic dataset we decided to use string a kernel based on character n-grams, since the text is obtained through ASR systems and most certainly the transcript contains errors. Character n-grams are able to cover sub-parts of words and can theoretically increase the overall classification accuracy, especially in a language recognition task. We used a string kernel in combination with a support vector machine classifier. A kernel function can be used either to embed the data in a higher dimensional space to achieve linear separability, or to replace the dot product between vectors with values that are more appropriate for the data used. Previous studies on text classification revealed that character n-gram-based string kernels can be effective tools for authorship attribution, native language identification or plagiarism detection (Grozea and Popescu, 2010). The kernel we propose is computed by summing the number of common character n-grams between two examples, where n varies between 2 and 7. Formally, given an alphabet A, we define the mapping function Φ n : D → {0, 1} Qn for an example e ∈ C in the corpus to be the vector of all the binary values of existence of the n-gram g in the document: The function φ g (e) = 1 if the n-gram g is in the example e and equal to zero otherwise. Computationally, Q n depends on all the possible character n-grams between two examples at certain instance.
The corresponding Gram matrix K of size |C| × |C| has the following elements: The Gram matrix is then normalized to the [0, 1] interval: The kernel function, in our case, is computed between every pair of training and testing examples. This type of approach is less scalable for large amounts of data, which is the main reason for not applying this technique on sub-task 1. However, additional optimizations can be taken into consideration, such as using just the upper half of the symmetric Gram matrix, aggregating multiple kernels trained on subsamples of the data or hashing techniques for faster computation. In our vanilla approach we did not make use of any of these techniques.
In practice, kernel methods over strings for text classification work remarkably well covering fine-grained similarities such as content, punctuation marks, affixes etc., however one important downside of this method is usually the lack of linguistic features available within the classifier, making almost impossible to analyze from the Gram matrix the actual features that lead to good or bad results.

Experiments
Using the experimental setup previously described, we developed several systems for discriminating between similar languages and language varieties (sub-task 1) and between Arabic dialects (sub-task 2).
The organizers provided three test datasets, two for sub-task 1 and one for sub-task 2. In Table 3 we provide a brief characterization of the datasets:

Dataset Description
Task # instances A In-domain: newspaper texts Sub-task 1 12,000 B1 Out of domain: social media data Sub-task 1 500 B2 Out of domain: social media data Sub-task 1 500 C ASR texts from Arabic dialects Sub-task 2 1,540

Sub-task 1
Our two runs for sub-task 1 are as follows: • Run 1: a one-level system. The first system consists of a single logistic regression classifier that predicts the language or language variety.
• Run 2: a two-level system. The second system consists of multiple logistic regression classifiers: we train a classifier to predict the language group ("inter-group classifier"), and one classifier for each language group ("intra-group classifier"), to predict the language or language variety within the group.
For the one-level system we obtained 0.8441 accuracy when evaluating on the development dataset. For the two-level system we obtained 0.9972 accuracy for the inter-group classifier, and the following values for the intra-group classifiers: 0.7510 for es, 0.8940 for fr, 0.9207 for pt, 0.7848 for bs-hr-sr, 0.9820 for id-my.   In Table 4 we report the results that we obtained for the test datasets. Our best results for each dataset are as follows: 0.8648 accuracy (11 th place) for dataset A, 0.8980 accuracy (3 rd place) for dataset B1 and 0.8380 accuracy (4 th place) for dataset B2. For two of the three datasets (A, B2), the two-level system obtained better results than the one-level system. However, our highest accuracy (0.8990) was obtained by the one-level system for dataset B1.  Table 6: The most informative features for the one-level system for sub-task 1.
In Tables 6 and 7 we report the most informative features for each class. With few exceptions, most of the informative features are unigrams. While for the language classifiers many of these features are named entities (such as references to geographical regions or names of persons), as expected, for the language group classifier (Table 7a) the situation is different: mostly very short words prove to have high discriminative power. Among others, we identified definite and indefinite articles -"los" (es), "le" (fr)and functional words -"nao" (pt), "dalam" (id-my) -ranked among the most informative features.
Despite the fact that quite many of the top features are named entities, which could a suggest a topic bias in classification, our systems obtain a good performance on out-of-domain data, ranking 3 rd and 4 th on the social media datasets. Both our systems outperform significantly a random baseline that obtains 0.0883 F1 score for dataset A and 0.20 for datasets B1 and B2.
In Table 5 we report the results that we obtained for the test dataset. As expected, the accuracy of the system increases as the range of n-grams becomes wider. Our best result for sub-task 2 is 0.4753 accuracy (8 th place). In Figures 1, 2 and 3 we render the confusion matrices for the classification of the Arabic dialects. We observe a different behavior for the five classes, along the three runs: for EGY and LAV, the number of correctly classified instances is very similar over the three runs. For GLF there is a slight increase in correctly classified instances at run 2. For MSA the increase is significant (from 92 in run 1 to 190 -more than double -in run 2), and for NOR there is a certain decrease (from 180 in run 1 to 145 in run 2).

Conclusions
In this paper we described the submission of the UniBuc-NLP team for the DSL 2016 shared task. We participated in the closed track of both sub-tasks (sub-task 1: similar languages and language varieties, sub-task 2: Arabic dialects), submitting a total of 5 runs (2 for sub-task 1 and 3 for sub-task 2). We used linear classification methods based on word and character n-gram features. For sub-task 1 we used a logistic regressions classifier with tf-idf feature weighting and for sub-task 2 an SVM classifier with a string kernel. Our best system obtains 89.80% accuracy for sub-task 1, dataset B1 (3 rd place). Our results suggest that relatively good results may be obtained with plain vanilla linear classifiers, with no hyper-parameter optimization or special feature selection. When compared to other competitors in the shared task, our logistic regression results were at most 0.03% lower compared to the top score on sub-task 1, dataset A and among the top scoring for the datasets B1 and B2. On the Arabic dataset, the kernel method stands 0.04% from the first position and while additional parameters can improve the model, we believe the dataset created using ASR had a great impact on the results. To conclude, plain vanilla methods can be good enough to distinguish between similar languages, however we are still a long way from claiming this task solved and clearly more research is needed in this direction to create robust models that capture linguistic variation.