Automatic language identification is a natural language processing problem that tries to determine the natural language of a given content. In this paper we present a statistical method for automatic language identification of written text using dictionaries containing stop words and diacritics. We propose different approaches that combine the two dictionaries to accurately determine the language of textual corpora. This method was chosen because stop words and diacritics are very specific to a language, although some languages have some similar words and special characters they are not all common. The languages taken into account were romance languages because they are very similar and usually it is hard to distinguish between them from a computational point of view. We have tested our method using a Twitter corpus and a news article corpus. Both corpora consists of UTF-8 encoded text, so the diacritics could be taken into account, in the case that the text has no diacritics only the stop words are used to determine the language of the text. The experimental results show that the proposed method has an accuracy of over 90 for small texts and over 99.8 for large texts.
used the discrete hidden Markov models (HMMs) with three models for comparison on a data set containing a collection of Web Pages collected over hotel Web Sites in 5 languages. The texts in the tested corpus were not noisy, because this genre of Web Sites presents the information as clean as possible @cite_5 . Their accuracy is between 95 Timothy performed a study based on different classification methods using the nearest neighbor model with three distances, Naive Bayes and support vector machine (SVM) model @cite_3 . Their results show that the nearest neighbor model using is the most suitable for the tested datasets, with accuracy ranging form 87 On short messages, like tweets, used Prediction by Pattern Matching and Logistic Regression with different features to classify languages under the same family of languages (Arabic, Slavic and Indo-Iranian) @cite_1 . Their accuracy is between 97 Hybrid approaches use one or more of these methods. propose different approaches that use term based and characters based methods @cite_4 . Their experiments show that the proposed hybrid methods are quite interesting and present good language identification performances in noisy texts, with accuracy ranging between 72
@cite_5 @cite_4 @cite_1 @cite_3
2151773777 2543512085 2251862917 1951381097
This paper deals with language identification in the domain of web documents. The proposed system is built on hidden Markov models (HMMs) that enable the modeling of character sequences. Furthermore, the use of HMMs provides the means for language tracking, that is, language identification across the segments of a multilingual document. This paper deals with the problem of automatic language identification of noisy texts, which represents an important task in natural language processing. Actually, there exist several works in this field, which are based on statistical and machine learning approaches for different categories of texts. Unfortunately, most of the proposed methods work fine on clean texts and or long texts, but often present a failure when the text is corrupted or too short. In this research work, we use a typical dataset consisting of short texts collected from several discussion forums containing several types of noises. Our dataset contains 32 different languages, where we notice that some languages are quite different while some others are too closed. In this investigation, we propose two types of methods to identify the text language: term-based method and character-based method. Moreover, we propose two hybrid methods to enhance the performances of those techniques. Experiments show that the proposed hybrid methods are quite interesting and present good language identification performances in noisy texts. Social media services such as Twitter offer an immense volume of real-world linguistic data. We explore the use of Twitter to obtain authentic user-generated text in low-resource languages such as Nepali, Urdu, and Ukrainian. Automatic language identification (LID) can be used to extract language-specific data from Twitter, but it is unclear how well LID performs on short, informal texts in low-resource languages. We address this question by annotating and releasing a large collection of tweets in nine languages, focusing on confusable languages using the Cyrillic, Arabic, and Devanagari scripts. This is the first publicly-available collection of LID-annotated tweets in non-Latin scripts, and should become a standard evaluation set for LID systems. We also advance the state-of-the-art by evaluating new, highly-accurate LID systems, trained both on our new corpus and on standard materials only. Both types of systems achieve a huge performance improvement over the existing state-of-the-art, correctly classifying around 98 of our gold standard tweets. We provide a detailed analysis showing how the accuracy of our systems vary along certain dimensions, such as the tweet-length and the amount of in- and out-of-domain training data. Language identification is the task of identifying the language a given document is written in. This paper describes a detailed examination of what models perform best under different conditions, based on experiments across three separate datasets and a range of tokenisation strategies. We demonstrate that the task becomes increasingly difficult as we increase the number of languages, reduce the amount of training data and reduce the length of documents. We also show that it is possible to perform language identification without having to perform explicit character encoding detection.