We describe a data-driven approach for automatically explaining new, non-standard English expressions in a given sentence, building on a large dataset that includes 15 years of crowdsourced examples from UrbanDictionary.com. Unlike prior studies that focus on matching keywords from a slang dictionary, we investigate the possibility of learning a neural sequence-to-sequence model that generates explanations of unseen non-standard English expressions given context. We propose a dual encoder approach---a word-level encoder learns the representation of context, and a second character-level encoder to learn the hidden representation of the target non-standard expression. Our model can produce reasonable definitions of new non-standard English expressions given their context with certain confidence.
The study of non-standard language is of interests to many researchers in the social media and NLP communities. For example, propose a latent variable model to study the lexical variation of the language on Twitter, where many regional slang words are discovered. Zappavigna identifies the Internet slang as an important component in the discourse of Twitter and social media. provide a contextual analysis of how social media users shorten their messages. Notably, a study on Tweet normalization @cite_11 finds that, even when using a small slang dictionary of 5K words, Slang makes up 12 the NLP community, slang dictionary is widely used in many tasks and applications @cite_10 @cite_2 @cite_4 . However, we argue that using a small, fixed-size dictionary approach may be suboptimal: it suffers from the low coverage problem, and to keep the dictionary up to date, maintaining such dictionary is also expensive and time-consuming. To the best of our knowledge, our work is the first to build a general purpose machine learning model for explaining non-standard English terms, using a large crowdsourced dataset.
@cite_2 @cite_10 @cite_4 @cite_11
2132210327 2152460337 2146867136
We introduce the novel task of determining whether a newswire article is "true" or satirical. We experiment with SVMs, feature scaling, and a number of lexical and semantic feature types, and achieve promising results over the task. We investigate whether wording, stylistic choices, and online behavior can be used to predict the age category of blog authors. Our hypothesis is that significant changes in writing style distinguish pre-social media bloggers from post-social media bloggers. Through experimentation with a range of years, we found that the birth dates of students in college at the time when social media such as AIM, SMS text messaging, MySpace and Facebook first became popular, enable accurate age prediction. We also show that internet writing characteristics are important features for age prediction, but that lexical content is also needed to produce significantly more accurate results. Our best results allow for 81.57 accuracy. Twitter provides access to large volumes of data in real time, but is notoriously noisy, hampering its utility for NLP. In this paper, we target out-of-vocabulary words in short text messages and propose a method for identifying and normalising ill-formed words. Our method uses a classifier to detect ill-formed words, and generates correction candidates based on morphophonemic similarity. Both word similarity and context are then exploited to select the most probable correction candidate for the word. The proposed method doesn't require any annotations, and achieves state-of-the-art performance over an SMS corpus and a novel dataset based on Twitter.