Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> Sarcasm transforms the polarity of an apparently positive or negative utterance into its opposite. We report on a method for constructing a corpus of sarcastic Twitter messages in which determination of the sarcasm of each message has been made by its author. We use this reliable corpus to compare sarcastic utterances in Twitter to utterances that express positive or negative attitudes without sarcasm. We investigate the impact of lexical and pragmatic factors on machine learning effectiveness for identifying sarcastic utterances and we compare the performance of machine learning techniques and human judges on this task. Perhaps unsurprisingly, neither the human judges nor the machine learning techniques perform very well. <s> BIB001 </s> Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> Traditional theories of sarcasm treat it as a case of a speaker’s meaning the opposite of what she says. Recently, ‘expressivists’ have argued that sarcasm is not a type of speaker meaning at all, but merely the expression of a dissociative attitude toward an evoked thought or perspective. I argue that we should analyze sarcasm in terms of meaning inversion, as the traditional theory does; but that we need to construe ‘meaning’ more broadly, to include illocutionary force and evaluative attitudes as well as propositional content. I distinguish four subclasses of sarcasm, individuated in terms of the target of inversion. Three of these classes raise serious challenges for a standard implicature analysis. The standard view of sarcasm or verbal irony 1 was articulated by Quintilian roughly two millennia ago, as speech in which “we understand something which is the opposite of what is actually said” (95/1920, 401). Indeed, sarcasm is often presented as “the textbook case,” as Robyn Carston (2002, 15) says, of the fact that speaker’s meaning can come apart from sentence meaning. We also have a widely accepted pragmatic explanation of how sarcasm works. According to the standard Gricean reconstruction, in speaking ∗ Thanks for fertile discussion to audiences at the Conference on Meaning and Commu <s> BIB002 </s> Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> The research described in this work focuses on identifying key components for the task of irony detection. By means of analyzing a set of customer reviews, which are considered ironic both in social and mass media, we try to find hints about how to deal with this task from a computational point of view. Our objective is to gather a set of discriminating elements to represent irony, in particular, the kind of irony expressed in such reviews. To this end, we built a freely available data set with ironic reviews collected from Amazon. Such reviews were posted on the basis of an online viral effect; i.e. contents that trigger a chain reaction in people. The findings were assessed employing three classifiers. Initial results are largely positive, and provide valuable insights into the subjective issues of language facing tasks such as sentiment analysis, opinion mining and decision making. <s> BIB003 </s> Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> While a fair amount of work has been done on automatically detecting emotion in human speech, there has been little research on sarcasm detection. Although sarcastic speech acts are inherently subjective, humans have relatively clear intuitions as to what constitutes sarcastic speech. In this paper, we present a system for automatic sarcasm detection. Using a new acted speech corpus that is annotated for sarcastic and sincere speech, we examine a number of features that are indicative of sarcasm. The first set of features looks at a baseline of basic acoustic features that have been found to be helpful in human sarcasm identification. We then present an effective way of modeling and applying prosodic contours to the task of automatic sarcasm detection. This approach applies sequential modeling to categorical representations of pitch and intensity contours obtained via k-means clustering. Using a SimpleLogistic (LogitBoost) classifier, we are able to predict sarcasm with 81.57% accuracy. This result suggests that certain pitch and intensity contours are predictive of sarcastic speech. <s> BIB004 </s> Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> This document describes the senti.ue system and how it was used for participation in SemEval-2014 Task 9 challenge. Our system is an evolution of our prior work, also used in last year’s edition of Sentiment Analysis in Twitter. This system maintains a supervised machine learning approach to classify the tweet overall sentiment, but with a change in the used features and the algorithm. We use a restricted set of 47 features in subtask B and 31 features in subtask A. In the constrained mode, and for the five data sources, senti.ue achieved a score between 78,72 and 84,05 in subtask A, and a score between 55,31 and 71,39 in subtask B. For the unconstrained mode, our score was slightly below, except for one case in subtask A. <s> BIB005 </s> Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> This report summarizes the objectives and evaluation of the SemEval 2015 task on the sentiment analysis of figurative language on Twitter (Task 11). This is the first sentiment analysis task wholly dedicated to analyzing figurative language on Twitter. Specifically, three broad classes of figurative language are considered: irony, sarcasm and metaphor. Gold standard sets of 8000 training tweets and 4000 test tweets were annotated using workers on the crowdsourcing platform CrowdFlower. Participating systems were required to provide a fine-grained sentiment score on an 11-point scale (-5 to +5, including 0 for neutral intent) for each tweet, and systems were evaluated against the gold standard using both a Cosinesimilarity and a Mean-Squared-Error measure. <s> BIB006 </s> Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> Sarcasm is a nuanced form of language in which individuals state the opposite of what is implied. With this intentional ambiguity, sarcasm detection has always been a challenging task, even for humans. Current approaches to automatic sarcasm detection rely primarily on lexical and linguistic cues. This paper aims to address the difficult task of sarcasm detection on Twitter by leveraging behavioral traits intrinsic to users expressing sarcasm. We identify such traits using the user's past tweets. We employ theories from behavioral and psychological studies to construct a behavioral modeling framework tuned for detecting sarcasm. We evaluate our framework and demonstrate its efficiency in identifying sarcastic tweets. <s> BIB007 </s> Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> Sarcasm understanding may require information beyond the text itself, as in the case of ‘I absolutely love this restaurant!’ which may be sarcastic, depending on the contextual situation. We present the first quantitative evidence to show that historical tweets by an author can provide additional context for sarcasm detection. Our sarcasm detection approach uses two components: a contrast-based predictor (that identifies if there is a sentiment contrast within a target tweet), and a historical tweet-based predictor (that identifies if the sentiment expressed towards an entity in the target tweet agrees with sentiment expressed by the author towards that entity in the past). <s> BIB008 </s> Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> Irony has been proven to be pervasive in social media, posing a challenge to sentiment analysis systems. It is a creative linguistic phenomenon where affect-related aspects play a key role. In this work, we address the problem of detecting irony in tweets, casting it as a classification problem. We propose a novel model that explores the use of affective features based on a wide range of lexical resources available for English, reflecting different facets of affect. Classification experiments over different corpora show that affective information helps in distinguishing among ironic and nonironic tweets. Our model outperforms the state of the art in almost all cases. <s> BIB009 </s> Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> This paper is a novel study that views sarcasm detection in dialogue as a sequence labeling task, where a dialogue is made up of a sequence of utterances. We create a manuallylabeled dataset of dialogue from TV series ‘Friends’ annotated with sarcasm. Our goal is to predict sarcasm in each utterance, using sequential nature of a scene. We show performance gain using sequence labeling as compared to classification-based approaches. Our experiments are based on three sets of features, one is derived from information in our dataset, the other two are from past works. Two sequence labeling algorithms (SVM-HMM and SEARN) outperform three classification algorithms (SVM, Naive Bayes) for all these feature sets, with an increase in F-score of around 4%. Our observations highlight the viability of sequence labeling techniques for sarcasm detection of dialogue. <s> BIB010 </s> Automatic Sarcasm Detection: A Survey <s> Shared Tasks & Benchmark Datasets <s> We describe the Sentiment Analysis in Twitter task, ran as part of SemEval-2014. It is a continuation of the last year's task that ran successfully as part of SemEval-2013. As in 2013, this was the most popular SemEval task; a total of 46 teams contributed 27 submissions for subtask A (21 teams) and 50 submissions for subtask B (44 teams). This year, we introduced three new test sets: (i) regular tweets, (ii) sarcastic tweets, and (iii) LiveJournal sentences. We further tested on (iv) 2013 tweets, and (v) 2013 SMS messages. The highest F1-score on (i) was achieved by NRC-Canada at 86.63 for subtask A and by TeamX at 70.96 for subtask B. <s> BIB011
Shared tasks allow comparative evaluation of multiple approaches on a common dataset. Reference BIB011 describes a shared task on sentiment analysis in Twitter, from SemEval-2014. The organizers provided multiple datasets (SMS, LiveJournal, etc.), one of which was a dataset of sarcastic tweets. It is interesting to note that the system at rank 10 gave the best performance for the dataset of sarcastic tweets (82.75, versus 77.13, which was the performance of the system that ranked highest). This best performing system was by Reference BIB005 . They use features that separately capture message and contextual polarity. Reference BIB006 describes a shared task from SemEval-2015 that deals with sentiment analysis of figurative language. The organizers provided a dataset of ironic and metaphorical statements labeled as positive, negative and neutral. The participants were expected to correctly identify the sentiment polarity in the case of figurative expressions like irony. The teams that participated in the shared task used affective resources, character n-grams, and so on. The winning team used "four lexica, one that was automatically generated and three than were manually crafted (sic)." The second shared task was a data science contest organized as a part of PAKDD 2016. BIB002 The competition dataset consisted of Reddit comments labeled as either sarcastic or non-sarcastic. BIB001 shows that unigram-based features outperform the use of a subset of words as derived from a sentiment lexicon. The authors compare the accuracy of the sarcasm classifier with the human ability to detect sarcasm. Reference BIB003 reports sentiment-based features as their top discriminating features. The Logistic Regression classifier in Reference BIB004 results in an accuracy of 81.5%. Reference BIB007 shows that historical features along with flip-based features are the most discriminating features, resulting in an accuracy of 83.46%. These are also the features presented in a rule-based setting by Reference BIB008 . The authors of Reference BIB009 compare their features against several reported works and show that their approach outperforms them with an F-score of 0.82. An analysis of systems based on types of sarcasm is not available in most past work, primarily due to lack of corpora in which different kinds of sarcasm are marked. However, such an analysis is seen in Reference BIB010 with respect to errors made by their system. They look at examples that their approach predicts correctly but a past approach does not. Then, they label a subset of these examples with one of the four sarcasm types. They report these values for the four types given in Reference BIB002 and observe that nearly 71% require context, that is, are illocutionary or like-prefixed.