Evaluation Measures for the SemEval 2016 Task 4 "Sentiment Analysis in Twitter" (Draft: Version 1.13)
This informal document details the evaluation measures that will be used in SemEval2016 Task 4 "Sentiment Analysis in Twitter" a revamped edition of SemEval 2015 Task 10 (Rosenthal et al. 2015) Note: changes that have been made to the present document since Version 1.0 onwards are typeset in blue. Task 4 consists of five subtasks; the evaluation measures that we will use for them will be discussed in Sections 1 to 5. Subtasks B to E conceptually form a "2x2 matrix" (see Table 1) where the rows indicate the goal of the task (classification vs. quantification) and the columns indicate the granularity of the task (two point scale vs. five point scale) Note that, for Subtasks B to E, the dataset is subdivided into a number of "topics" and the subtask needs to be carried out independently for each topic. As a result, each of the evaluation measures described below is "macroaveraged" across the topics, i.e. we compute the measure individually for each topic, and we then average the results across the topics.