Human evaluators were asked to evaluate on a scale from 1 to 5 the validity of the lexicon annotations made by the experts and crowd contributors.
ABSTRACTIVE
1000 term groups based on the number of total annotations(200 term groups with 2 total annotations, 200 term groups with 3 total annotations, and so on up to term groups with 6 total annotations)
EXTRACTIVE
We perform a direct comparison of expert and crowd contributors, for 1000 term groups based on the number of total annotations(200 term groups with 2 total annotations, 200 term groups with 3 total annotations, and so on up to term groups with 6 total annotations). The experts are two Ph.D. linguists, while the crowd is made up of random high quality contributors that choose to participate in the task. As a reference, the cost of hiring two experts is equal to the cost of employing nineteen contributors in Crowdflower. Evaluators were given a summary of the annotations received for the term group in the form of:The term group "inequality inequity" received annotations as 50.0% sadness, 33.33% disgust, 16.67% anger. Then, they were asked to evaluate on a scale from 1 to 5, how valid these annotations were considered.
We perform a direct comparison of expert and crowd contributors, for 1000 term groups based on the number of total annotations(200 term groups with 2 total annotations, 200 term groups with 3 total annotations, and so on up to term groups with 6 total annotations). The experts are two Ph.D. linguists, while the crowd is made up of random high quality contributors that choose to participate in the task. As a reference, the cost of hiring two experts is equal to the cost of employing nineteen contributors in Crowdflower.
1710.04203
Introduction </s> Sent iment Ġanalysis Ġaims Ġto Ġuncover Ġthe Ġemotion Ġconveyed Ġthrough Ġinformation . ĠIn Ġonline Ġsocial Ġnetworks , Ġsentiment Ġanalysis Ġis Ġmainly Ġperformed Ġfor Ġpolitical Ġand Ġmarketing Ġpurposes , Ġproduct Ġacceptance Ġand Ġfeedback Ġsystems . ĠThis Ġinvolves Ġthe Ġanalysis Ġof Ġvarious Ġsocial Ġmedia Ġinformation Ġtypes , Ġsuch Ġas Ġtext ĠB IB REF 0 Ġ, Ġemot icons Ġand Ġhasht ags , Ġor Ġmultimedia ĠB IB REF 1 Ġ. ĠHowever , Ġto Ġperform Ġsentiment Ġanalysis , Ġinformation Ġhas Ġto Ġbe Ġlabelled Ġwith Ġa Ġsentiment . ĠThis Ġrelationship Ġis Ġdefined Ġin Ġa Ġlex icon . </s> Lex icon Ġacquisition Ġis Ġa Ġrequirement Ġfor Ġsentiment Ġclassification . ĠDuring Ġthe Ġacquisition Ġprocess , Ġindividual Ġor Ġgrouped Ġinformation Ġelements Ġare Ġlabelled Ġbased Ġon Ġa Ġclass , Ġusually Ġan Ġemotion . ĠSent iment Ġclassification Ġis Ġthe Ġtask Ġthat Ġuses Ġthe Ġacquired Ġlex icon Ġand Ġa Ġclassification Ġmethod Ġto Ġclassify Ġa Ġsentence , Ġphrase , Ġor Ġsocial Ġmedia Ġsubmission Ġas Ġa Ġwhole , Ġbased Ġon Ġthe Ġaggregation Ġof Ġits Ġlabels . ĠThus , Ġlex icon Ġquality Ġdirectly Ġaffects Ġsentiment Ġclassification Ġaccuracy . </s> Both Ġtasks Ġcan Ġeither Ġbe Ġperformed Ġautomatically ĠB IB REF 2 Ġor Ġmanually ĠB IB REF 3 Ġwhere Ġthe Ġlab elling Ġby Ġlingu ists Ġor Ġresearchers Ġthemselves ĠB IB REF 4 Ġ. ĠApart Ġfrom Ġexperts , Ġmanual Ġlab bel ing Ġcan Ġalso Ġbe Ġperformed Ġwith Ġthe Ġhelp Ġof Ġa Ġwide Ġnetwork Ġof Ġpeople , Ġknown Ġas Ġcrowds ourcing ĠB IB REF 5 Ġ. ĠCrowd s ourcing Ġis Ġwidely Ġused Ġfor Ġpol arity Ġlex icons , Ġbut Ġrarely Ġfor Ġbeyond Ġpol arity Ġand Ġnever Ġfor Ġthe Ġdiscovery Ġof Ġlinguistic Ġelements . </s> Sent iment Ġanalysis Ġis Ġcommonly Ġperformed Ġin Ġpol arity Ġbasis , Ġi . e . Ġthe Ġdistinction Ġbetween Ġpositive Ġand Ġnegative Ġemotion Ġ. ĠThese Ġpoles Ġcorrespond Ġto Ġagreement Ġand Ġdisagreement , Ġor Ġacceptance Ġand Ġdisapproval , Ġfor Ġcandidates Ġand Ġproducts Ġrep sect ively ĠB IB REF 6 Ġ. </s> Beyond Ġpol arity Ġ( also Ġknown Ġas Ġpure Ġemotion ) Ġsentiment Ġanalysis Ġaims Ġto Ġuncover Ġan Ġexact Ġemotion , Ġbased Ġon Ġemotional Ġtheories ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠApplications Ġsuch Ġas Ġsentiment Ġtracking , Ġmarketing , Ġtext Ġcorrection , Ġand Ġtext Ġto Ġspeech Ġsystems Ġcan Ġbe Ġimproved Ġwith Ġthe Ġuse Ġof Ġdistinct Ġemotion Ġlex icons . </s> However , Ġbeyond Ġpol arity Ġstudies Ġacquire Ġlex icons Ġbased Ġon Ġa Ġset Ġof Ġstrict Ġrules , Ġand Ġthe Ġevaluation Ġof Ġexperts . ĠThese Ġlex icons Ġuse Ġonly Ġa Ġsingle Ġemotion Ġper Ġterm ĠB IB REF 9 Ġ. ĠThe Ġproblems Ġof Ġthese Ġapproaches Ġis Ġthe Ġlack Ġof Ġuniform ity Ġand Ġcontribution Ġfreedom Ġwhen Ġrelying Ġon Ġgold Ġstandards , Ġand Ġhigh Ġcosts Ġwith Ġlow Ġscal ability Ġwhen Ġemploying Ġexperts . ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġapplications Ġthat Ġonly Ġrely Ġon Ġexperts Ġare Ġless Ġcomprehensive , Ġrestricted , Ġand Ġnot Ġscalable , Ġcompared Ġto Ġcrowds ourced ĠN LP Ġapplications ĠB IB REF 10 Ġ. </s> This Ġpaper Ġpresents Ġour Ġapproach Ġfor Ġthe Ġacquisition Ġof Ġa Ġmultic lass Ġand Ġscalable Ġcrowds ourced Ġpure Ġemotion Ġlex icon Ġ( P EL ), Ġbased Ġon ĠPl utch ik 's Ġeight Ġbasic Ġemotions . ĠFurthermore , Ġthe Ġcrowd Ġis Ġalso Ġresponsible Ġfor Ġidentifying Ġlinguistic Ġelements , Ġnamely Ġintens ifiers , Ġneg ators , Ġand Ġstop Ġwords . ĠUsually Ġthese Ġelements Ġare Ġpooled Ġfrom Ġexisting Ġlists ĠB IB REF 11 Ġcreated Ġby Ġexperts . ĠWe Ġalso Ġintroduce Ġa Ġworker Ġfiltering Ġmethod Ġto Ġidentify Ġand Ġexclude Ġdishonest Ġor Ġspam ming Ġcontributors , Ġthat Ġdoesn 't Ġrequire Ġgold Ġstandards . ĠOur Ġgoal Ġis Ġto Ġmaintain Ġan Ġend Ġto Ġend Ġautomated Ġwork - flow Ġfor Ġa Ġcrowds ourced Ġ( ann otation Ġand Ġevaluation Ġwise ) Ġlex icon Ġacquisition Ġprocess . ĠTherefore , Ġto Ġhighlight Ġcrowd 's Ġperformance Ġon Ġevaluation , Ġwe Ġcompare Ġevaluations Ġfrom Ġlinguistic Ġexperts Ġand Ġthe Ġcrowd Ġitself . </s> Related ĠWork </s> According Ġto ĠB IB REF 12 Ġ, Ġan Ġemotion Ġis Ġdefined Ġwith Ġreference Ġto Ġa Ġlist . ĠEk am Ġet Ġal . ĠB IB REF 8 Ġproposed Ġthe Ġsix Ġbasic Ġemotions Ġjoy , Ġanger , Ġfear , Ġsadness , Ġdisgust , Ġand Ġsurprise . ĠYears Ġlater , ĠPl utch ik ĠB IB REF 7 Ġproposed Ġthe Ġaddition Ġof Ġtrust Ġand Ġanticipation Ġas Ġbasic Ġemotions , Ġand Ġpresented Ġa Ġcirc ump lex Ġmodel Ġof Ġemotions Ġas Ġseen Ġin ĠFigure ĠFIG REF 1 Ġ, Ġwhich Ġdefines Ġemotional Ġcontradictions Ġand Ġsome Ġof Ġthe Ġpossible Ġcombinations . </s> Sent iment Ġanalysis Ġaims Ġto Ġclassify Ġinformation Ġbased Ġon Ġthe Ġemotion Ġconveyed . ĠDepending Ġon Ġthe Ġnumber Ġof Ġclasses / em otions Ġrequired , Ġwe Ġcan Ġseparate Ġthe Ġanalysis Ġinto : Ġpol arity Ġand Ġbeyond Ġpol arity . </s> P olar ity Ġsentiment Ġanalysis Ġstudies Ġdefine Ġtwo Ġopposite Ġemotional Ġstates , Ġpositive Ġand Ġnegative , Ġor Ġgood Ġand Ġbad , Ġwith Ġthe Ġaddition Ġof Ġa Ġneutral Ġstate . ĠFurthermore , Ġsome Ġresearchers Ġhave Ġclassified Ġinformation Ġon Ġlevels Ġfor Ġeach Ġpole ( e . g . Ġvery Ġpositive , Ġpositive , Ġneutral , Ġnegative , Ġvery Ġnegative Ġetc .), Ġalso Ġknown Ġas Ġfine Ġgr ained Ġsentiment Ġanalysis ĠB IB REF 13 Ġ. </s> Beyond Ġpol arity , Ġalso Ġknown Ġas Ġpure Ġemotion , Ġsentiment Ġanalysis Ġis Ġa Ġmore Ġrefined Ġapproach Ġto Ġthe Ġsame Ġproblem Ġwith Ġa Ġwider Ġrange Ġof Ġpossible Ġemotion Ġclasses , Ġsee ĠFigure ĠFIG REF 1 Ġ. ĠEssentially , Ġany Ġsentiment Ġanalysis Ġthat Ġinvolves Ġspecific Ġemotional Ġlab elling , Ġis Ġconsidered Ġas Ġa Ġbeyond Ġpol arity Ġanalysis . ĠExamples Ġof Ġemotional Ġlabels Ġmight Ġbe Ġ- but Ġare Ġnot Ġlimited Ġto - : Ġsadness , Ġboredom , Ġjoy , Ġsadness , Ġsurprise , Ġanger , Ġfear , Ġdisgust Ġetc . </s> As Ġdiscussed Ġin ĠSection Ġ1 , Ġone Ġof Ġthe Ġcore Ġtasks Ġof Ġsentiment Ġanalysis Ġis Ġlex icon Ġacquisition . ĠA Ġlex icon Ġcan Ġbe Ġacquired Ġthrough Ġmanual Ġor Ġautomatic Ġannotation . ĠHowever , Ġnatural Ġlanguage Ġhas Ġa Ġvery Ġsubjective Ġnature ĠB IB REF 14 Ġwhich Ġsignificantly Ġinhibits Ġautomated Ġsentiment Ġlex icon Ġa q cu isition Ġmethods Ġfrom Ġachieving Ġrelevance Ġequal Ġto Ġmanual Ġmethods ĠB IB REF 15 Ġ. ĠThus Ġa Ġlot Ġof Ġresearchers Ġchoose Ġto Ġmanually Ġannot ate Ġtheir Ġterm Ġcorpor a ĠB IB REF 16 Ġ, Ġor Ġuse Ġestablished Ġlex icon Ġsuch Ġas ĠWord Net , ĠSent i Word Net , Ġand Ġvarious Ġother Ġlex icons ĠB IB REF 13 Ġ. ĠOther Ġstudies Ġcombine Ġmanual Ġlabeling Ġor Ġmachine Ġlearning Ġwith Ġlex icons ĠB IB REF 17 Ġ. </s> Man ual Ġlex icon Ġacquisition Ġis Ġconstrained Ġby Ġthe Ġnumber Ġof Ġpeople Ġcontributing Ġto Ġthe Ġtask , Ġand Ġthe Ġnumber Ġof Ġannotations Ġfrom Ġeach Ġparticipant . ĠThese Ġconstraints Ġcan Ġbe Ġeliminated Ġby Ġincreasing Ġthe Ġnumber Ġof Ġpeople Ġinvolved , Ġfor Ġinstance , Ġby Ġusing Ġcrowds ourcing ĠB IB REF 18 Ġ. ĠAmazon 's ĠMechanical ĠTurk Ġ( MT ur k ) Ġis Ġa Ġcrowds ourcing Ġplatform Ġfrequently Ġused Ġfor Ġpol arity Ġsentiment Ġlex icon Ġacquisition Ġvia Ġcrowds ourcing ĠB IB REF 19 Ġ. ĠM Tur k Ġis Ġalso Ġused , Ġfor Ġthe Ġannotation Ġof Ġone Ġthousand Ġtweets Ġin ĠB IB REF 20 Ġ, Ġten Ġthousand Ġterms Ġin ĠB IB REF 21 Ġwith Ġgold Ġstandards , Ġand Ġthe Ġannotation Ġof Ġninety Ġfive Ġemot icons Ġout Ġof Ġone Ġthousand Ġtotal Ġemot icons Ġfound Ġin ĠB IB REF 22 Ġ. ĠWhile ĠB IB REF 23 Ġhad Ġone Ġthousand Ġfour Ġhundred Ġterms Ġlabelled Ġwith Ġa Ġsupervised Ġmachine Ġlearning Ġand Ġcrowd Ġvalid ators . ĠThe Ġchallenge Ġis Ġto Ġintroduce Ġa Ġwork - flow Ġthat Ġis Ġscalable , Ġun super vised Ġand Ġapplicable Ġto Ġdifferent Ġinformation Ġtypes . </s> The Ġsecond Ġcore Ġpart Ġin Ġsentiment Ġanalysis , Ġis Ġsentiment Ġclassification . ĠA Ġclassification Ġthat Ġoccurs Ġat Ġphrase / sent ence / sub mission Ġlevel , Ġand Ġis Ġusually Ġbased Ġon Ġthe Ġaggregation Ġof Ġthe Ġterm 's Ġlabeled Ġemotions . ĠAs Ġwith Ġlex icon Ġa q cu isition , Ġthe Ġclassification Ġtask Ġcan Ġbe Ġautomated ĠB IB REF 13 Ġor Ġperformed Ġmanually ĠB IB REF 24 Ġ. </s> Regardless Ġof Ġmanual Ġor Ġautomated Ġsentiment Ġclassification , Ġon Ġtextual Ġinformation Ġscenarios , Ġterm Ġand Ġphrase Ġsentiment Ġis Ġthe Ġmain Ġinput Ġof Ġthe Ġclassification Ġmethod . ĠIn Ġsome Ġcases Ġthe Ġdecision Ġmight Ġbe Ġtotally Ġdifferent Ġfrom Ġthe Ġindividual Ġterm Ġemotion , Ġleading Ġto Ġrel abel ing Ġof Ġthe Ġterms Ġthemselves ĠB IB REF 25 Ġ. ĠMan ually Ġlabelled Ġclassification Ġcan Ġachieve Ġhigh Ġrelevance , Ġbut Ġit Ġrequires Ġadditional Ġresources , Ġand Ġis Ġnot Ġeasily Ġscalable . ĠOn Ġthe Ġother Ġhand , Ġautomated Ġprocesses Ġare Ġscalable Ġbut Ġwith Ġlower Ġrelevance ĠB IB REF 24 Ġ. </s> Our Ġapproach </s> Our Ġaim Ġis Ġto Ġcreate Ġan Ġend Ġto Ġend Ġautomated Ġwork - flow Ġfor Ġthe Ġcreation , Ġevaluation Ġand Ġenrichment Ġof Ġa Ġpure Ġemotion Ġlex icon . ĠThe Ġwork - flow , ĠFigure ĠFIG REF 3 Ġ, Ġcan Ġbe Ġseparated Ġin Ġtwo Ġmain Ġcomponents . ĠPre - processing Ġis Ġthe Ġun super vised Ġprocess Ġby Ġwhich Ġwe Ġderive Ġthe Ġlex icon Ġterms Ġfrom Ġany Ġtextual Ġresource , Ġwhile Ġcrowds ourcing Ġdeals Ġwith Ġthe Ġcrowds ourcing Ġaspect Ġof Ġthe Ġlex icon . ĠThe ĠPure ĠEm otions ĠLex icon Ġincludes Ġemotional Ġterm Ġgroups , Ġintens ifiers Ġand Ġneg ators , Ġand Ġstop Ġwords . </s> Pre - processing Ġis Ġcomprised Ġof Ġ3 Ġun super vised Ġsteps , Ġtoken ization , Ġstemming Ġand Ġspell Ġcheck . ĠText ual Ġcontent Ġis Ġtoken ized Ġas Ġun i - gram s , Ġstemmed Ġbased Ġon Ġtheir Ġrooted Ġand Ġchecked Ġfor Ġspelling . ĠThe Ġresulting Ġstems Ġalong Ġwith Ġtheir Ġstem Ġgroups Ġare Ġstored Ġin Ġa Ġlex icon Ġdatabase . ĠCrowd s ourcing Ġis Ġusing Ġthe Ġlex icon Ġdatabase Ġand Ġthe Ġcrowd Ġto Ġannot ate Ġeach Ġentry Ġin Ġthe Ġdatabase . ĠParticipants Ġsubmit Ġtheir Ġanswers Ġthat Ġgo Ġthrough Ġa Ġfiltering Ġprocess . ĠIf Ġthe Ġanswers Ġare Ġconsidered Ġvalid , Ġthey Ġupdate Ġthe Ġlex icon Ġentries . ĠThe Ġcrowd Ġalso Ġevaluates Ġexisting Ġannotations , Ġto Ġdetermine Ġthe Ġlex icon Ġquality . ĠAs Ġcrowd Ġevaluation Ġmethods Ġare Ġnew Ġin Ġlex icon Ġacquisition Ġtasks , Ġwe Ġcompare Ġcrowd Ġevaluations Ġto Ġthose Ġof Ġexpert Ġlingu ists . </s> Data </s> During ĠJanuary Ġ2017 , Ġwe Ġperformed Ġa Ġkeyword Ġbased Ġcrawl Ġfor Ġarticles Ġand Ġcomments Ġin Ġthe ĠEurope Ġsubreddit Ġand Ġtweets Ġin ĠTwitter , Ġwhich Ġcontained Ġthe Ġword Ġ" Brexit ". ĠThe Ġuse Ġof Ġa Ġpolitical Ġand Ġcontroversial Ġterm Ġin Ġthe Ġquery Ġis Ġdeliberate , Ġto Ġcapture Ġthe Ġemotional Ġdiversity Ġof Ġa Ġpolitically Ġoriented Ġcorpus . </s> We Ġcrawled Ġone Ġhundred Ġarticles Ġfrom ĠReddit , Ġwith Ġmore Ġthan Ġforty Ġthousand Ġcomments Ġand Ġmore Ġthan Ġthree Ġthousand Ġtweets . ĠFor Ġthe ĠReddit Ġdata , Ġwe Ġcollected Ġinformation Ġon Ġlocation , Ġtime , Ġand Ġthe Ġnumber Ġof Ġup votes . ĠFor Ġthe ĠTwitter Ġdata , Ġwe Ġstored Ġthe Ġnumber Ġor Ġre - t we ets Ġand Ġfavourites , Ġtime Ġand Ġlocation Ġinformation . </s> Our Ġfocus Ġis Ġon Ġsingle Ġterm Ġ( also Ġknown Ġas Ġun ig ram ) Ġsentiment , Ġthus Ġposts Ġin Ġboth Ġnetworks Ġwere Ġprocessed Ġto Ġa Ġsingle Ġterm Ġlist . ĠIn Ġtotal , Ġthe Ġnumber Ġof Ġunique Ġterms Ġin Ġour Ġcorpus Ġwas Ġ30 227 . ĠBased Ġon Ġthe Ġenchant Ġpython Ġlibrary , Ġused Ġin ĠB IB REF 26 Ġ, Ġand Ġthe Ġsupported ĠGreat ĠBritish ĠEnglish Ġdictionary , Ġ19 193 Ġwere Ġvalidated Ġand Ġ110 34 Ġwere Ġinvalid ated . ĠOur Ġanalysis Ġwill Ġfocus Ġon Ġthe Ġ19 193 Ġvalid Ġterms , Ġthat Ġfollow ĠZip f 's ĠLaw Ġwith Ġscaling - law Ġcoefficient ĠIN LINE FORM 0 Ġis Ġa Ġgood Ġfit . </s> After Ġvalidation , Ġterms Ġwere Ġstemmed Ġwith ĠPorter ĠSt em ming ĠAl gorithm ĠB IB REF 27 Ġ. ĠSt em ming Ġidentified Ġ10 953 Ġdistinct Ġterm Ġgroups Ġwith Ġone Ġor Ġmore Ġterms . ĠStop - words , Ġintens ifiers Ġand Ġneg ators Ġare Ġalso Ġincluded Ġin Ġthe Ġvalid Ġterm Ġgroups . ĠBoth Ġterm Ġvalidation Ġand Ġstemming Ġare Ġun super vised , Ġsince Ġour Ġgoal Ġis Ġto Ġmaintain Ġscal ability Ġin Ġbeyond Ġpol arity Ġlex icon Ġacquisition Ġand Ġsentiment Ġclassification . </s> C rowd s ourcing </s> The Ġcrowds ourcing Ġtask , Ġhosted Ġin ĠCrowd Fl ower , Ġrequired Ġcontributors Ġto Ġlabel Ġterm Ġgroups Ġin Ġthree Ġdifferent Ġmain Ġclasses , Ġemotion , Ġintens ifier Ġand Ġnone , Ġwithout Ġa Ġgolden Ġstandard , Ġrules Ġor Ġany Ġparticipation Ġrestrictions Ġ. ĠEm otion Ġlab elling Ġincluded Ġthe Ġ8 Ġbasic Ġemotions Ġas Ġdefined Ġby ĠPl utch ik . ĠInt ens ifier Ġclass Ġincluded Ġintens ifiers Ġand Ġneg ators . ĠFinally , Ġnone Ġreferred Ġto Ġstop - words Ġor Ġwords Ġwith Ġno Ġparticular Ġemotion . </s> Each Ġof Ġthe Ġeleven Ġoptions Ġfor Ġthe Ġmain Ġclasses , Ġwill Ġbe Ġreferred Ġto Ġas Ġ" sub class ". ĠTerms Ġare Ġgrouped Ġbased Ġon Ġtheir Ġstem . ĠEach Ġterm Ġgroup Ġhas Ġa Ġmain Ġannotation Ġclass Ġdefined Ġby Ġmajority , Ġand Ġseveral Ġsub Ġannotation Ġclasses , Ġdefined Ġby Ġthe Ġnon Ġmajority Ġannotations . ĠHowever , Ġto Ġaid Ġmulti Ġclass Ġanalysis Ġof Ġthe Ġresults , Ġevery Ġannotation Ġis Ġlogged Ġin Ġthe Ġlex icon Ġdatabase . </s> Task ĠInterface </s> The Ġtask Ġinterface Ġwas Ġthe Ġresult Ġof Ġseveral Ġexperiments . ĠThree Ġmajor Ġchanges Ġimplemented Ġbased Ġon Ġthese Ġexperimental Ġinterfaces Ġwere : Ġthe Ġsimpl ification Ġof Ġthe Ġtask Ġquestion , ĠFigure ĠFIG REF 8 Ġ, Ġthe Ġinclusion Ġof Ġonly Ġthree Ġmain Ġclasses , Ġand Ġthe Ġreplacement Ġof Ġwords Ġpositive Ġand Ġnegative , Ġwith Ġampl ifying Ġand Ġweakening Ġin Ġthe Ġintens ifying Ġclass Ġoptions . ĠAll Ġexperiments Ġand Ġthe Ġannotation Ġtask Ġrequired Ġhighly Ġexperienced Ġcontributors , Ġas Ġdefined Ġby ĠCrowd flower Ġplatform . </s> As Ġseen Ġin ĠFigure ĠFIG REF 8 Ġcontributors Ġselect Ġone Ġof Ġthe Ġthree Ġchoices . ĠIf Ġthey Ġchoose ĠEm otion Ġev oking , Ġthey Ġare Ġpresented Ġwith Ġa Ġdrop - down Ġmenu Ġto Ġchoose Ġfrom Ġthe Ġeight Ġbasic Ġemotions . ĠSimilarly , Ġif Ġthey Ġselect ĠInt ens ifying Ġcontext Ġthey Ġhave Ġto Ġspecify Ġwhether Ġit Ġwas ĠAmpl ifying Ġor ĠWe akening Ġthe Ġcontext , Ġessentially Ġannot ating Ġthe Ġintens ifiers Ġand Ġneg ators . ĠFinally Ġif Ġthey Ġchoose ĠNone Ġthey Ġare Ġpresented Ġwith Ġthe Ġnext Ġterm Ġgroup . ĠTo Ġassist Ġcontributors Ġwith Ġterm Ġdefinitions , Ġevery Ġterm Ġgroup Ġhad Ġa Ġhyper link Ġto Ġan ĠEnglish Ġdictionary . </s> C rowd </s> More Ġthan Ġone Ġhundred Ġeighty Ġcontributors Ġperformed Ġeighty Ġthousand Ġannotations . ĠBy Ġdesign , Ġeach Ġuser Ġcould Ġnot Ġperform Ġmore Ġthan Ġ660 Ġunique Ġannotations , Ġexcluding Ġthe Ġassessment Ġquestions , Ġto Ġengage Ġat Ġleast Ġ100 Ġcontributors . ĠMost Ġof Ġthe Ġworkers Ġannot ated Ġthe Ġmaximum Ġallowed Ġterms , Ġthe Ġfirst Ġhalf Ġof Ġworkers Ġannot ated Ġ15 % Ġof Ġthe Ġterm Ġgroups Ġin Ġour Ġcorpus , Ġwhile Ġthe Ġsecond Ġhalf Ġof Ġworkers Ġannot ate Ġthe Ġrest Ġ85 %. ĠThe Ġsimplicity Ġof Ġthe Ġtask Ġresulted Ġin Ġhigh Ġoverall Ġworker Ġengagement , Ġwith Ġmean Ġand Ġmedian Ġannotations Ġper Ġworker , Ġat Ġ429 Ġand Ġ580 Ġrespectively . </s> Ass essment </s> Based Ġon Ġa Ġset Ġof Ġexperiments , Ġwe Ġidentified Ġ136 Ġterm Ġgroups Ġthat Ġwould Ġtest Ġthe Ġability Ġof Ġa Ġcontributor Ġin Ġall Ġof Ġthe Ġthree Ġmain Ġclasses , Ġemotion Ġev oking , Ġintens ifying Ġcontext , Ġand Ġnone . ĠAs Ġthe Ġassessment Ġterm Ġgroups Ġhad Ġmore Ġthan Ġten Ġthousand Ġannotations , Ġwe Ġanalyse Ġit Ġseparately Ġfrom Ġthe Ġlex icon . </s> In Ġorder Ġfor Ġa Ġworker Ġto Ġgain Ġthe Ġability Ġto Ġcontribute Ġto Ġthe Ġcrowds ourcing Ġtask Ġand Ġeventually Ġget Ġpaid , Ġhe / she Ġhad Ġto Ġproperly Ġannot ate Ġ80 % Ġof Ġthe Ġassessment Ġterm Ġgroups Ġencountered . ĠThe Ġannotations Ġshould Ġbe Ġwithin Ġthe Ġdominant Ġclasses , Ġand Ġnot Ġsub classes , Ġas Ġdefined Ġfrom Ġthe Ġassessment Ġannot ators . ĠE . g ., Ġfor Ġan Ġassessment Ġterm Ġgroup Ġthat Ġreceived Ġ100 Ġannotations Ġin Ġvarious Ġemotions , Ġwe Ġcheck Ġif Ġthe Ġworker Ġannot ates Ġthe Ġterm Ġgroup Ġas Ġemotion Ġev oking . </s> Let ĠIN LINE FORM 0 Ġbe Ġthe Ġset Ġof Ġworkers ĠIN LINE FORM 1 Ġand Ġand ĠIN LINE FORM 2 Ġthe Ġset Ġof Ġeleven ĠIN LINE FORM 3 Ġsub classes : Ġeight Ġemotions Ġ, Ġtwo Ġintens ifiers , Ġand Ġnone Ġof Ġthe Ġformer . ĠWe Ġdefine ĠIN LINE FORM 4 Ġas Ġthe Ġnumber Ġof Ġtotal Ġannotations Ġfor Ġeach Ġworker ĠIN LINE FORM 5 Ġ. ĠThen : ĠDIS PLAY FORM 0 </s> We Ġdefine ĠIN LINE FORM 0 Ġbe Ġthe Ġset Ġof Ġworkers ĠIN LINE FORM 1 Ġin Ġthe Ġassessment Ġprocess , ĠIN LINE FORM 2 Ġthe Ġset Ġof Ġworkers ĠIN LINE FORM 3 Ġin Ġthe Ġacquisition Ġprocess . ĠThen , Ġfor ĠIN LINE FORM 4 Ġwe Ġdefine : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> and : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> The Ġoptimal ĠIN LINE FORM 0 Ġis Ġfound Ġfor ĠIN LINE FORM 1 Ġ. ĠFor Ġthis Ġstudy , Ġthe Ġoptimal Ġfiltering Ġpercentage Ġwas Ġfound Ġat Ġ40 %, ĠIN LINE FORM 2 Ġ. </s> Work ers Ġfrom ĠIndia Ġand ĠVenezuela , Ġwho Ġcontributed Ġ92 % Ġof Ġthe Ġtask , Ġhave Ġannot ated Ġmore Ġthan Ġ30 % Ġof Ġthe Ġterm Ġgroups Ġwith Ġjoy . ĠHowever , Ġannotations Ġfrom Ġcountries Ġwith Ġmore Ġthan Ġ300 Ġannotations , Ġdon 't Ġfollow Ġthe Ġsame Ġdistribution . ĠSpecifically , Ġworkers Ġfrom ĠPhilippines , ĠUnited ĠStates , ĠColombia , ĠPoland , ĠUnited ĠKingdom , ĠRussia , Ġand ĠEgypt , Ġperformed Ġa Ġsmoother Ġdistributed Ġemotion Ġannotation . ĠIn Ġcomparison , Ġthe Ġmost Ġannot ated Ġemotion Ġin ĠB IB REF 21 Ġwas Ġfear Ġin Ġ18 % Ġof Ġthe Ġtotal Ġterms . </s> By Ġfurther Ġanalys ing Ġworker Ġannotation Ġdistribution , Ġwe Ġidentified Ġworkers Ġthat Ġhad Ġa Ġsignificant Ġfraction Ġof Ġtheir Ġtotal Ġannotations Ġin Ġa Ġsingle Ġsubclass . ĠE . g . Ġone Ġspecific Ġworker Ġannot ated Ġ99 % Ġof Ġthe Ġassessment Ġterm Ġgroups Ġhe Ġencountered Ġas Ġjoy . ĠDish on esty Ġor Ġspam ming Ġis Ġa Ġknown Ġproblem Ġin Ġcrowds ourcing ĠB IB REF 28 Ġand Ġmultiple Ġproposed Ġsolutions Ġexist ĠB IB REF 28 Ġ, Ġbut Ġthey Ġrequire Ġgold Ġstandards Ġor Ġobjective Ġcrowds ourcing Ġtasks . </s> As Ġwe Ġdon 't Ġhave Ġa Ġgold Ġstandard , Ġand Ġthe Ġtask Ġis Ġmore Ġsubjective , Ġthese Ġspam ming Ġelimination Ġmethods Ġare Ġnot Ġapplicable . ĠOur Ġsolution Ġis Ġthe Ġimplementation Ġof Ġa Ġfast Ġand Ġefficient Ġfilter , Ġwhich Ġonly Ġrelies Ġon Ġthe Ġobtained Ġannotations Ġand Ġthe Ġassessment . ĠIf Ġworkers ' Ġanswers Ġwere Ġabove Ġa Ġcertain Ġpercentage Ġon Ġa Ġsingle Ġsubclass Ġfor Ġboth Ġthe Ġassessment Ġand Ġthe Ġannotation Ġprocess , Ġthen Ġthe Ġuser Ġwould Ġbe Ġflagged Ġas Ġdishonest Ġand Ġthe Ġtotal Ġof Ġtheir Ġannotations Ġwould Ġbe Ġdiscarded . ĠThis Ġrule Ġwas Ġapplied Ġto Ġthe Ġwhole Ġrange Ġof Ġpossible Ġanswers , Ġincluding Ġthe Ġ8 Ġemotions , Ġ2 Ġintens ifiers Ġand Ġ" none ". </s> Prior Ġto Ġimplementing Ġthe Ġfilter , Ġwe Ġanalysed Ġhow Ġit Ġwould Ġaffect Ġthe Ġnumber Ġof Ġeligible , Ġnot Ġconsidered Ġas Ġspam ming , Ġworkers . ĠThe Ġthick Ġline Ġin ĠFigure ĠFIG REF 22 Ġshows Ġthe Ġpercentage Ġduring Ġthe Ġassessment Ġterm Ġgroups , Ġand Ġthe Ġdotted Ġline Ġshows Ġthe Ġpercentage Ġduring Ġthe Ġlex icon Ġacquisition . </s> The Ġhigher Ġthe Ġsingle Ġannotation Ġpercentage , Ġthe Ġhigher Ġthe Ġcertainty Ġof Ġspam ming Ġbehaviour . ĠIn Ġlarge Ġpart , Ġthe Ġexclusion Ġrate Ġwas Ġsimilar Ġfor Ġboth Ġthe Ġassessment Ġand Ġthe Ġlex icon Ġannotations . ĠA Ġnumber Ġof Ġworkers Ġhad Ġa Ġmore Ġcautious Ġbehaviour Ġin Ġthe Ġtest Ġquestions , Ġresulting Ġin Ġreduced Ġpercentage Ġof Ġexclus ions Ġduring Ġassessment Ġprocess . ĠThis Ġbehaviour Ġis Ġjustified , Ġas Ġthe Ġfirst Ġset Ġof Ġquestions Ġencounter Ġby Ġa Ġworker Ġare Ġknowingly Ġa Ġset Ġof Ġassessment Ġquestions . </s> Each Ġassessment Ġterm Ġgroup Ġwas Ġannot ated Ġmore Ġthan Ġ120 Ġtimes , Ġby Ġ187 Ġannot ators . ĠThis Ġis Ġa Ġcritical Ġmass Ġof Ġcontributors Ġand Ġprovides Ġvaluable Ġfindings Ġwith Ġregards Ġto Ġtask . ĠThese Ġare : </s> Work ers Ġrarely Ġclicked Ġthe Ġinformative Ġdictionary Ġlink . ĠAs Ġa Ġresult , Ġthey Ġwould Ġannot ate Ġemotional Ġwords Ġto Ġnone , Ġprobably Ġdue Ġto Ġmisinterpret ation . ĠWe Ġavoided Ġthe Ġdirect Ġinclusion Ġof Ġthe Ġdictionary Ġdefinition , Ġas Ġit Ġcould Ġbe Ġconsidered Ġas Ġa Ġform Ġof Ġleverage . ĠE . g . Ġ" ve hement , Ġhal cy on " Ġ, Ġtwo Ġ- un common - Ġemotion Ġb aring Ġwords , Ġwere Ġboth Ġannot ated Ġas Ġnone , Ġand Ġless Ġthan Ġ0 . 2 % Ġof Ġthe Ġworkers Ġ( on Ġaverage ) Ġclicked Ġa Ġdictionary Ġlink . </s> The Ġconcept Ġof Ġintens ifiers Ġis Ġunderstandable Ġbut Ġrequires Ġcritical Ġmass ĠB IB REF 29 Ġ. ĠA Ġsmall Ġnumber Ġof Ġannot ators Ġwould Ġinitially Ġannot ate Ġintens ifiers / neg ators Ġwith Ġan Ġemotion , Ġbut Ġthe Ġdistribution Ġwould Ġslowly Ġshift Ġtowards Ġthe Ġcorrect Ġclass . ĠE . g . Ġ" red uce , Ġlittle , Ġplethora " Ġ, Ġwere Ġinitially Ġannot ated Ġas Ġsad Ġsad Ġjoy , Ġbut Ġafter Ġtens Ġof Ġannotations Ġthey Ġwere Ġannot ated Ġas Ġweakening Ġweakening Ġintens ifying . </s> All Ġwords Ġshould Ġbe Ġevaluated , Ġeven Ġthose Ġthat Ġseemingly Ġdon 't Ġcarry Ġa Ġspecific Ġemotion . ĠAs Ġtimes Ġchange , Ġwords Ġand Ġemotions Ġacquire Ġnew Ġlinks . ĠE . g . Ġ" anti , Ġserious " Ġ, Ġwere Ġboth Ġannot ated Ġas Ġfear Ġev oking Ġwith Ġa Ġgreat Ġemotional Ġdiversity . </s> Lex icon ĠAnalysis </s> The Ġlex icon Ġ( will Ġbe Ġreferred Ġas Ġsimply Ġ" P EL ") Ġis Ġcreated Ġafter Ġthe Ġexclusion Ġof Ġannotations Ġfollowing Ġthe Ġ40 % Ġsingle Ġannotation Ġfiltering Ġcheck . ĠWe Ġreceived Ġmore Ġthan Ġseventy Ġthousands Ġannotations Ġfor Ġ10 593 Ġterm Ġgroups , Ġof Ġthose Ġonly Ġ22 Ġthousand Ġannotations Ġfor Ġ97 37 Ġterm Ġgroups Ġare Ġincluded Ġin Ġthe Ġfinal Ġlex icon , Ġas Ġa Ġresult Ġof Ġfiltering . ĠEach Ġterm Ġgroup Ġhad Ġa Ġmean Ġ2 . 3 Ġannotations Ġfrom Ġa Ġtotal Ġof Ġ95 Ġdifferent Ġannot ators . ĠAlthough Ġthe Ġnumber Ġof Ġmean Ġannotations Ġin Ġlex icon Ġis Ġless Ġthan Ġhalf Ġthe Ġmean Ġannotations Ġin Ġthe Ġunf il tered Ġcorpus , Ġthe ĠP EL Ġannotations Ġare Ġconsidered Ġof Ġhigher Ġquality . </s> Each Ġlex icon Ġterm Ġgroup Ġhas Ġmultiple Ġsubclass Ġannotations , Ġand Ġthe Ġmain Ġsubclass Ġis Ġdefined Ġby Ġmajority . ĠEven Ġafter Ġfiltering , Ġthe Ġdominant Ġemotion Ġin Ġour Ġlex icon Ġis Ġjoy , Ġwhile Ġthe Ġleast Ġannot ated Ġemotion Ġis Ġdisgust . ĠAdditionally , Ġ148 Ġterms Ġwere Ġannot ated Ġas Ġintens ifiers , Ġ43 Ġterms Ġas Ġneg ators , Ġand Ġ68 01 Ġterms Ġas Ġnone . ĠA Ġsample Ġof Ġfive Ġterms Ġfor Ġeach Ġof Ġthe Ġthree Ġsub classes Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 27 Ġ. ĠThe Ġfull Ġlex icon Ġcan Ġbe Ġfound Ġon Ġgithub . </s> Int ens ifiers Ġand Ġneg ators Ġserve Ġas Ġmodifiers Ġto Ġthe Ġemotional Ġcontext Ġof Ġa Ġword . ĠWorkers Ġidentified Ġmostly Ġvalid Ġintens ifiers Ġand Ġneg ators Ġthat Ġcan Ġmodify Ġemotion Ġev oking Ġwords , Ġin Ġthe Ġabsence Ġof Ġcontext . ĠJudging Ġfrom Ġthe Ġreceived Ġannotations , Ġthere Ġis Ġroom Ġfor Ġimprovement Ġon Ġthe Ġdescription Ġof Ġthe Ġintens ifier Ġclass Ġand Ġthe Ġprovided Ġexamples , Ġas Ġa Ġnumber Ġof Ġnon Ġintens ifying Ġwords Ġwere Ġfalsely Ġannot ated . </s> Ter ms Ġin Ġour Ġlex icon Ġare Ġgrouped Ġbased Ġon Ġtheir Ġstem . ĠSt em ming Ġsignificantly Ġreduced Ġcost Ġ( by Ġhalf ) Ġand Ġtime - required Ġfor Ġthe Ġtask . ĠGroup ing Ġterms Ġmay Ġcreate Ġunnecessary Ġmulti - class Ġannotations Ġagreements , Ġfor Ġterms Ġin Ġthe Ġsame Ġterm Ġgroup Ġthat Ġmight Ġhave Ġdifferent Ġmeanings . ĠAn notation Ġagreement Ġrefers Ġto Ġequal Ġnumber Ġof Ġannotations Ġin Ġmultiple Ġsub classes Ġor Ġemotions . ĠHowever , Ġthe Ġvast Ġmajority Ġof Ġterm Ġgroups Ġin Ġour Ġlex icon , Ġdon 't Ġdisplay Ġany Ġform Ġof Ġcontradict ing Ġannotation . ĠCont rad ict ing Ġemotions Ġare Ġportrayed Ġin Ġopposite Ġedges Ġof Ġthe Ġcirc ump lex ĠFigure ĠFIG REF 1 Ġ, Ġwhile Ġemotional Ġcombinations Ġare Ġdec ribed Ġin ĠB IB REF 7 Ġ. ĠIn Ġthe Ġlex icon , Ġonly Ġ21 % Ġand Ġ20 % Ġof Ġthe Ġterm Ġgroups Ġhad Ġa Ġsubclass Ġand Ġan Ġemotional Ġagreement Ġrespectively . ĠWith Ġregards Ġto Ġemotion , Ġcontradict ing Ġor Ġmulti - em otion Ġagreement , Ġcould Ġbe Ġobserved Ġin Ġonly Ġ8 . 6 % Ġof Ġthe Ġtotal Ġterm Ġgroups . </s> Let ĠIN LINE FORM 0 Ġbe Ġa Ġterm Ġgroup Ġin Ġthe Ġlex icon Ġand ĠIN LINE FORM 1 Ġthe Ġset Ġof Ġeleven ĠIN LINE FORM 2 Ġsub classes : Ġeight Ġemotions Ġ, Ġtwo Ġintens ifiers , Ġand Ġnone Ġof Ġthe Ġformer . ĠWe Ġdefine ĠIN LINE FORM 3 Ġas Ġthe Ġnumber Ġof Ġannotations Ġfor Ġeach Ġterm Ġgroup ĠIN LINE FORM 4 Ġ. ĠFor Ġeach ĠIN LINE FORM 5 Ġ, Ġthe Ġannotations Ġfor Ġemotion Ġsub classes Ġare ĠIN LINE FORM 6 Ġ, Ġthe Ġannotations Ġfor Ġintens ifying Ġsub classes Ġare ĠIN LINE FORM 7 Ġ, Ġand Ġthe Ġnumber Ġof Ġnone Ġannotations Ġis ĠIN LINE FORM 8 Ġ. </s> Therefore , Ġeach ĠIN LINE FORM 0 Ġcan Ġhave Ġan Ġmon oton ically Ġincreasing Ġfinite Ġsequence ĠIN LINE FORM 1 Ġwith ĠIN LINE FORM 2 Ġ, Ġwhere : ĠDIS PLAY FORM 0 </s> We Ġsay Ġthat Ġterm Ġgroup ĠIN LINE FORM 0 Ġhas Ġsubclass Ġagreement Ġif Ġand Ġonly Ġif : ĠDIS PLAY FORM 0 </s> While ĠIN LINE FORM 0 Ġhas Ġemotional Ġagreement Ġif Ġand Ġonly Ġif Ġthere Ġis Ġa Ġsubclass Ġagreement Ġwith Ġthe Ġsequence ĠIN LINE FORM 1 Ġand : ĠDIS PLAY FORM 0 </s> Sub class Ġagreement , Ġrefers Ġto Ġequal Ġannotations , Ġbetween Ġemotional Ġsubclass ( es ) Ġand Ġat Ġleast Ġone Ġnon - em otional Ġsubclass , Ġor Ġbetween Ġmultiple Ġnon - em otional Ġsubclass , ĠEqu ation ĠEQ REF 30 Ġ. ĠOn Ġthe Ġother Ġhand , Ġemotional Ġagreement Ġrefers Ġto Ġmultiple Ġemotion Ġsub classes Ġwith Ġequal Ġannotations , ĠEqu ation ĠEQ REF 31 Ġ. </s> The Ġnumber Ġof Ġsub classes Ġin Ġagreement Ġand Ġthe Ġnumber Ġof Ġterms Ġin Ġa Ġterm Ġgroup Ġare Ġnegatively Ġcorrelated . ĠTerm Ġgroups Ġwith Ġtwo Ġterms Ġappear Ġto Ġhave Ġthe Ġhighest Ġsubclass Ġagreement Ġwith Ġexactly Ġtwo Ġsub classes . ĠThe Ġmost Ġcommon Ġoccurring Ġagreements Ġare Ġsubclass Ġnone Ġpaired Ġwith Ġan Ġemotion , Ġand Ġjoy Ġpaired Ġwith Ġan Ġemotion . ĠThe Ġnumber Ġof Ġmulti - class Ġagreement Ġoccurrences Ġis Ġdispro port ional Ġto Ġthe Ġnumber Ġof Ġterms Ġin Ġa Ġterm Ġgroup . ĠThis Ġis Ġa Ġstrong Ġindication Ġthat Ġstemming Ġdidn 't Ġconfuse Ġworkers . </s> Similarly , Ġfor Ġemotional Ġagreement , Ġthe Ġnumber Ġof Ġoccurrences Ġis Ġdisproportionate Ġto Ġthe Ġnumber Ġof Ġterms Ġin Ġthe Ġterm Ġgroup . ĠFurthermore , Ġemotional Ġagreement Ġappeared Ġin Ġ10 % Ġof Ġthe Ġterm Ġgroups , Ġwhile Ġsubclass Ġagreement Ġwas Ġfound Ġin Ġ20 % Ġof Ġthe Ġterm Ġgroups . ĠIn Ġthe Ġagreement Ġannotations , Ġjoy Ġis Ġthe Ġmost Ġcommon Ġemotion . ĠAccording Ġto ĠPl utch ik 's Ġcirc ump lex ĠFigure ĠFIG REF 1 Ġ, Ġeach Ġemotion Ġhas Ġa Ġcontradict ing Ġone , Ġand Ġpairs Ġof Ġemotions Ġindicate Ġa Ġmore Ġ" complex " Ġemotion . ĠThere Ġare Ġ6 97 Ġemotional Ġagreeing Ġterm Ġgroups , Ġof Ġ14 34 Ġterms , Ġwith Ġexactly Ġtwo Ġemotions . ĠThese Ġemotional Ġdy ads ĠB IB REF 7 Ġcan Ġbe Ġcombined Ġas Ġseen Ġin ĠTable ĠT AB REF 32 Ġ. ĠSimple Ġbasic Ġemotion Ġannotation Ġtasks Ġcan Ġindirectly Ġprovide Ġcomplex Ġemotional Ġannotations . </s> D y adic Ġemotional Ġagreements Ġcould Ġbe Ġinterpreted Ġas Ġthe Ġresulting Ġcomplex Ġemotion , Ġor Ġfurther Ġannot ated Ġto Ġobtain Ġa Ġsingle Ġdominant Ġemotion . ĠThere Ġwas Ġa Ġnumber Ġof Ġterm Ġgroups Ġwith Ġopposite Ġemotion Ġdy ads , Ġpresented Ġin ĠTable ĠT AB REF 33 Ġ, but Ġas Ġthe Ġnumber Ġof Ġannotations Ġincreases , Ġemotional Ġagreement Ġoccurrences Ġ- comb ination Ġor Ġopposition - Ġdecreases . </s> In Ġtotal , Ġthe Ġlex icon Ġfeatures Ġ177 40 Ġannot ated Ġterms Ġwith Ġ3 Ġclasses Ġand Ġ11 Ġsub classes . The Ġdominant Ġclass Ġfor Ġ70 30 Ġterms Ġwas Ġemotion , Ġ191 Ġintens ifying , Ġ68 01 Ġnone , Ġand Ġ37 18 Ġin Ġsome Ġform Ġof Ġsubclass Ġagreement . ĠLex icon Ġterms Ġare Ġmainly Ġjoy Ġannot ated , Ġand Ġemotional Ġagreement Ġis Ġprevalent Ġin Ġ10 % Ġof Ġthe Ġterms . ĠOnly Ġ21 % Ġof Ġtotal Ġterms Ġhave Ġa Ġsubclass Ġagreement . </s> Rel iability </s> Single Ġannotation Ġreliability Ġagreement Ġis Ġthe Ġdegree Ġof Ġagreement Ġbetween Ġannot ators , Ġfor Ġterm Ġgroups Ġthat Ġhave Ġannotation Ġmajority Ġin Ġexactly Ġone Ġsub Ġclass . ĠIn Ġour Ġlex icon , Ġsingle Ġannotation Ġreliability Ġagreement Ġwas Ġlow , Ġmainly Ġdue Ġto Ġthe Ġlow Ġnumber Ġof Ġannot ators Ġfor Ġeach Ġterm Ġgroup Ġin Ġrelation Ġto Ġthe Ġhigh Ġnumber Ġof Ġpossible Ġcategories . </s> Based Ġon ĠFle iss ĠKappa ĠB IB REF 30 Ġ( sim ply Ġreferred Ġas Ġk ), Ġand Ġas Ġseen Ġin ĠTable ĠT AB REF 35 Ġ, Ġterm Ġgroups Ġwith Ġ2 Ġannotations Ġhad Ġthe Ġlowest Ġreliability Ġagreement , Ġwhile Ġterm Ġgroups Ġwith Ġ6 Ġannotations Ġthe Ġhighest Ġreliability Ġagreement . ĠAs Ġthe Ġnumber Ġof Ġannot ators Ġrises , Ġthe Ġnumber Ġof Ġpossible Ġagreement Ġperm utations Ġincreases Ġbut Ġthe Ġnumber Ġof Ġmajor Ġannot ated Ġsub classes Ġdecreases . ĠMore Ġannot ators Ġhave Ġa Ġpositive Ġeffect Ġin Ġboth Ġk Ġand Ġcertainty Ġof Ġclassification . </s> As Ġwe Ġrestrict Ġour Ġlex icon Ġto Ġemotions , Ġreliability Ġincreases Ġfor Ġany Ġnumber Ġof Ġannot ators Ġexcept Ġtwo . ĠThis Ġis Ġexplained Ġby Ġthe Ġdecrease Ġin Ġthe Ġnumber Ġof Ġpossible Ġcategories . ĠWhen Ġwe Ġrestrict Ġour Ġanalysis Ġon Ġemotion Ġrelated Ġannotation Ġthe Ġprobability Ġfor Ġagreement Ġin Ġannotations Ġincreases , Ġresulting Ġin Ġa Ġhigh Ġemotional Ġk . ĠThe Ġbest Ġway Ġto Ġincrease Ġk Ġis Ġto Ġprovide Ġadditional Ġannotations Ġthat Ġwill Ġeventually Ġconverge Ġto Ġa Ġmajority Ġclass Ġor Ġa Ġlimited Ġgroup Ġof Ġclasses . </s> C rowd Ġand Ġexperts Ġcomparison </s> We Ġperform Ġa Ġdirect Ġcomparison Ġof Ġexpert Ġand Ġcrowd Ġcontributors , Ġfor Ġ1000 Ġterm Ġgroups Ġbased Ġon Ġthe Ġnumber Ġof Ġtotal Ġannotations ( 200 Ġterm Ġgroups Ġwith Ġ2 Ġtotal Ġannotations , Ġ200 Ġterm Ġgroups Ġwith Ġ3 Ġtotal Ġannotations , Ġand Ġso Ġon Ġup Ġto Ġterm Ġgroups Ġwith Ġ6 Ġtotal Ġannotations ). ĠThe Ġexperts Ġare Ġtwo ĠPh . D . Ġlingu ists , Ġwhile Ġthe Ġcrowd Ġis Ġmade Ġup Ġof Ġrandom Ġhigh Ġquality Ġcontributors Ġthat Ġchoose Ġto Ġparticipate Ġin Ġthe Ġtask . ĠAs Ġa Ġreference , Ġthe Ġcost Ġof Ġhiring Ġtwo Ġexperts Ġis Ġequal Ġto Ġthe Ġcost Ġof Ġemploying Ġnineteen Ġcontributors Ġin ĠCrowd flower . </s> E val u ators Ġwere Ġgiven Ġa Ġsummary Ġof Ġthe Ġannotations Ġreceived Ġfor Ġthe Ġterm Ġgroup Ġin Ġthe Ġform Ġof : The Ġterm Ġgroup Ġ" ine quality Ġinequ ity " Ġreceived Ġannotations Ġas Ġ50 . 0 % Ġsadness , Ġ33 . 33 % Ġdisgust , Ġ16 . 67 % Ġanger . ĠThen , Ġthey Ġwere Ġasked Ġto Ġevaluate Ġon Ġa Ġscale Ġfrom Ġ1 Ġto Ġ5 , Ġhow Ġvalid Ġthese Ġannotations Ġwere Ġconsidered . </s> The Ġsummary Ġof Ġthe Ġevaluation Ġfor Ġboth Ġexperts Ġand Ġcrowd Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 36 Ġ. ĠThe Ġfirst Ġgraph Ġpresents Ġthe Ġvalidity Ġover Ġthe Ġnumber Ġof Ġannotations Ġin Ġthe Ġmain Ġclass Ġof Ġthe Ġterm Ġgroup . ĠAlthough Ġthis Ġinformation Ġis Ġhidden Ġfrom Ġthe Ġeval u ators , Ġhigh Ġannot ational Ġagreement Ġresults Ġin Ġhigh Ġevaluation Ġscores . ĠBoth Ġexperts Ġand Ġthe Ġcrowd Ġfollow Ġthat Ġpositive Ġtrend . ĠCrowd Ġcontributors Ġare Ġmore Ġstrict Ġin Ġtheir Ġevaluations , Ġbut Ġafter Ġfour Ġannotations Ġwe Ġobserve Ġa Ġsignificant Ġvalidity Ġincrease Ġon Ġboth Ġcrowd Ġand Ġexperts . </s> Likewise , Ġthe Ġannotation Ġpercentage Ġfor Ġthe Ġmajority Ġclass Ġhas Ġa Ġpositive Ġinfluence Ġto Ġthe Ġevaluation Ġscore , Ġwith Ġthe Ġexception Ġof Ġ100 % Ġagreement , Ġsecond Ġgraph ĠFigure ĠFIG REF 36 Ġ. ĠThe Ġweighing Ġfactor Ġfor Ġterm Ġgroups Ġwith Ġ100 % Ġannotation Ġagreement Ġis Ġthe Ġreduced Ġnumber Ġof Ġtotal Ġannotations , Ġas Ġthe Ġmean Ġnumber Ġof Ġtotal Ġannotations Ġdrops Ġabruptly Ġon Ġthe Ġ100 %, Ġand Ġtotal Ġagreement Ġis Ġmore Ġfrequent Ġin Ġterm Ġgroups Ġwith Ġlow Ġnumber Ġof Ġtotal Ġannotations . ĠIt 's Ġworth Ġnoting Ġthat Ġcertain Ġpercentages Ġcan Ġonly Ġoccur Ġon Ġspecific Ġnumber Ġof Ġtotal Ġannotations , Ġe . g . Ġ17 % Ġand Ġ83 % Ġcan Ġonly Ġoccur Ġwhen Ġthe Ġnumber Ġof Ġtotal Ġannotations Ġis Ġsix . </s> In Ġemotion Ġannotations , Ġas Ġseen Ġon Ġthe Ġthird Ġgraph Ġof ĠFigure ĠFIG REF 36 Ġcrowd Ġand Ġexperts Ġfollow Ġa Ġsimilar Ġevaluation Ġpattern . ĠAnt icip ation Ġand Ġjoy Ġhad Ġthe Ġexact Ġsame Ġevaluation , Ġwhile Ġevery Ġother Ġemotion Ġand Ġstop Ġwords Ġwere Ġevaluated Ġlower Ġfrom Ġthe Ġcrowd . ĠThe Ġonly Ġsub classes Ġevaluated Ġhigher Ġfrom Ġthe Ġcrowd Ġwere Ġintens ifiers Ġand Ġneg ators , Ġwith Ġa Ġsignificant Ġdifference Ġin Ġthe Ġevaluations Ġfor Ġthe Ġlatter . ĠSection Ġ6 . 3 Ġprovides Ġa Ġmore Ġdetailed Ġevaluation Ġfor Ġterm Ġgroups Ġthat Ġreceived Ġat Ġleast Ġone Ġannotation Ġas Ġintens ifiers Ġor Ġneg ators . </s> The Ġfinal Ġgraph Ġin ĠFigure ĠFIG REF 36 Ġpresents Ġa Ġclear Ġnegative Ġcorrelation Ġof Ġsubclass Ġagreement Ġand Ġevaluation Ġscores . ĠThe Ġhighest Ġnumber Ġof Ġsub classes Ġthat Ġdo Ġnot Ġaffect Ġevaluation Ġscores Ġis Ġthree , Ġabove Ġthat Ġthere Ġis Ġa Ġsteady Ġdecline Ġof Ġthe Ġevaluation Ġscores , Ġfor Ġboth Ġthe Ġcrowd Ġand Ġthe Ġexperts . </s> The Ġevaluation Ġresults Ġprovide Ġsome Ġkey Ġinsights Ġin Ġthe Ġimportance Ġof Ġthe Ġnumber Ġof Ġannotations . ĠThe Ġevaluation Ġscores Ġstart Ġto Ġimprove Ġafter Ġfour Ġannotations . ĠAn not ational Ġagreement Ġand Ġmajority Ġvoting Ġare Ġless Ġimportant . ĠSub class Ġagreement Ġhas Ġa Ġnegative Ġeffect Ġon Ġthree Ġor Ġmore Ġsub classes . ĠMost Ġimportantly Ġand Ġcompared Ġto Ġexperts , Ġthe Ġcrowd Ġis Ġa Ġstricter Ġeval u ator Ġwith Ġsignificantly Ġlower Ġcosts , Ġand Ġhigher Ġscal ability . ĠSince Ġstrict Ġevaluation Ġleads Ġto Ġhigher Ġquality Ġannotations , Ġthe Ġevaluation Ġcan Ġbe Ġperformed Ġby Ġthe Ġcrowd Ġinstead Ġof Ġexperts . ĠCrowd Ġcontributors Ġcan Ġbe Ġfound Ġin Ġhigh Ġnumbers Ġand Ġmultiple Ġplatforms , Ġcompared Ġto Ġexpert Ġlingu ists . </s> E val uation Ġof Ġintens ifiers Ġand Ġneg ators , Ġwas Ġalso Ġa Ġbatch Ġof Ġevaluation Ġand Ġannotation Ġtasks , Ġas Ġmentioned Ġin ĠSection Ġ6 . 2 . ĠHowever , Ġthe Ġdifference Ġwas Ġthat Ġnow Ġeval u ators Ġhad Ġto Ġanswer Ġif Ġa Ġterm Ġgroup Ġincluded Ġat Ġleast Ġone Ġvalid Ġintens ifier Ġor Ġneg ator . ĠThe Ġevaluation Ġwas Ġagain Ġperformed Ġby Ġexperts Ġand Ġthe Ġcrowd , Ġas Ġdescribed Ġin ĠSection Ġ6 . 2 . 1 . ĠBased Ġon Ġthe Ġannotations Ġreceived Ġin ĠP EL , Ġwe Ġused Ġ5 41 Ġterm Ġgroups Ġthat Ġhad Ġat Ġleast Ġone Ġannotation Ġin Ġany Ġof Ġthe Ġintens ifying Ġsub classes . ĠAlthough , Ġthe Ġparticular Ġselection Ġof Ġterm Ġgroups Ġis Ġstatistically Ġsignificant , Ġwe Ġexpect Ġrelatively Ġlow Ġevaluation Ġscores . ĠThat Ġis Ġbecause Ġthe Ġnumber Ġof Ġintens ifying Ġannotations Ġis Ġlow Ġin Ġmost Ġof Ġthe Ġselected Ġterm Ġgroups . </s> In ĠFigure ĠFIG REF 40 Ġ, Ġwe Ġdefine Ġvarying Ġlevels Ġof Ġagreement Ġon Ġthe Ġvalidity Ġof Ġthe Ġintens ifying Ġclass , Ġbased Ġon Ġthe Ġagreement Ġof Ġevaluations . ĠFor Ġthe Ġexperts Ġgroup , Ġlow Ġagreement Ġrefers Ġto Ġterm Ġgroups Ġthat Ġreceived Ġat Ġleast Ġone Ġout Ġof Ġtwo Ġevaluations Ġas Ġvalid , Ġwhile Ġhigh Ġagreement Ġrequires Ġthe Ġevaluation Ġagreement Ġof Ġboth Ġexperts . ĠSimilarly Ġfor Ġthe Ġcrowd , Ġlow Ġagreement Ġrefers Ġto Ġa Ġminimum Ġof Ġtwo Ġvalid Ġevaluations , Ġmid Ġagreement Ġcorresponds Ġto Ġat Ġleast Ġthree , Ġand Ġhigh Ġagreement Ġrequires Ġan Ġabsolute Ġagreement Ġof Ġall Ġfour Ġeval u ators . </s> Experts Ġare Ġfar Ġmore Ġstrict Ġthan Ġthe Ġcrowd Ġin Ġthe Ġevaluation Ġof Ġintens ifiers Ġand Ġneg ators . ĠWhen Ġthe Ġvalidity Ġagreement Ġis Ġlow Ġon Ġboth Ġevaluation Ġgroups , Ġthe Ġaverage Ġvalid Ġterm Ġgroup Ġdifference Ġis Ġmore Ġthan Ġ40 %, Ġbut Ġthe Ġhigh Ġvalidity Ġagreement Ġthe Ġdifference Ġis Ġjust Ġ5 . 33 %. ĠWhen Ġhigh Ġagreement Ġevaluation Ġis Ġapplied , Ġthe Ġcrowd Ġand Ġexpert Ġevaluations Ġare Ġalmost Ġidentical . ĠThe Ġnumber Ġof Ġcrowd Ġevaluations Ġis Ġthe Ġfactor Ġthat Ġprovides Ġa Ġdegree Ġof Ġfreedom Ġin Ġthe Ġevaluation Ġstrict ness . </s> Lim itations </s> Lex icon Ġacquisition Ġis Ġa Ġcomplex Ġtask Ġthat Ġincludes Ġa Ġmixture Ġof Ġobjective Ġand Ġsubjective Ġtasks . ĠWhile Ġannotation Ġof Ġemotions Ġis Ġmore Ġsubjective , Ġannotation Ġof Ġlinguistic Ġelements Ġ( such Ġas Ġstop Ġwords , Ġemotion Ġshift Ġterms , Ġintens ifiers Ġetc .) Ġis Ġpurely Ġobjective . ĠWe Ġpresented Ġa Ġnovel Ġwork Ġflow Ġthat Ġprovides Ġquality Ġresults Ġfor Ġboth Ġsubjective Ġand Ġobjective Ġtasks . </s> Sub comp onents Ġof Ġthe Ġlex icon Ġacquisition Ġcould Ġbe Ġimproved Ġon Ġan Ġindividual Ġbasis . ĠSpell Ġcheck Ġcan Ġinclude Ġspelling Ġrecommendations , Ġfiltering Ġcould Ġincorporate Ġrewarding Ġand Ġpenalties , Ġevaluation Ġprocess Ġcan Ġinclude Ġexperts Ġand Ġso Ġon . </s> C rowd Ġdiversity Ġin Ġthe Ġannotation Ġand Ġevaluation Ġprocess Ġis Ġanother Ġlimiting Ġfactor . ĠIdeally Ġwe Ġwould Ġprefer Ġa Ġfixed Ġnumber Ġof Ġindividuals , Ġeach Ġto Ġannot ate Ġand Ġevaluate Ġthe Ġwhole Ġcorpus . ĠHowever , Ġthe Ġuniform ity Ġof Ġexpert Ġjudgement Ġis Ġreplaced Ġwith Ġthe Ġdiversity Ġand Ġmass Ġof Ġcontributors . </s> The Ġcorpus Ġmay Ġbe Ġlimiting Ġthe Ġterm Ġgroups Ġin Ġthe Ġlex icon Ġto Ġspecific Ġdomain - specific Ġsubjects . ĠCompar isons Ġwith Ġexisting Ġlex icons , Ġsuch Ġas ĠN RC ĠB IB REF 21 Ġindicate Ġa Ġmoderate Ġoverlap Ġwith Ġ40 % Ġcommon Ġterms . ĠAdditionally , Ġthe Ġnumber Ġof Ġannotations Ġfor Ġa Ġnumber Ġof Ġterm Ġgroups Ġis Ġrelatively Ġlow . ĠHowever , Ġthe Ġbatch Ġtask Ġof Ġevaluation Ġand Ġannotation Ġprovided Ġalmost Ġten Ġthousand Ġannotations , Ġand Ġincreased Ġthe Ġmean Ġnumber Ġof Ġannotations Ġfrom Ġ2 . 3 Ġto Ġ3 . 2 . </s> Conclusion Ġand Ġfuture Ġwork </s> We Ġdemonstrated Ġthat Ġthe Ġcrowd Ġis Ġcapable Ġof Ġproducing Ġand Ġevaluating Ġa Ġquality Ġpure Ġemotion Ġlex icon Ġwithout Ġgold Ġstandards . ĠOur Ġwork - flow Ġis Ġun super vised , Ġsignificantly Ġlower Ġcosts , Ġand Ġimproves Ġscal ability . ĠThere Ġare Ġhowever , Ġvarious Ġparameters Ġthat Ġshould Ġbe Ġtaken Ġinto Ġaccount . ĠSp am Ġis Ġvery Ġcommon Ġand Ġquality Ġassessment Ġpost - annot ations Ġshould Ġbe Ġimplemented . </s> Our Ġapproach Ġrequired Ġworkers Ġto Ġlabel Ġterm Ġgroups Ġas Ġemotion , Ġintens ifiers , Ġand Ġstop Ġwords . ĠAgreement Ġis Ġnot Ġnecessary Ġand Ġmulti Ġemotional Ġterm Ġgroups , Ġwith Ġup Ġto Ġthree Ġemotions , Ġare Ġconsidered Ġequally Ġvalid Ġto Ġsingle Ġemotion Ġterm Ġgroups . ĠThe Ġhardest Ġtask Ġfor Ġthe Ġcrowd Ġproved Ġto Ġbe Ġthe Ġclassification Ġof Ġintens ifiers Ġand Ġneg ators , Ġprobably Ġbecause Ġit Ġrequired Ġa Ġcertain Ġlevel Ġof Ġobject ivity Ġwhich Ġcontradicted Ġthe Ġoverall Ġsubject ivity Ġof Ġthe Ġemotional Ġannotation Ġtask . ĠBased Ġon Ġthe Ġevaluation Ġof Ġterm Ġgroups Ġand Ġthe Ġresults Ġfrom Ġthe Ġassessment , Ġas Ġthe Ġnumber Ġof Ġoverall Ġannot ators Ġrises Ġthe Ġnumber Ġof Ġvalid Ġannotations Ġincreases Ġproportion ally . ĠThis Ġindicates Ġthe Ġimportance Ġof Ġa Ġcritical Ġmass Ġin Ġlex icon Ġacquisition Ġtasks . </s> St em ming Ġreduced Ġtime Ġand Ġcosts Ġrequirements , Ġwith Ġminimal Ġemotional Ġand Ġsubclass Ġagreement . ĠCosts Ġwere Ġreduced Ġby Ġ45 %, Ġand Ġmulti - em otion Ġclassification Ġwas Ġlower Ġthan Ġ10 %. ĠTerm Ġgroups Ġdid Ġnot Ġcreate Ġconfusion Ġamongst Ġworkers , Ġand Ġonly Ġa Ġsmall Ġfraction Ġof Ġterm Ġgroups Ġhad Ġsubclass Ġagreement . ĠOn Ġthe Ġcontrary , Ġincluding Ġthe Ġstem Ġand Ġdescription Ġin Ġthe Ġtask Ġconfused Ġworkers , Ġand Ġwere Ġexcluded Ġfrom Ġthe Ġinterface . ĠWe Ġtested Ġseveral Ġinterface Ġdesigns , Ġand Ġthe Ġone Ġthat Ġworked Ġbest Ġhad Ġminimal Ġinstructions . ĠLex icon Ġacquisition Ġinterfaces Ġin Ġpaid Ġmicro - task Ġenvironments Ġshould Ġbe Ġfurther Ġstudied , Ġwith Ġregards Ġto Ġvarious Ġother Ġcontribution Ġincentives . </s> The Ġcrowd Ġis Ġas Ġcapable Ġof Ġevaluating Ġlex icons , Ġas Ġexperts . ĠL ingu istic Ġelement Ġevaluation Ġcan Ġbe Ġefficiently Ġcrowds ourced , Ġand Ġthe Ġevaluation Ġof Ġemotional Ġor Ġnon Ġemotional Ġelements Ġcan Ġbe Ġas Ġstrict Ġas Ġneeded . ĠThe Ġnumber Ġof Ġeval u ators Ġplays Ġa Ġkey Ġrole Ġin Ġboth Ġemotional Ġand Ġlinguistic Ġevaluations . ĠThe Ġcrowd Ġis Ġstrict Ġon Ġemotional Ġevaluations , Ġwhile Ġthe Ġexperts Ġare Ġstrict Ġin Ġlinguistic Ġevaluations . ĠHowever , Ġa Ġhigh Ġnumber Ġof Ġcrowd Ġevaluations Ġbroad ens Ġthe Ġstrict ness Ġfreedom , Ġwith Ġa Ġsmall Ġfraction Ġof Ġthe Ġexperts ' Ġhiring Ġcosts . ĠDepending Ġon Ġthe Ġnumber Ġof Ġevaluations , Ġvarying Ġlevels Ġof Ġevaluation Ġagreement Ġcan Ġbe Ġimplemented . </s> Our Ġlong Ġterm Ġgoal Ġis Ġto Ġcreate Ġa Ġvoluntary Ġplatform Ġfor Ġpure Ġemotion Ġlex icon Ġacquisition , Ġto Ġfurther Ġstudy Ġthe Ġeffects Ġof Ġcritical Ġmass Ġin Ġlex icon Ġacquisition . ĠIn Ġshort Ġterm , Ġwe Ġwill Ġperform Ġthe Ġexact Ġsame Ġcrowds ourcing Ġtask Ġin Ġa Ġvoluntary Ġplatform , ĠCrowd 4 U Ġor Ġsimilar Ġplatforms , Ġto Ġstudy Ġthe Ġeffect Ġof Ġmonetary Ġand Ġcontribution Ġincentives Ġin Ġpure Ġemotion Ġsentiment Ġannotation . ĠIn Ġparallel , Ġwe Ġwill Ġperform Ġa Ġqualitative Ġanalysis Ġwith Ġregards Ġto Ġunderstanding Ġof Ġintens ifiers Ġand Ġneg ators , Ġto Ġcreate Ġthe Ġoptimal Ġset Ġof Ġinstructions Ġand Ġexamples . ĠFinally , Ġwe Ġare Ġconsidering Ġhow Ġwe Ġcan Ġextend Ġthe Ġapproach Ġto Ġvarious Ġother Ġlinguistic Ġelements , Ġsuch Ġas Ġwords Ġthat Ġsplit Ġthe Ġsentence , Ġwords Ġthat Ġindicate Ġmore Ġimportant Ġparts Ġof Ġa Ġsentence Ġand Ġso Ġon . </s> We Ġbelieve Ġthat Ġbeyond Ġpol arity Ġsentiment Ġanalysis Ġcan Ġenhance Ġand Ġextend Ġsimple Ġpol arity Ġbased Ġapplications . ĠSent iment Ġanalysis Ġin Ġmarketing , Ġpolitics , Ġhealth Ġmonitoring , Ġonline Ġsocial Ġnetworks , Ġand Ġevaluation Ġprocesses Ġwould Ġbenefit Ġfrom Ġa Ġcrowds ourced Ġpure Ġemotion Ġlex icon . </s> Acknowled gement </s> This Ġwork Ġwas Ġsupported Ġby Ġthe ĠEU Ġproject Ġ" Q R OW D Ġ- ĠBecause ĠBig ĠData ĠIntegration Ġis ĠHuman ly ĠPossible ".
Introduction Sentiment analysis aims to uncover the emotion conveyed through information. In online social networks, sentiment analysis is mainly performed for political and marketing purposes, product acceptance and feedback systems. This involves the analysis of various social media information types, such as text BIBREF0 , emoticons and hashtags, or multimedia BIBREF1 . However, to perform sentiment analysis, information has to be labelled with a sentiment. This relationship is defined in a lexicon. Lexicon acquisition is a requirement for sentiment classification. During the acquisition process, individual or grouped information elements are labelled based on a class, usually an emotion. Sentiment classification is the task that uses the acquired lexicon and a classification method to classify a sentence, phrase, or social media submission as a whole, based on the aggregation of its labels. Thus, lexicon quality directly affects sentiment classification accuracy. Both tasks can either be performed automatically BIBREF2 or manually BIBREF3 where the labelling by linguists or researchers themselves BIBREF4 . Apart from experts, manual labbeling can also be performed with the help of a wide network of people, known as crowdsourcing BIBREF5 . Crowdsourcing is widely used for polarity lexicons, but rarely for beyond polarity and never for the discovery of linguistic elements. Sentiment analysis is commonly performed in polarity basis, i.e. the distinction between positive and negative emotion . These poles correspond to agreement and disagreement, or acceptance and disapproval, for candidates and products repsectively BIBREF6 . Beyond polarity (also known as pure emotion) sentiment analysis aims to uncover an exact emotion, based on emotional theories BIBREF7 , BIBREF8 . Applications such as sentiment tracking, marketing, text correction, and text to speech systems can be improved with the use of distinct emotion lexicons. However, beyond polarity studies acquire lexicons based on a set of strict rules, and the evaluation of experts. These lexicons use only a single emotion per term BIBREF9 . The problems of these approaches is the lack of uniformity and contribution freedom when relying on gold standards, and high costs with low scalability when employing experts. Natural Language Processing (NLP) applications that only rely on experts are less comprehensive, restricted, and not scalable, compared to crowdsourced NLP applications BIBREF10 . This paper presents our approach for the acquisition of a multiclass and scalable crowdsourced pure emotion lexicon (PEL), based on Plutchik's eight basic emotions. Furthermore, the crowd is also responsible for identifying linguistic elements, namely intensifiers, negators, and stop words. Usually these elements are pooled from existing lists BIBREF11 created by experts. We also introduce a worker filtering method to identify and exclude dishonest or spamming contributors, that doesn't require gold standards. Our goal is to maintain an end to end automated work-flow for a crowdsourced (annotation and evaluation wise) lexicon acquisition process. Therefore, to highlight crowd's performance on evaluation, we compare evaluations from linguistic experts and the crowd itself. Related Work According to BIBREF12 , an emotion is defined with reference to a list. Ekam et al. BIBREF8 proposed the six basic emotions joy, anger, fear, sadness, disgust, and surprise. Years later, Plutchik BIBREF7 proposed the addition of trust and anticipation as basic emotions, and presented a circumplex model of emotions as seen in Figure FIGREF1 , which defines emotional contradictions and some of the possible combinations. Sentiment analysis aims to classify information based on the emotion conveyed. Depending on the number of classes/emotions required, we can separate the analysis into: polarity and beyond polarity. Polarity sentiment analysis studies define two opposite emotional states, positive and negative, or good and bad, with the addition of a neutral state. Furthermore, some researchers have classified information on levels for each pole(e.g. very positive, positive, neutral, negative, very negative etc.), also known as fine grained sentiment analysis BIBREF13 . Beyond polarity, also known as pure emotion, sentiment analysis is a more refined approach to the same problem with a wider range of possible emotion classes, see Figure FIGREF1 . Essentially, any sentiment analysis that involves specific emotional labelling, is considered as a beyond polarity analysis. Examples of emotional labels might be -but are not limited to-: sadness, boredom, joy, sadness, surprise, anger, fear, disgust etc. As discussed in Section 1, one of the core tasks of sentiment analysis is lexicon acquisition. A lexicon can be acquired through manual or automatic annotation. However, natural language has a very subjective nature BIBREF14 which significantly inhibits automated sentiment lexicon aqcuisition methods from achieving relevance equal to manual methods BIBREF15 . Thus a lot of researchers choose to manually annotate their term corpora BIBREF16 , or use established lexicon such as WordNet, SentiWordNet, and various other lexicons BIBREF13 . Other studies combine manual labeling or machine learning with lexicons BIBREF17 . Manual lexicon acquisition is constrained by the number of people contributing to the task, and the number of annotations from each participant. These constraints can be eliminated by increasing the number of people involved, for instance, by using crowdsourcing BIBREF18 . Amazon's Mechanical Turk (MTurk) is a crowdsourcing platform frequently used for polarity sentiment lexicon acquisition via crowdsourcing BIBREF19 . MTurk is also used, for the annotation of one thousand tweets in BIBREF20 , ten thousand terms in BIBREF21 with gold standards, and the annotation of ninety five emoticons out of one thousand total emoticons found in BIBREF22 . While BIBREF23 had one thousand four hundred terms labelled with a supervised machine learning and crowd validators. The challenge is to introduce a work-flow that is scalable, unsupervised and applicable to different information types. The second core part in sentiment analysis, is sentiment classification. A classification that occurs at phrase/sentence/submission level, and is usually based on the aggregation of the term's labeled emotions. As with lexicon aqcuisition, the classification task can be automated BIBREF13 or performed manually BIBREF24 . Regardless of manual or automated sentiment classification, on textual information scenarios, term and phrase sentiment is the main input of the classification method. In some cases the decision might be totally different from the individual term emotion, leading to relabeling of the terms themselves BIBREF25 . Manually labelled classification can achieve high relevance, but it requires additional resources, and is not easily scalable. On the other hand, automated processes are scalable but with lower relevance BIBREF24 . Our approach Our aim is to create an end to end automated work-flow for the creation, evaluation and enrichment of a pure emotion lexicon. The work-flow, Figure FIGREF3 , can be separated in two main components. Pre-processing is the unsupervised process by which we derive the lexicon terms from any textual resource, while crowdsourcing deals with the crowdsourcing aspect of the lexicon. The Pure Emotions Lexicon includes emotional term groups, intensifiers and negators, and stop words. Pre-processing is comprised of 3 unsupervised steps, tokenization, stemming and spell check. Textual content is tokenized as uni-grams, stemmed based on their rooted and checked for spelling. The resulting stems along with their stem groups are stored in a lexicon database. Crowdsourcing is using the lexicon database and the crowd to annotate each entry in the database. Participants submit their answers that go through a filtering process. If the answers are considered valid, they update the lexicon entries. The crowd also evaluates existing annotations, to determine the lexicon quality. As crowd evaluation methods are new in lexicon acquisition tasks, we compare crowd evaluations to those of expert linguists. Data During January 2017, we performed a keyword based crawl for articles and comments in the Europe subreddit and tweets in Twitter, which contained the word "Brexit". The use of a political and controversial term in the query is deliberate, to capture the emotional diversity of a politically oriented corpus. We crawled one hundred articles from Reddit, with more than forty thousand comments and more than three thousand tweets. For the Reddit data, we collected information on location, time, and the number of upvotes. For the Twitter data, we stored the number or re-tweets and favourites, time and location information. Our focus is on single term (also known as unigram) sentiment, thus posts in both networks were processed to a single term list. In total, the number of unique terms in our corpus was 30227. Based on the enchant python library, used in BIBREF26 , and the supported Great British English dictionary, 19193 were validated and 11034 were invalidated. Our analysis will focus on the 19193 valid terms, that follow Zipf's Law with scaling-law coefficient INLINEFORM0 is a good fit. After validation, terms were stemmed with Porter Stemming Algorithm BIBREF27 . Stemming identified 10953 distinct term groups with one or more terms. Stop-words, intensifiers and negators are also included in the valid term groups. Both term validation and stemming are unsupervised, since our goal is to maintain scalability in beyond polarity lexicon acquisition and sentiment classification. Crowdsourcing The crowdsourcing task, hosted in CrowdFlower, required contributors to label term groups in three different main classes, emotion, intensifier and none, without a golden standard, rules or any participation restrictions . Emotion labelling included the 8 basic emotions as defined by Plutchik. Intensifier class included intensifiers and negators. Finally, none referred to stop-words or words with no particular emotion. Each of the eleven options for the main classes, will be referred to as "subclass". Terms are grouped based on their stem. Each term group has a main annotation class defined by majority, and several sub annotation classes, defined by the non majority annotations. However, to aid multi class analysis of the results, every annotation is logged in the lexicon database. Task Interface The task interface was the result of several experiments. Three major changes implemented based on these experimental interfaces were: the simplification of the task question, Figure FIGREF8 , the inclusion of only three main classes, and the replacement of words positive and negative, with amplifying and weakening in the intensifying class options. All experiments and the annotation task required highly experienced contributors, as defined by Crowdflower platform. As seen in Figure FIGREF8 contributors select one of the three choices. If they choose Emotion evoking, they are presented with a drop-down menu to choose from the eight basic emotions. Similarly, if they select Intensifying context they have to specify whether it was Amplifying or Weakening the context, essentially annotating the intensifiers and negators. Finally if they choose None they are presented with the next term group. To assist contributors with term definitions, every term group had a hyperlink to an English dictionary. Crowd More than one hundred eighty contributors performed eighty thousand annotations. By design, each user could not perform more than 660 unique annotations, excluding the assessment questions, to engage at least 100 contributors. Most of the workers annotated the maximum allowed terms, the first half of workers annotated 15% of the term groups in our corpus, while the second half of workers annotate the rest 85%. The simplicity of the task resulted in high overall worker engagement, with mean and median annotations per worker, at 429 and 580 respectively. Assessment Based on a set of experiments, we identified 136 term groups that would test the ability of a contributor in all of the three main classes, emotion evoking, intensifying context, and none. As the assessment term groups had more than ten thousand annotations, we analyse it separately from the lexicon. In order for a worker to gain the ability to contribute to the crowdsourcing task and eventually get paid, he/she had to properly annotate 80% of the assessment term groups encountered. The annotations should be within the dominant classes, and not subclasses, as defined from the assessment annotators. E.g., for an assessment term group that received 100 annotations in various emotions, we check if the worker annotates the term group as emotion evoking. Let INLINEFORM0 be the set of workers INLINEFORM1 and and INLINEFORM2 the set of eleven INLINEFORM3 subclasses: eight emotions , two intensifiers, and none of the former. We define INLINEFORM4 as the number of total annotations for each worker INLINEFORM5 . Then: DISPLAYFORM0 We define INLINEFORM0 be the set of workers INLINEFORM1 in the assessment process, INLINEFORM2 the set of workers INLINEFORM3 in the acquisition process. Then, for INLINEFORM4 we define: DISPLAYFORM0 DISPLAYFORM1 and: DISPLAYFORM0 DISPLAYFORM1 The optimal INLINEFORM0 is found for INLINEFORM1 . For this study, the optimal filtering percentage was found at 40%, INLINEFORM2 . Workers from India and Venezuela, who contributed 92% of the task, have annotated more than 30% of the term groups with joy. However, annotations from countries with more than 300 annotations, don't follow the same distribution. Specifically, workers from Philippines, United States, Colombia, Poland, United Kingdom, Russia, and Egypt, performed a smoother distributed emotion annotation. In comparison, the most annotated emotion in BIBREF21 was fear in 18% of the total terms. By further analysing worker annotation distribution, we identified workers that had a significant fraction of their total annotations in a single subclass. E.g. one specific worker annotated 99% of the assessment term groups he encountered as joy. Dishonesty or spamming is a known problem in crowdsourcing BIBREF28 and multiple proposed solutions exist BIBREF28 , but they require gold standards or objective crowdsourcing tasks. As we don't have a gold standard, and the task is more subjective, these spamming elimination methods are not applicable. Our solution is the implementation of a fast and efficient filter, which only relies on the obtained annotations and the assessment. If workers' answers were above a certain percentage on a single subclass for both the assessment and the annotation process, then the user would be flagged as dishonest and the total of their annotations would be discarded. This rule was applied to the whole range of possible answers, including the 8 emotions, 2 intensifiers and "none". Prior to implementing the filter, we analysed how it would affect the number of eligible, not considered as spamming, workers. The thick line in Figure FIGREF22 shows the percentage during the assessment term groups, and the dotted line shows the percentage during the lexicon acquisition. The higher the single annotation percentage, the higher the certainty of spamming behaviour. In large part, the exclusion rate was similar for both the assessment and the lexicon annotations. A number of workers had a more cautious behaviour in the test questions, resulting in reduced percentage of exclusions during assessment process. This behaviour is justified, as the first set of questions encounter by a worker are knowingly a set of assessment questions. Each assessment term group was annotated more than 120 times, by 187 annotators. This is a critical mass of contributors and provides valuable findings with regards to task. These are: Workers rarely clicked the informative dictionary link. As a result, they would annotate emotional words to none, probably due to misinterpretation. We avoided the direct inclusion of the dictionary definition, as it could be considered as a form of leverage. E.g. "vehement, halcyon" , two -uncommon- emotion baring words, were both annotated as none, and less than 0.2% of the workers (on average) clicked a dictionary link. The concept of intensifiers is understandable but requires critical mass BIBREF29 . A small number of annotators would initially annotate intensifiers/negators with an emotion, but the distribution would slowly shift towards the correct class. E.g. "reduce, little, plethora" , were initially annotated as sad sad joy, but after tens of annotations they were annotated as weakening weakening intensifying. All words should be evaluated, even those that seemingly don't carry a specific emotion. As times change, words and emotions acquire new links. E.g. "anti, serious" , were both annotated as fear evoking with a great emotional diversity. Lexicon Analysis The lexicon (will be referred as simply "PEL") is created after the exclusion of annotations following the 40% single annotation filtering check. We received more than seventy thousands annotations for 10593 term groups, of those only 22 thousand annotations for 9737 term groups are included in the final lexicon, as a result of filtering. Each term group had a mean 2.3 annotations from a total of 95 different annotators. Although the number of mean annotations in lexicon is less than half the mean annotations in the unfiltered corpus, the PEL annotations are considered of higher quality. Each lexicon term group has multiple subclass annotations, and the main subclass is defined by majority. Even after filtering, the dominant emotion in our lexicon is joy, while the least annotated emotion is disgust. Additionally, 148 terms were annotated as intensifiers, 43 terms as negators, and 6801 terms as none. A sample of five terms for each of the three subclasses can be seen in Table TABREF27 . The full lexicon can be found on github. Intensifiers and negators serve as modifiers to the emotional context of a word. Workers identified mostly valid intensifiers and negators that can modify emotion evoking words, in the absence of context. Judging from the received annotations, there is room for improvement on the description of the intensifier class and the provided examples, as a number of non intensifying words were falsely annotated. Terms in our lexicon are grouped based on their stem. Stemming significantly reduced cost (by half) and time-required for the task. Grouping terms may create unnecessary multi-class annotations agreements, for terms in the same term group that might have different meanings. Annotation agreement refers to equal number of annotations in multiple subclasses or emotions. However, the vast majority of term groups in our lexicon, don't display any form of contradicting annotation. Contradicting emotions are portrayed in opposite edges of the circumplex Figure FIGREF1 , while emotional combinations are decribed in BIBREF7 . In the lexicon, only 21% and 20% of the term groups had a subclass and an emotional agreement respectively. With regards to emotion, contradicting or multi-emotion agreement, could be observed in only 8.6% of the total term groups. Let INLINEFORM0 be a term group in the lexicon and INLINEFORM1 the set of eleven INLINEFORM2 subclasses: eight emotions , two intensifiers, and none of the former. We define INLINEFORM3 as the number of annotations for each term group INLINEFORM4 . For each INLINEFORM5 , the annotations for emotion subclasses are INLINEFORM6 , the annotations for intensifying subclasses are INLINEFORM7 , and the number of none annotations is INLINEFORM8 . Therefore, each INLINEFORM0 can have an monotonically increasing finite sequence INLINEFORM1 with INLINEFORM2 , where: DISPLAYFORM0 We say that term group INLINEFORM0 has subclass agreement if and only if: DISPLAYFORM0 While INLINEFORM0 has emotional agreement if and only if there is a subclass agreement with the sequence INLINEFORM1 and: DISPLAYFORM0 Subclass agreement, refers to equal annotations, between emotional subclass(es) and at least one non-emotional subclass, or between multiple non-emotional subclass, Equation EQREF30 . On the other hand, emotional agreement refers to multiple emotion subclasses with equal annotations, Equation EQREF31 . The number of subclasses in agreement and the number of terms in a term group are negatively correlated. Term groups with two terms appear to have the highest subclass agreement with exactly two subclasses. The most common occurring agreements are subclass none paired with an emotion, and joy paired with an emotion. The number of multi-class agreement occurrences is disproportional to the number of terms in a term group. This is a strong indication that stemming didn't confuse workers. Similarly, for emotional agreement, the number of occurrences is disproportionate to the number of terms in the term group. Furthermore, emotional agreement appeared in 10% of the term groups, while subclass agreement was found in 20% of the term groups. In the agreement annotations, joy is the most common emotion. According to Plutchik's circumplex Figure FIGREF1 , each emotion has a contradicting one, and pairs of emotions indicate a more "complex" emotion. There are 697 emotional agreeing term groups, of 1434 terms, with exactly two emotions. These emotional dyads BIBREF7 can be combined as seen in Table TABREF32 . Simple basic emotion annotation tasks can indirectly provide complex emotional annotations. Dyadic emotional agreements could be interpreted as the resulting complex emotion, or further annotated to obtain a single dominant emotion. There was a number of term groups with opposite emotion dyads, presented in Table TABREF33 ,but as the number of annotations increases, emotional agreement occurrences -combination or opposition- decreases. In total, the lexicon features 17740 annotated terms with 3 classes and 11 subclasses.The dominant class for 7030 terms was emotion, 191 intensifying, 6801 none, and 3718 in some form of subclass agreement. Lexicon terms are mainly joy annotated, and emotional agreement is prevalent in 10% of the terms. Only 21% of total terms have a subclass agreement. Reliability Single annotation reliability agreement is the degree of agreement between annotators, for term groups that have annotation majority in exactly one sub class. In our lexicon, single annotation reliability agreement was low, mainly due to the low number of annotators for each term group in relation to the high number of possible categories. Based on Fleiss Kappa BIBREF30 (simply referred as k), and as seen in Table TABREF35 , term groups with 2 annotations had the lowest reliability agreement, while term groups with 6 annotations the highest reliability agreement. As the number of annotators rises, the number of possible agreement permutations increases but the number of major annotated subclasses decreases. More annotators have a positive effect in both k and certainty of classification. As we restrict our lexicon to emotions, reliability increases for any number of annotators except two. This is explained by the decrease in the number of possible categories. When we restrict our analysis on emotion related annotation the probability for agreement in annotations increases, resulting in a high emotional k. The best way to increase k is to provide additional annotations that will eventually converge to a majority class or a limited group of classes. Crowd and experts comparison We perform a direct comparison of expert and crowd contributors, for 1000 term groups based on the number of total annotations(200 term groups with 2 total annotations, 200 term groups with 3 total annotations, and so on up to term groups with 6 total annotations). The experts are two Ph.D. linguists, while the crowd is made up of random high quality contributors that choose to participate in the task. As a reference, the cost of hiring two experts is equal to the cost of employing nineteen contributors in Crowdflower. Evaluators were given a summary of the annotations received for the term group in the form of:The term group "inequality inequity" received annotations as 50.0% sadness, 33.33% disgust, 16.67% anger. Then, they were asked to evaluate on a scale from 1 to 5, how valid these annotations were considered. The summary of the evaluation for both experts and crowd can be seen in Figure FIGREF36 . The first graph presents the validity over the number of annotations in the main class of the term group. Although this information is hidden from the evaluators, high annotational agreement results in high evaluation scores. Both experts and the crowd follow that positive trend. Crowd contributors are more strict in their evaluations, but after four annotations we observe a significant validity increase on both crowd and experts. Likewise, the annotation percentage for the majority class has a positive influence to the evaluation score, with the exception of 100% agreement, second graph Figure FIGREF36 . The weighing factor for term groups with 100% annotation agreement is the reduced number of total annotations, as the mean number of total annotations drops abruptly on the 100%, and total agreement is more frequent in term groups with low number of total annotations. It's worth noting that certain percentages can only occur on specific number of total annotations, e.g. 17% and 83% can only occur when the number of total annotations is six. In emotion annotations, as seen on the third graph of Figure FIGREF36 crowd and experts follow a similar evaluation pattern. Anticipation and joy had the exact same evaluation, while every other emotion and stop words were evaluated lower from the crowd. The only subclasses evaluated higher from the crowd were intensifiers and negators, with a significant difference in the evaluations for the latter. Section 6.3 provides a more detailed evaluation for term groups that received at least one annotation as intensifiers or negators. The final graph in Figure FIGREF36 presents a clear negative correlation of subclass agreement and evaluation scores. The highest number of subclasses that do not affect evaluation scores is three, above that there is a steady decline of the evaluation scores, for both the crowd and the experts. The evaluation results provide some key insights in the importance of the number of annotations. The evaluation scores start to improve after four annotations. Annotational agreement and majority voting are less important. Subclass agreement has a negative effect on three or more subclasses. Most importantly and compared to experts, the crowd is a stricter evaluator with significantly lower costs, and higher scalability. Since strict evaluation leads to higher quality annotations, the evaluation can be performed by the crowd instead of experts. Crowd contributors can be found in high numbers and multiple platforms, compared to expert linguists. Evaluation of intensifiers and negators, was also a batch of evaluation and annotation tasks, as mentioned in Section 6.2. However, the difference was that now evaluators had to answer if a term group included at least one valid intensifier or negator. The evaluation was again performed by experts and the crowd, as described in Section 6.2.1. Based on the annotations received in PEL, we used 541 term groups that had at least one annotation in any of the intensifying subclasses. Although, the particular selection of term groups is statistically significant, we expect relatively low evaluation scores. That is because the number of intensifying annotations is low in most of the selected term groups. In Figure FIGREF40 , we define varying levels of agreement on the validity of the intensifying class, based on the agreement of evaluations. For the experts group, low agreement refers to term groups that received at least one out of two evaluations as valid, while high agreement requires the evaluation agreement of both experts. Similarly for the crowd, low agreement refers to a minimum of two valid evaluations, mid agreement corresponds to at least three, and high agreement requires an absolute agreement of all four evaluators. Experts are far more strict than the crowd in the evaluation of intensifiers and negators. When the validity agreement is low on both evaluation groups, the average valid term group difference is more than 40%, but the high validity agreement the difference is just 5.33%. When high agreement evaluation is applied, the crowd and expert evaluations are almost identical. The number of crowd evaluations is the factor that provides a degree of freedom in the evaluation strictness. Limitations Lexicon acquisition is a complex task that includes a mixture of objective and subjective tasks. While annotation of emotions is more subjective, annotation of linguistic elements (such as stop words, emotion shift terms, intensifiers etc.) is purely objective. We presented a novel work flow that provides quality results for both subjective and objective tasks. Subcomponents of the lexicon acquisition could be improved on an individual basis. Spell check can include spelling recommendations, filtering could incorporate rewarding and penalties, evaluation process can include experts and so on. Crowd diversity in the annotation and evaluation process is another limiting factor. Ideally we would prefer a fixed number of individuals, each to annotate and evaluate the whole corpus. However, the uniformity of expert judgement is replaced with the diversity and mass of contributors. The corpus may be limiting the term groups in the lexicon to specific domain-specific subjects. Comparisons with existing lexicons, such as NRC BIBREF21 indicate a moderate overlap with 40% common terms. Additionally, the number of annotations for a number of term groups is relatively low. However, the batch task of evaluation and annotation provided almost ten thousand annotations, and increased the mean number of annotations from 2.3 to 3.2. Conclusion and future work We demonstrated that the crowd is capable of producing and evaluating a quality pure emotion lexicon without gold standards. Our work-flow is unsupervised, significantly lower costs, and improves scalability. There are however, various parameters that should be taken into account. Spam is very common and quality assessment post-annotations should be implemented. Our approach required workers to label term groups as emotion, intensifiers, and stop words. Agreement is not necessary and multi emotional term groups, with up to three emotions, are considered equally valid to single emotion term groups. The hardest task for the crowd proved to be the classification of intensifiers and negators, probably because it required a certain level of objectivity which contradicted the overall subjectivity of the emotional annotation task. Based on the evaluation of term groups and the results from the assessment, as the number of overall annotators rises the number of valid annotations increases proportionally. This indicates the importance of a critical mass in lexicon acquisition tasks. Stemming reduced time and costs requirements, with minimal emotional and subclass agreement. Costs were reduced by 45%, and multi-emotion classification was lower than 10%. Term groups did not create confusion amongst workers, and only a small fraction of term groups had subclass agreement. On the contrary, including the stem and description in the task confused workers, and were excluded from the interface. We tested several interface designs, and the one that worked best had minimal instructions. Lexicon acquisition interfaces in paid micro-task environments should be further studied, with regards to various other contribution incentives. The crowd is as capable of evaluating lexicons, as experts. Linguistic element evaluation can be efficiently crowdsourced, and the evaluation of emotional or non emotional elements can be as strict as needed. The number of evaluators plays a key role in both emotional and linguistic evaluations. The crowd is strict on emotional evaluations, while the experts are strict in linguistic evaluations. However, a high number of crowd evaluations broadens the strictness freedom, with a small fraction of the experts' hiring costs. Depending on the number of evaluations, varying levels of evaluation agreement can be implemented. Our long term goal is to create a voluntary platform for pure emotion lexicon acquisition, to further study the effects of critical mass in lexicon acquisition. In short term, we will perform the exact same crowdsourcing task in a voluntary platform, Crowd4U or similar platforms, to study the effect of monetary and contribution incentives in pure emotion sentiment annotation. In parallel, we will perform a qualitative analysis with regards to understanding of intensifiers and negators, to create the optimal set of instructions and examples. Finally, we are considering how we can extend the approach to various other linguistic elements, such as words that split the sentence, words that indicate more important parts of a sentence and so on. We believe that beyond polarity sentiment analysis can enhance and extend simple polarity based applications. Sentiment analysis in marketing, politics, health monitoring, online social networks, and evaluation processes would benefit from a crowdsourced pure emotion lexicon. Acknowledgement This work was supported by the EU project "QROWD - Because Big Data Integration is Humanly Possible".
How do they compare lexicons?
c09a92e25e6a81369fcc4ae6045491f2690ccc10
How Ġdo Ġthey Ġcompare Ġlex icons ?
Human Ġeval u ators Ġwere Ġasked Ġto Ġevaluate Ġon Ġa Ġscale Ġfrom Ġ1 Ġto Ġ5 Ġthe Ġvalidity Ġof Ġthe Ġlex icon Ġannotations Ġmade Ġby Ġthe Ġexperts Ġand Ġcrowd Ġcontributors .
<s> How Ġdo Ġthey Ġcompare Ġlex icons ? </s> Introduction </s> Sent iment Ġanalysis Ġaims Ġto Ġuncover Ġthe Ġemotion Ġconveyed Ġthrough Ġinformation . ĠIn Ġonline Ġsocial Ġnetworks , Ġsentiment Ġanalysis Ġis Ġmainly Ġperformed Ġfor Ġpolitical Ġand Ġmarketing Ġpurposes , Ġproduct Ġacceptance Ġand Ġfeedback Ġsystems . ĠThis Ġinvolves Ġthe Ġanalysis Ġof Ġvarious Ġsocial Ġmedia Ġinformation Ġtypes , Ġsuch Ġas Ġtext ĠB IB REF 0 Ġ, Ġemot icons Ġand Ġhasht ags , Ġor Ġmultimedia ĠB IB REF 1 Ġ. ĠHowever , Ġto Ġperform Ġsentiment Ġanalysis , Ġinformation Ġhas Ġto Ġbe Ġlabelled Ġwith Ġa Ġsentiment . ĠThis Ġrelationship Ġis Ġdefined Ġin Ġa Ġlex icon . </s> Lex icon Ġacquisition Ġis Ġa Ġrequirement Ġfor Ġsentiment Ġclassification . ĠDuring Ġthe Ġacquisition Ġprocess , Ġindividual Ġor Ġgrouped Ġinformation Ġelements Ġare Ġlabelled Ġbased Ġon Ġa Ġclass , Ġusually Ġan Ġemotion . ĠSent iment Ġclassification Ġis Ġthe Ġtask Ġthat Ġuses Ġthe Ġacquired Ġlex icon Ġand Ġa Ġclassification Ġmethod Ġto Ġclassify Ġa Ġsentence , Ġphrase , Ġor Ġsocial Ġmedia Ġsubmission Ġas Ġa Ġwhole , Ġbased Ġon Ġthe Ġaggregation Ġof Ġits Ġlabels . ĠThus , Ġlex icon Ġquality Ġdirectly Ġaffects Ġsentiment Ġclassification Ġaccuracy . </s> Both Ġtasks Ġcan Ġeither Ġbe Ġperformed Ġautomatically ĠB IB REF 2 Ġor Ġmanually ĠB IB REF 3 Ġwhere Ġthe Ġlab elling Ġby Ġlingu ists Ġor Ġresearchers Ġthemselves ĠB IB REF 4 Ġ. ĠApart Ġfrom Ġexperts , Ġmanual Ġlab bel ing Ġcan Ġalso Ġbe Ġperformed Ġwith Ġthe Ġhelp Ġof Ġa Ġwide Ġnetwork Ġof Ġpeople , Ġknown Ġas Ġcrowds ourcing ĠB IB REF 5 Ġ. ĠCrowd s ourcing Ġis Ġwidely Ġused Ġfor Ġpol arity Ġlex icons , Ġbut Ġrarely Ġfor Ġbeyond Ġpol arity Ġand Ġnever Ġfor Ġthe Ġdiscovery Ġof Ġlinguistic Ġelements . </s> Sent iment Ġanalysis Ġis Ġcommonly Ġperformed Ġin Ġpol arity Ġbasis , Ġi . e . Ġthe Ġdistinction Ġbetween Ġpositive Ġand Ġnegative Ġemotion Ġ. ĠThese Ġpoles Ġcorrespond Ġto Ġagreement Ġand Ġdisagreement , Ġor Ġacceptance Ġand Ġdisapproval , Ġfor Ġcandidates Ġand Ġproducts Ġrep sect ively ĠB IB REF 6 Ġ. </s> Beyond Ġpol arity Ġ( also Ġknown Ġas Ġpure Ġemotion ) Ġsentiment Ġanalysis Ġaims Ġto Ġuncover Ġan Ġexact Ġemotion , Ġbased Ġon Ġemotional Ġtheories ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠApplications Ġsuch Ġas Ġsentiment Ġtracking , Ġmarketing , Ġtext Ġcorrection , Ġand Ġtext Ġto Ġspeech Ġsystems Ġcan Ġbe Ġimproved Ġwith Ġthe Ġuse Ġof Ġdistinct Ġemotion Ġlex icons . </s> However , Ġbeyond Ġpol arity Ġstudies Ġacquire Ġlex icons Ġbased Ġon Ġa Ġset Ġof Ġstrict Ġrules , Ġand Ġthe Ġevaluation Ġof Ġexperts . ĠThese Ġlex icons Ġuse Ġonly Ġa Ġsingle Ġemotion Ġper Ġterm ĠB IB REF 9 Ġ. ĠThe Ġproblems Ġof Ġthese Ġapproaches Ġis Ġthe Ġlack Ġof Ġuniform ity Ġand Ġcontribution Ġfreedom Ġwhen Ġrelying Ġon Ġgold Ġstandards , Ġand Ġhigh Ġcosts Ġwith Ġlow Ġscal ability Ġwhen Ġemploying Ġexperts . ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġapplications Ġthat Ġonly Ġrely Ġon Ġexperts Ġare Ġless Ġcomprehensive , Ġrestricted , Ġand Ġnot Ġscalable , Ġcompared Ġto Ġcrowds ourced ĠN LP Ġapplications ĠB IB REF 10 Ġ. </s> This Ġpaper Ġpresents Ġour Ġapproach Ġfor Ġthe Ġacquisition Ġof Ġa Ġmultic lass Ġand Ġscalable Ġcrowds ourced Ġpure Ġemotion Ġlex icon Ġ( P EL ), Ġbased Ġon ĠPl utch ik 's Ġeight Ġbasic Ġemotions . ĠFurthermore , Ġthe Ġcrowd Ġis Ġalso Ġresponsible Ġfor Ġidentifying Ġlinguistic Ġelements , Ġnamely Ġintens ifiers , Ġneg ators , Ġand Ġstop Ġwords . ĠUsually Ġthese Ġelements Ġare Ġpooled Ġfrom Ġexisting Ġlists ĠB IB REF 11 Ġcreated Ġby Ġexperts . ĠWe Ġalso Ġintroduce Ġa Ġworker Ġfiltering Ġmethod Ġto Ġidentify Ġand Ġexclude Ġdishonest Ġor Ġspam ming Ġcontributors , Ġthat Ġdoesn 't Ġrequire Ġgold Ġstandards . ĠOur Ġgoal Ġis Ġto Ġmaintain Ġan Ġend Ġto Ġend Ġautomated Ġwork - flow Ġfor Ġa Ġcrowds ourced Ġ( ann otation Ġand Ġevaluation Ġwise ) Ġlex icon Ġacquisition Ġprocess . ĠTherefore , Ġto Ġhighlight Ġcrowd 's Ġperformance Ġon Ġevaluation , Ġwe Ġcompare Ġevaluations Ġfrom Ġlinguistic Ġexperts Ġand Ġthe Ġcrowd Ġitself . </s> Related ĠWork </s> According Ġto ĠB IB REF 12 Ġ, Ġan Ġemotion Ġis Ġdefined Ġwith Ġreference Ġto Ġa Ġlist . ĠEk am Ġet Ġal . ĠB IB REF 8 Ġproposed Ġthe Ġsix Ġbasic Ġemotions Ġjoy , Ġanger , Ġfear , Ġsadness , Ġdisgust , Ġand Ġsurprise . ĠYears Ġlater , ĠPl utch ik ĠB IB REF 7 Ġproposed Ġthe Ġaddition Ġof Ġtrust Ġand Ġanticipation Ġas Ġbasic Ġemotions , Ġand Ġpresented Ġa Ġcirc ump lex Ġmodel Ġof Ġemotions Ġas Ġseen Ġin ĠFigure ĠFIG REF 1 Ġ, Ġwhich Ġdefines Ġemotional Ġcontradictions Ġand Ġsome Ġof Ġthe Ġpossible Ġcombinations . </s> Sent iment Ġanalysis Ġaims Ġto Ġclassify Ġinformation Ġbased Ġon Ġthe Ġemotion Ġconveyed . ĠDepending Ġon Ġthe Ġnumber Ġof Ġclasses / em otions Ġrequired , Ġwe Ġcan Ġseparate Ġthe Ġanalysis Ġinto : Ġpol arity Ġand Ġbeyond Ġpol arity . </s> P olar ity Ġsentiment Ġanalysis Ġstudies Ġdefine Ġtwo Ġopposite Ġemotional Ġstates , Ġpositive Ġand Ġnegative , Ġor Ġgood Ġand Ġbad , Ġwith Ġthe Ġaddition Ġof Ġa Ġneutral Ġstate . ĠFurthermore , Ġsome Ġresearchers Ġhave Ġclassified Ġinformation Ġon Ġlevels Ġfor Ġeach Ġpole ( e . g . Ġvery Ġpositive , Ġpositive , Ġneutral , Ġnegative , Ġvery Ġnegative Ġetc .), Ġalso Ġknown Ġas Ġfine Ġgr ained Ġsentiment Ġanalysis ĠB IB REF 13 Ġ. </s> Beyond Ġpol arity , Ġalso Ġknown Ġas Ġpure Ġemotion , Ġsentiment Ġanalysis Ġis Ġa Ġmore Ġrefined Ġapproach Ġto Ġthe Ġsame Ġproblem Ġwith Ġa Ġwider Ġrange Ġof Ġpossible Ġemotion Ġclasses , Ġsee ĠFigure ĠFIG REF 1 Ġ. ĠEssentially , Ġany Ġsentiment Ġanalysis Ġthat Ġinvolves Ġspecific Ġemotional Ġlab elling , Ġis Ġconsidered Ġas Ġa Ġbeyond Ġpol arity Ġanalysis . ĠExamples Ġof Ġemotional Ġlabels Ġmight Ġbe Ġ- but Ġare Ġnot Ġlimited Ġto - : Ġsadness , Ġboredom , Ġjoy , Ġsadness , Ġsurprise , Ġanger , Ġfear , Ġdisgust Ġetc . </s> As Ġdiscussed Ġin ĠSection Ġ1 , Ġone Ġof Ġthe Ġcore Ġtasks Ġof Ġsentiment Ġanalysis Ġis Ġlex icon Ġacquisition . ĠA Ġlex icon Ġcan Ġbe Ġacquired Ġthrough Ġmanual Ġor Ġautomatic Ġannotation . ĠHowever , Ġnatural Ġlanguage Ġhas Ġa Ġvery Ġsubjective Ġnature ĠB IB REF 14 Ġwhich Ġsignificantly Ġinhibits Ġautomated Ġsentiment Ġlex icon Ġa q cu isition Ġmethods Ġfrom Ġachieving Ġrelevance Ġequal Ġto Ġmanual Ġmethods ĠB IB REF 15 Ġ. ĠThus Ġa Ġlot Ġof Ġresearchers Ġchoose Ġto Ġmanually Ġannot ate Ġtheir Ġterm Ġcorpor a ĠB IB REF 16 Ġ, Ġor Ġuse Ġestablished Ġlex icon Ġsuch Ġas ĠWord Net , ĠSent i Word Net , Ġand Ġvarious Ġother Ġlex icons ĠB IB REF 13 Ġ. ĠOther Ġstudies Ġcombine Ġmanual Ġlabeling Ġor Ġmachine Ġlearning Ġwith Ġlex icons ĠB IB REF 17 Ġ. </s> Man ual Ġlex icon Ġacquisition Ġis Ġconstrained Ġby Ġthe Ġnumber Ġof Ġpeople Ġcontributing Ġto Ġthe Ġtask , Ġand Ġthe Ġnumber Ġof Ġannotations Ġfrom Ġeach Ġparticipant . ĠThese Ġconstraints Ġcan Ġbe Ġeliminated Ġby Ġincreasing Ġthe Ġnumber Ġof Ġpeople Ġinvolved , Ġfor Ġinstance , Ġby Ġusing Ġcrowds ourcing ĠB IB REF 18 Ġ. ĠAmazon 's ĠMechanical ĠTurk Ġ( MT ur k ) Ġis Ġa Ġcrowds ourcing Ġplatform Ġfrequently Ġused Ġfor Ġpol arity Ġsentiment Ġlex icon Ġacquisition Ġvia Ġcrowds ourcing ĠB IB REF 19 Ġ. ĠM Tur k Ġis Ġalso Ġused , Ġfor Ġthe Ġannotation Ġof Ġone Ġthousand Ġtweets Ġin ĠB IB REF 20 Ġ, Ġten Ġthousand Ġterms Ġin ĠB IB REF 21 Ġwith Ġgold Ġstandards , Ġand Ġthe Ġannotation Ġof Ġninety Ġfive Ġemot icons Ġout Ġof Ġone Ġthousand Ġtotal Ġemot icons Ġfound Ġin ĠB IB REF 22 Ġ. ĠWhile ĠB IB REF 23 Ġhad Ġone Ġthousand Ġfour Ġhundred Ġterms Ġlabelled Ġwith Ġa Ġsupervised Ġmachine Ġlearning Ġand Ġcrowd Ġvalid ators . ĠThe Ġchallenge Ġis Ġto Ġintroduce Ġa Ġwork - flow Ġthat Ġis Ġscalable , Ġun super vised Ġand Ġapplicable Ġto Ġdifferent Ġinformation Ġtypes . </s> The Ġsecond Ġcore Ġpart Ġin Ġsentiment Ġanalysis , Ġis Ġsentiment Ġclassification . ĠA Ġclassification Ġthat Ġoccurs Ġat Ġphrase / sent ence / sub mission Ġlevel , Ġand Ġis Ġusually Ġbased Ġon Ġthe Ġaggregation Ġof Ġthe Ġterm 's Ġlabeled Ġemotions . ĠAs Ġwith Ġlex icon Ġa q cu isition , Ġthe Ġclassification Ġtask Ġcan Ġbe Ġautomated ĠB IB REF 13 Ġor Ġperformed Ġmanually ĠB IB REF 24 Ġ. </s> Regardless Ġof Ġmanual Ġor Ġautomated Ġsentiment Ġclassification , Ġon Ġtextual Ġinformation Ġscenarios , Ġterm Ġand Ġphrase Ġsentiment Ġis Ġthe Ġmain Ġinput Ġof Ġthe Ġclassification Ġmethod . ĠIn Ġsome Ġcases Ġthe Ġdecision Ġmight Ġbe Ġtotally Ġdifferent Ġfrom Ġthe Ġindividual Ġterm Ġemotion , Ġleading Ġto Ġrel abel ing Ġof Ġthe Ġterms Ġthemselves ĠB IB REF 25 Ġ. ĠMan ually Ġlabelled Ġclassification Ġcan Ġachieve Ġhigh Ġrelevance , Ġbut Ġit Ġrequires Ġadditional Ġresources , Ġand Ġis Ġnot Ġeasily Ġscalable . ĠOn Ġthe Ġother Ġhand , Ġautomated Ġprocesses Ġare Ġscalable Ġbut Ġwith Ġlower Ġrelevance ĠB IB REF 24 Ġ. </s> Our Ġapproach </s> Our Ġaim Ġis Ġto Ġcreate Ġan Ġend Ġto Ġend Ġautomated Ġwork - flow Ġfor Ġthe Ġcreation , Ġevaluation Ġand Ġenrichment Ġof Ġa Ġpure Ġemotion Ġlex icon . ĠThe Ġwork - flow , ĠFigure ĠFIG REF 3 Ġ, Ġcan Ġbe Ġseparated Ġin Ġtwo Ġmain Ġcomponents . ĠPre - processing Ġis Ġthe Ġun super vised Ġprocess Ġby Ġwhich Ġwe Ġderive Ġthe Ġlex icon Ġterms Ġfrom Ġany Ġtextual Ġresource , Ġwhile Ġcrowds ourcing Ġdeals Ġwith Ġthe Ġcrowds ourcing Ġaspect Ġof Ġthe Ġlex icon . ĠThe ĠPure ĠEm otions ĠLex icon Ġincludes Ġemotional Ġterm Ġgroups , Ġintens ifiers Ġand Ġneg ators , Ġand Ġstop Ġwords . </s> Pre - processing Ġis Ġcomprised Ġof Ġ3 Ġun super vised Ġsteps , Ġtoken ization , Ġstemming Ġand Ġspell Ġcheck . ĠText ual Ġcontent Ġis Ġtoken ized Ġas Ġun i - gram s , Ġstemmed Ġbased Ġon Ġtheir Ġrooted Ġand Ġchecked Ġfor Ġspelling . ĠThe Ġresulting Ġstems Ġalong Ġwith Ġtheir Ġstem Ġgroups Ġare Ġstored Ġin Ġa Ġlex icon Ġdatabase . ĠCrowd s ourcing Ġis Ġusing Ġthe Ġlex icon Ġdatabase Ġand Ġthe Ġcrowd Ġto Ġannot ate Ġeach Ġentry Ġin Ġthe Ġdatabase . ĠParticipants Ġsubmit Ġtheir Ġanswers Ġthat Ġgo Ġthrough Ġa Ġfiltering Ġprocess . ĠIf Ġthe Ġanswers Ġare Ġconsidered Ġvalid , Ġthey Ġupdate Ġthe Ġlex icon Ġentries . ĠThe Ġcrowd Ġalso Ġevaluates Ġexisting Ġannotations , Ġto Ġdetermine Ġthe Ġlex icon Ġquality . ĠAs Ġcrowd Ġevaluation Ġmethods Ġare Ġnew Ġin Ġlex icon Ġacquisition Ġtasks , Ġwe Ġcompare Ġcrowd Ġevaluations Ġto Ġthose Ġof Ġexpert Ġlingu ists . </s> Data </s> During ĠJanuary Ġ2017 , Ġwe Ġperformed Ġa Ġkeyword Ġbased Ġcrawl Ġfor Ġarticles Ġand Ġcomments Ġin Ġthe ĠEurope Ġsubreddit Ġand Ġtweets Ġin ĠTwitter , Ġwhich Ġcontained Ġthe Ġword Ġ" Brexit ". ĠThe Ġuse Ġof Ġa Ġpolitical Ġand Ġcontroversial Ġterm Ġin Ġthe Ġquery Ġis Ġdeliberate , Ġto Ġcapture Ġthe Ġemotional Ġdiversity Ġof Ġa Ġpolitically Ġoriented Ġcorpus . </s> We Ġcrawled Ġone Ġhundred Ġarticles Ġfrom ĠReddit , Ġwith Ġmore Ġthan Ġforty Ġthousand Ġcomments Ġand Ġmore Ġthan Ġthree Ġthousand Ġtweets . ĠFor Ġthe ĠReddit Ġdata , Ġwe Ġcollected Ġinformation Ġon Ġlocation , Ġtime , Ġand Ġthe Ġnumber Ġof Ġup votes . ĠFor Ġthe ĠTwitter Ġdata , Ġwe Ġstored Ġthe Ġnumber Ġor Ġre - t we ets Ġand Ġfavourites , Ġtime Ġand Ġlocation Ġinformation . </s> Our Ġfocus Ġis Ġon Ġsingle Ġterm Ġ( also Ġknown Ġas Ġun ig ram ) Ġsentiment , Ġthus Ġposts Ġin Ġboth Ġnetworks Ġwere Ġprocessed Ġto Ġa Ġsingle Ġterm Ġlist . ĠIn Ġtotal , Ġthe Ġnumber Ġof Ġunique Ġterms Ġin Ġour Ġcorpus Ġwas Ġ30 227 . ĠBased Ġon Ġthe Ġenchant Ġpython Ġlibrary , Ġused Ġin ĠB IB REF 26 Ġ, Ġand Ġthe Ġsupported ĠGreat ĠBritish ĠEnglish Ġdictionary , Ġ19 193 Ġwere Ġvalidated Ġand Ġ110 34 Ġwere Ġinvalid ated . ĠOur Ġanalysis Ġwill Ġfocus Ġon Ġthe Ġ19 193 Ġvalid Ġterms , Ġthat Ġfollow ĠZip f 's ĠLaw Ġwith Ġscaling - law Ġcoefficient ĠIN LINE FORM 0 Ġis Ġa Ġgood Ġfit . </s> After Ġvalidation , Ġterms Ġwere Ġstemmed Ġwith ĠPorter ĠSt em ming ĠAl gorithm ĠB IB REF 27 Ġ. ĠSt em ming Ġidentified Ġ10 953 Ġdistinct Ġterm Ġgroups Ġwith Ġone Ġor Ġmore Ġterms . ĠStop - words , Ġintens ifiers Ġand Ġneg ators Ġare Ġalso Ġincluded Ġin Ġthe Ġvalid Ġterm Ġgroups . ĠBoth Ġterm Ġvalidation Ġand Ġstemming Ġare Ġun super vised , Ġsince Ġour Ġgoal Ġis Ġto Ġmaintain Ġscal ability Ġin Ġbeyond Ġpol arity Ġlex icon Ġacquisition Ġand Ġsentiment Ġclassification . </s> C rowd s ourcing </s> The Ġcrowds ourcing Ġtask , Ġhosted Ġin ĠCrowd Fl ower , Ġrequired Ġcontributors Ġto Ġlabel Ġterm Ġgroups Ġin Ġthree Ġdifferent Ġmain Ġclasses , Ġemotion , Ġintens ifier Ġand Ġnone , Ġwithout Ġa Ġgolden Ġstandard , Ġrules Ġor Ġany Ġparticipation Ġrestrictions Ġ. ĠEm otion Ġlab elling Ġincluded Ġthe Ġ8 Ġbasic Ġemotions Ġas Ġdefined Ġby ĠPl utch ik . ĠInt ens ifier Ġclass Ġincluded Ġintens ifiers Ġand Ġneg ators . ĠFinally , Ġnone Ġreferred Ġto Ġstop - words Ġor Ġwords Ġwith Ġno Ġparticular Ġemotion . </s> Each Ġof Ġthe Ġeleven Ġoptions Ġfor Ġthe Ġmain Ġclasses , Ġwill Ġbe Ġreferred Ġto Ġas Ġ" sub class ". ĠTerms Ġare Ġgrouped Ġbased Ġon Ġtheir Ġstem . ĠEach Ġterm Ġgroup Ġhas Ġa Ġmain Ġannotation Ġclass Ġdefined Ġby Ġmajority , Ġand Ġseveral Ġsub Ġannotation Ġclasses , Ġdefined Ġby Ġthe Ġnon Ġmajority Ġannotations . ĠHowever , Ġto Ġaid Ġmulti Ġclass Ġanalysis Ġof Ġthe Ġresults , Ġevery Ġannotation Ġis Ġlogged Ġin Ġthe Ġlex icon Ġdatabase . </s> Task ĠInterface </s> The Ġtask Ġinterface Ġwas Ġthe Ġresult Ġof Ġseveral Ġexperiments . ĠThree Ġmajor Ġchanges Ġimplemented Ġbased Ġon Ġthese Ġexperimental Ġinterfaces Ġwere : Ġthe Ġsimpl ification Ġof Ġthe Ġtask Ġquestion , ĠFigure ĠFIG REF 8 Ġ, Ġthe Ġinclusion Ġof Ġonly Ġthree Ġmain Ġclasses , Ġand Ġthe Ġreplacement Ġof Ġwords Ġpositive Ġand Ġnegative , Ġwith Ġampl ifying Ġand Ġweakening Ġin Ġthe Ġintens ifying Ġclass Ġoptions . ĠAll Ġexperiments Ġand Ġthe Ġannotation Ġtask Ġrequired Ġhighly Ġexperienced Ġcontributors , Ġas Ġdefined Ġby ĠCrowd flower Ġplatform . </s> As Ġseen Ġin ĠFigure ĠFIG REF 8 Ġcontributors Ġselect Ġone Ġof Ġthe Ġthree Ġchoices . ĠIf Ġthey Ġchoose ĠEm otion Ġev oking , Ġthey Ġare Ġpresented Ġwith Ġa Ġdrop - down Ġmenu Ġto Ġchoose Ġfrom Ġthe Ġeight Ġbasic Ġemotions . ĠSimilarly , Ġif Ġthey Ġselect ĠInt ens ifying Ġcontext Ġthey Ġhave Ġto Ġspecify Ġwhether Ġit Ġwas ĠAmpl ifying Ġor ĠWe akening Ġthe Ġcontext , Ġessentially Ġannot ating Ġthe Ġintens ifiers Ġand Ġneg ators . ĠFinally Ġif Ġthey Ġchoose ĠNone Ġthey Ġare Ġpresented Ġwith Ġthe Ġnext Ġterm Ġgroup . ĠTo Ġassist Ġcontributors Ġwith Ġterm Ġdefinitions , Ġevery Ġterm Ġgroup Ġhad Ġa Ġhyper link Ġto Ġan ĠEnglish Ġdictionary . </s> C rowd </s> More Ġthan Ġone Ġhundred Ġeighty Ġcontributors Ġperformed Ġeighty Ġthousand Ġannotations . ĠBy Ġdesign , Ġeach Ġuser Ġcould Ġnot Ġperform Ġmore Ġthan Ġ660 Ġunique Ġannotations , Ġexcluding Ġthe Ġassessment Ġquestions , Ġto Ġengage Ġat Ġleast Ġ100 Ġcontributors . ĠMost Ġof Ġthe Ġworkers Ġannot ated Ġthe Ġmaximum Ġallowed Ġterms , Ġthe Ġfirst Ġhalf Ġof Ġworkers Ġannot ated Ġ15 % Ġof Ġthe Ġterm Ġgroups Ġin Ġour Ġcorpus , Ġwhile Ġthe Ġsecond Ġhalf Ġof Ġworkers Ġannot ate Ġthe Ġrest Ġ85 %. ĠThe Ġsimplicity Ġof Ġthe Ġtask Ġresulted Ġin Ġhigh Ġoverall Ġworker Ġengagement , Ġwith Ġmean Ġand Ġmedian Ġannotations Ġper Ġworker , Ġat Ġ429 Ġand Ġ580 Ġrespectively . </s> Ass essment </s> Based Ġon Ġa Ġset Ġof Ġexperiments , Ġwe Ġidentified Ġ136 Ġterm Ġgroups Ġthat Ġwould Ġtest Ġthe Ġability Ġof Ġa Ġcontributor Ġin Ġall Ġof Ġthe Ġthree Ġmain Ġclasses , Ġemotion Ġev oking , Ġintens ifying Ġcontext , Ġand Ġnone . ĠAs Ġthe Ġassessment Ġterm Ġgroups Ġhad Ġmore Ġthan Ġten Ġthousand Ġannotations , Ġwe Ġanalyse Ġit Ġseparately Ġfrom Ġthe Ġlex icon . </s> In Ġorder Ġfor Ġa Ġworker Ġto Ġgain Ġthe Ġability Ġto Ġcontribute Ġto Ġthe Ġcrowds ourcing Ġtask Ġand Ġeventually Ġget Ġpaid , Ġhe / she Ġhad Ġto Ġproperly Ġannot ate Ġ80 % Ġof Ġthe Ġassessment Ġterm Ġgroups Ġencountered . ĠThe Ġannotations Ġshould Ġbe Ġwithin Ġthe Ġdominant Ġclasses , Ġand Ġnot Ġsub classes , Ġas Ġdefined Ġfrom Ġthe Ġassessment Ġannot ators . ĠE . g ., Ġfor Ġan Ġassessment Ġterm Ġgroup Ġthat Ġreceived Ġ100 Ġannotations Ġin Ġvarious Ġemotions , Ġwe Ġcheck Ġif Ġthe Ġworker Ġannot ates Ġthe Ġterm Ġgroup Ġas Ġemotion Ġev oking . </s> Let ĠIN LINE FORM 0 Ġbe Ġthe Ġset Ġof Ġworkers ĠIN LINE FORM 1 Ġand Ġand ĠIN LINE FORM 2 Ġthe Ġset Ġof Ġeleven ĠIN LINE FORM 3 Ġsub classes : Ġeight Ġemotions Ġ, Ġtwo Ġintens ifiers , Ġand Ġnone Ġof Ġthe Ġformer . ĠWe Ġdefine ĠIN LINE FORM 4 Ġas Ġthe Ġnumber Ġof Ġtotal Ġannotations Ġfor Ġeach Ġworker ĠIN LINE FORM 5 Ġ. ĠThen : ĠDIS PLAY FORM 0 </s> We Ġdefine ĠIN LINE FORM 0 Ġbe Ġthe Ġset Ġof Ġworkers ĠIN LINE FORM 1 Ġin Ġthe Ġassessment Ġprocess , ĠIN LINE FORM 2 Ġthe Ġset Ġof Ġworkers ĠIN LINE FORM 3 Ġin Ġthe Ġacquisition Ġprocess . ĠThen , Ġfor ĠIN LINE FORM 4 Ġwe Ġdefine : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> and : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> The Ġoptimal ĠIN LINE FORM 0 Ġis Ġfound Ġfor ĠIN LINE FORM 1 Ġ. ĠFor Ġthis Ġstudy , Ġthe Ġoptimal Ġfiltering Ġpercentage Ġwas Ġfound Ġat Ġ40 %, ĠIN LINE FORM 2 Ġ. </s> Work ers Ġfrom ĠIndia Ġand ĠVenezuela , Ġwho Ġcontributed Ġ92 % Ġof Ġthe Ġtask , Ġhave Ġannot ated Ġmore Ġthan Ġ30 % Ġof Ġthe Ġterm Ġgroups Ġwith Ġjoy . ĠHowever , Ġannotations Ġfrom Ġcountries Ġwith Ġmore Ġthan Ġ300 Ġannotations , Ġdon 't Ġfollow Ġthe Ġsame Ġdistribution . ĠSpecifically , Ġworkers Ġfrom ĠPhilippines , ĠUnited ĠStates , ĠColombia , ĠPoland , ĠUnited ĠKingdom , ĠRussia , Ġand ĠEgypt , Ġperformed Ġa Ġsmoother Ġdistributed Ġemotion Ġannotation . ĠIn Ġcomparison , Ġthe Ġmost Ġannot ated Ġemotion Ġin ĠB IB REF 21 Ġwas Ġfear Ġin Ġ18 % Ġof Ġthe Ġtotal Ġterms . </s> By Ġfurther Ġanalys ing Ġworker Ġannotation Ġdistribution , Ġwe Ġidentified Ġworkers Ġthat Ġhad Ġa Ġsignificant Ġfraction Ġof Ġtheir Ġtotal Ġannotations Ġin Ġa Ġsingle Ġsubclass . ĠE . g . Ġone Ġspecific Ġworker Ġannot ated Ġ99 % Ġof Ġthe Ġassessment Ġterm Ġgroups Ġhe Ġencountered Ġas Ġjoy . ĠDish on esty Ġor Ġspam ming Ġis Ġa Ġknown Ġproblem Ġin Ġcrowds ourcing ĠB IB REF 28 Ġand Ġmultiple Ġproposed Ġsolutions Ġexist ĠB IB REF 28 Ġ, Ġbut Ġthey Ġrequire Ġgold Ġstandards Ġor Ġobjective Ġcrowds ourcing Ġtasks . </s> As Ġwe Ġdon 't Ġhave Ġa Ġgold Ġstandard , Ġand Ġthe Ġtask Ġis Ġmore Ġsubjective , Ġthese Ġspam ming Ġelimination Ġmethods Ġare Ġnot Ġapplicable . ĠOur Ġsolution Ġis Ġthe Ġimplementation Ġof Ġa Ġfast Ġand Ġefficient Ġfilter , Ġwhich Ġonly Ġrelies Ġon Ġthe Ġobtained Ġannotations Ġand Ġthe Ġassessment . ĠIf Ġworkers ' Ġanswers Ġwere Ġabove Ġa Ġcertain Ġpercentage Ġon Ġa Ġsingle Ġsubclass Ġfor Ġboth Ġthe Ġassessment Ġand Ġthe Ġannotation Ġprocess , Ġthen Ġthe Ġuser Ġwould Ġbe Ġflagged Ġas Ġdishonest Ġand Ġthe Ġtotal Ġof Ġtheir Ġannotations Ġwould Ġbe Ġdiscarded . ĠThis Ġrule Ġwas Ġapplied Ġto Ġthe Ġwhole Ġrange Ġof Ġpossible Ġanswers , Ġincluding Ġthe Ġ8 Ġemotions , Ġ2 Ġintens ifiers Ġand Ġ" none ". </s> Prior Ġto Ġimplementing Ġthe Ġfilter , Ġwe Ġanalysed Ġhow Ġit Ġwould Ġaffect Ġthe Ġnumber Ġof Ġeligible , Ġnot Ġconsidered Ġas Ġspam ming , Ġworkers . ĠThe Ġthick Ġline Ġin ĠFigure ĠFIG REF 22 Ġshows Ġthe Ġpercentage Ġduring Ġthe Ġassessment Ġterm Ġgroups , Ġand Ġthe Ġdotted Ġline Ġshows Ġthe Ġpercentage Ġduring Ġthe Ġlex icon Ġacquisition . </s> The Ġhigher Ġthe Ġsingle Ġannotation Ġpercentage , Ġthe Ġhigher Ġthe Ġcertainty Ġof Ġspam ming Ġbehaviour . ĠIn Ġlarge Ġpart , Ġthe Ġexclusion Ġrate Ġwas Ġsimilar Ġfor Ġboth Ġthe Ġassessment Ġand Ġthe Ġlex icon Ġannotations . ĠA Ġnumber Ġof Ġworkers Ġhad Ġa Ġmore Ġcautious Ġbehaviour Ġin Ġthe Ġtest Ġquestions , Ġresulting Ġin Ġreduced Ġpercentage Ġof Ġexclus ions Ġduring Ġassessment Ġprocess . ĠThis Ġbehaviour Ġis Ġjustified , Ġas Ġthe Ġfirst Ġset Ġof Ġquestions Ġencounter Ġby Ġa Ġworker Ġare Ġknowingly Ġa Ġset Ġof Ġassessment Ġquestions . </s> Each Ġassessment Ġterm Ġgroup Ġwas Ġannot ated Ġmore Ġthan Ġ120 Ġtimes , Ġby Ġ187 Ġannot ators . ĠThis Ġis Ġa Ġcritical Ġmass Ġof Ġcontributors Ġand Ġprovides Ġvaluable Ġfindings Ġwith Ġregards Ġto Ġtask . ĠThese Ġare : </s> Work ers Ġrarely Ġclicked Ġthe Ġinformative Ġdictionary Ġlink . ĠAs Ġa Ġresult , Ġthey Ġwould Ġannot ate Ġemotional Ġwords Ġto Ġnone , Ġprobably Ġdue Ġto Ġmisinterpret ation . ĠWe Ġavoided Ġthe Ġdirect Ġinclusion Ġof Ġthe Ġdictionary Ġdefinition , Ġas Ġit Ġcould Ġbe Ġconsidered Ġas Ġa Ġform Ġof Ġleverage . ĠE . g . Ġ" ve hement , Ġhal cy on " Ġ, Ġtwo Ġ- un common - Ġemotion Ġb aring Ġwords , Ġwere Ġboth Ġannot ated Ġas Ġnone , Ġand Ġless Ġthan Ġ0 . 2 % Ġof Ġthe Ġworkers Ġ( on Ġaverage ) Ġclicked Ġa Ġdictionary Ġlink . </s> The Ġconcept Ġof Ġintens ifiers Ġis Ġunderstandable Ġbut Ġrequires Ġcritical Ġmass ĠB IB REF 29 Ġ. ĠA Ġsmall Ġnumber Ġof Ġannot ators Ġwould Ġinitially Ġannot ate Ġintens ifiers / neg ators Ġwith Ġan Ġemotion , Ġbut Ġthe Ġdistribution Ġwould Ġslowly Ġshift Ġtowards Ġthe Ġcorrect Ġclass . ĠE . g . Ġ" red uce , Ġlittle , Ġplethora " Ġ, Ġwere Ġinitially Ġannot ated Ġas Ġsad Ġsad Ġjoy , Ġbut Ġafter Ġtens Ġof Ġannotations Ġthey Ġwere Ġannot ated Ġas Ġweakening Ġweakening Ġintens ifying . </s> All Ġwords Ġshould Ġbe Ġevaluated , Ġeven Ġthose Ġthat Ġseemingly Ġdon 't Ġcarry Ġa Ġspecific Ġemotion . ĠAs Ġtimes Ġchange , Ġwords Ġand Ġemotions Ġacquire Ġnew Ġlinks . ĠE . g . Ġ" anti , Ġserious " Ġ, Ġwere Ġboth Ġannot ated Ġas Ġfear Ġev oking Ġwith Ġa Ġgreat Ġemotional Ġdiversity . </s> Lex icon ĠAnalysis </s> The Ġlex icon Ġ( will Ġbe Ġreferred Ġas Ġsimply Ġ" P EL ") Ġis Ġcreated Ġafter Ġthe Ġexclusion Ġof Ġannotations Ġfollowing Ġthe Ġ40 % Ġsingle Ġannotation Ġfiltering Ġcheck . ĠWe Ġreceived Ġmore Ġthan Ġseventy Ġthousands Ġannotations Ġfor Ġ10 593 Ġterm Ġgroups , Ġof Ġthose Ġonly Ġ22 Ġthousand Ġannotations Ġfor Ġ97 37 Ġterm Ġgroups Ġare Ġincluded Ġin Ġthe Ġfinal Ġlex icon , Ġas Ġa Ġresult Ġof Ġfiltering . ĠEach Ġterm Ġgroup Ġhad Ġa Ġmean Ġ2 . 3 Ġannotations Ġfrom Ġa Ġtotal Ġof Ġ95 Ġdifferent Ġannot ators . ĠAlthough Ġthe Ġnumber Ġof Ġmean Ġannotations Ġin Ġlex icon Ġis Ġless Ġthan Ġhalf Ġthe Ġmean Ġannotations Ġin Ġthe Ġunf il tered Ġcorpus , Ġthe ĠP EL Ġannotations Ġare Ġconsidered Ġof Ġhigher Ġquality . </s> Each Ġlex icon Ġterm Ġgroup Ġhas Ġmultiple Ġsubclass Ġannotations , Ġand Ġthe Ġmain Ġsubclass Ġis Ġdefined Ġby Ġmajority . ĠEven Ġafter Ġfiltering , Ġthe Ġdominant Ġemotion Ġin Ġour Ġlex icon Ġis Ġjoy , Ġwhile Ġthe Ġleast Ġannot ated Ġemotion Ġis Ġdisgust . ĠAdditionally , Ġ148 Ġterms Ġwere Ġannot ated Ġas Ġintens ifiers , Ġ43 Ġterms Ġas Ġneg ators , Ġand Ġ68 01 Ġterms Ġas Ġnone . ĠA Ġsample Ġof Ġfive Ġterms Ġfor Ġeach Ġof Ġthe Ġthree Ġsub classes Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 27 Ġ. ĠThe Ġfull Ġlex icon Ġcan Ġbe Ġfound Ġon Ġgithub . </s> Int ens ifiers Ġand Ġneg ators Ġserve Ġas Ġmodifiers Ġto Ġthe Ġemotional Ġcontext Ġof Ġa Ġword . ĠWorkers Ġidentified Ġmostly Ġvalid Ġintens ifiers Ġand Ġneg ators Ġthat Ġcan Ġmodify Ġemotion Ġev oking Ġwords , Ġin Ġthe Ġabsence Ġof Ġcontext . ĠJudging Ġfrom Ġthe Ġreceived Ġannotations , Ġthere Ġis Ġroom Ġfor Ġimprovement Ġon Ġthe Ġdescription Ġof Ġthe Ġintens ifier Ġclass Ġand Ġthe Ġprovided Ġexamples , Ġas Ġa Ġnumber Ġof Ġnon Ġintens ifying Ġwords Ġwere Ġfalsely Ġannot ated . </s> Ter ms Ġin Ġour Ġlex icon Ġare Ġgrouped Ġbased Ġon Ġtheir Ġstem . ĠSt em ming Ġsignificantly Ġreduced Ġcost Ġ( by Ġhalf ) Ġand Ġtime - required Ġfor Ġthe Ġtask . ĠGroup ing Ġterms Ġmay Ġcreate Ġunnecessary Ġmulti - class Ġannotations Ġagreements , Ġfor Ġterms Ġin Ġthe Ġsame Ġterm Ġgroup Ġthat Ġmight Ġhave Ġdifferent Ġmeanings . ĠAn notation Ġagreement Ġrefers Ġto Ġequal Ġnumber Ġof Ġannotations Ġin Ġmultiple Ġsub classes Ġor Ġemotions . ĠHowever , Ġthe Ġvast Ġmajority Ġof Ġterm Ġgroups Ġin Ġour Ġlex icon , Ġdon 't Ġdisplay Ġany Ġform Ġof Ġcontradict ing Ġannotation . ĠCont rad ict ing Ġemotions Ġare Ġportrayed Ġin Ġopposite Ġedges Ġof Ġthe Ġcirc ump lex ĠFigure ĠFIG REF 1 Ġ, Ġwhile Ġemotional Ġcombinations Ġare Ġdec ribed Ġin ĠB IB REF 7 Ġ. ĠIn Ġthe Ġlex icon , Ġonly Ġ21 % Ġand Ġ20 % Ġof Ġthe Ġterm Ġgroups Ġhad Ġa Ġsubclass Ġand Ġan Ġemotional Ġagreement Ġrespectively . ĠWith Ġregards Ġto Ġemotion , Ġcontradict ing Ġor Ġmulti - em otion Ġagreement , Ġcould Ġbe Ġobserved Ġin Ġonly Ġ8 . 6 % Ġof Ġthe Ġtotal Ġterm Ġgroups . </s> Let ĠIN LINE FORM 0 Ġbe Ġa Ġterm Ġgroup Ġin Ġthe Ġlex icon Ġand ĠIN LINE FORM 1 Ġthe Ġset Ġof Ġeleven ĠIN LINE FORM 2 Ġsub classes : Ġeight Ġemotions Ġ, Ġtwo Ġintens ifiers , Ġand Ġnone Ġof Ġthe Ġformer . ĠWe Ġdefine ĠIN LINE FORM 3 Ġas Ġthe Ġnumber Ġof Ġannotations Ġfor Ġeach Ġterm Ġgroup ĠIN LINE FORM 4 Ġ. ĠFor Ġeach ĠIN LINE FORM 5 Ġ, Ġthe Ġannotations Ġfor Ġemotion Ġsub classes Ġare ĠIN LINE FORM 6 Ġ, Ġthe Ġannotations Ġfor Ġintens ifying Ġsub classes Ġare ĠIN LINE FORM 7 Ġ, Ġand Ġthe Ġnumber Ġof Ġnone Ġannotations Ġis ĠIN LINE FORM 8 Ġ. </s> Therefore , Ġeach ĠIN LINE FORM 0 Ġcan Ġhave Ġan Ġmon oton ically Ġincreasing Ġfinite Ġsequence ĠIN LINE FORM 1 Ġwith ĠIN LINE FORM 2 Ġ, Ġwhere : ĠDIS PLAY FORM 0 </s> We Ġsay Ġthat Ġterm Ġgroup ĠIN LINE FORM 0 Ġhas Ġsubclass Ġagreement Ġif Ġand Ġonly Ġif : ĠDIS PLAY FORM 0 </s> While ĠIN LINE FORM 0 Ġhas Ġemotional Ġagreement Ġif Ġand Ġonly Ġif Ġthere Ġis Ġa Ġsubclass Ġagreement Ġwith Ġthe Ġsequence ĠIN LINE FORM 1 Ġand : ĠDIS PLAY FORM 0 </s> Sub class Ġagreement , Ġrefers Ġto Ġequal Ġannotations , Ġbetween Ġemotional Ġsubclass ( es ) Ġand Ġat Ġleast Ġone Ġnon - em otional Ġsubclass , Ġor Ġbetween Ġmultiple Ġnon - em otional Ġsubclass , ĠEqu ation ĠEQ REF 30 Ġ. ĠOn Ġthe Ġother Ġhand , Ġemotional Ġagreement Ġrefers Ġto Ġmultiple Ġemotion Ġsub classes Ġwith Ġequal Ġannotations , ĠEqu ation ĠEQ REF 31 Ġ. </s> The Ġnumber Ġof Ġsub classes Ġin Ġagreement Ġand Ġthe Ġnumber Ġof Ġterms Ġin Ġa Ġterm Ġgroup Ġare Ġnegatively Ġcorrelated . ĠTerm Ġgroups Ġwith Ġtwo Ġterms Ġappear Ġto Ġhave Ġthe Ġhighest Ġsubclass Ġagreement Ġwith Ġexactly Ġtwo Ġsub classes . ĠThe Ġmost Ġcommon Ġoccurring Ġagreements Ġare Ġsubclass Ġnone Ġpaired Ġwith Ġan Ġemotion , Ġand Ġjoy Ġpaired Ġwith Ġan Ġemotion . ĠThe Ġnumber Ġof Ġmulti - class Ġagreement Ġoccurrences Ġis Ġdispro port ional Ġto Ġthe Ġnumber Ġof Ġterms Ġin Ġa Ġterm Ġgroup . ĠThis Ġis Ġa Ġstrong Ġindication Ġthat Ġstemming Ġdidn 't Ġconfuse Ġworkers . </s> Similarly , Ġfor Ġemotional Ġagreement , Ġthe Ġnumber Ġof Ġoccurrences Ġis Ġdisproportionate Ġto Ġthe Ġnumber Ġof Ġterms Ġin Ġthe Ġterm Ġgroup . ĠFurthermore , Ġemotional Ġagreement Ġappeared Ġin Ġ10 % Ġof Ġthe Ġterm Ġgroups , Ġwhile Ġsubclass Ġagreement Ġwas Ġfound Ġin Ġ20 % Ġof Ġthe Ġterm Ġgroups . ĠIn Ġthe Ġagreement Ġannotations , Ġjoy Ġis Ġthe Ġmost Ġcommon Ġemotion . ĠAccording Ġto ĠPl utch ik 's Ġcirc ump lex ĠFigure ĠFIG REF 1 Ġ, Ġeach Ġemotion Ġhas Ġa Ġcontradict ing Ġone , Ġand Ġpairs Ġof Ġemotions Ġindicate Ġa Ġmore Ġ" complex " Ġemotion . ĠThere Ġare Ġ6 97 Ġemotional Ġagreeing Ġterm Ġgroups , Ġof Ġ14 34 Ġterms , Ġwith Ġexactly Ġtwo Ġemotions . ĠThese Ġemotional Ġdy ads ĠB IB REF 7 Ġcan Ġbe Ġcombined Ġas Ġseen Ġin ĠTable ĠT AB REF 32 Ġ. ĠSimple Ġbasic Ġemotion Ġannotation Ġtasks Ġcan Ġindirectly Ġprovide Ġcomplex Ġemotional Ġannotations . </s> D y adic Ġemotional Ġagreements Ġcould Ġbe Ġinterpreted Ġas Ġthe Ġresulting Ġcomplex Ġemotion , Ġor Ġfurther Ġannot ated Ġto Ġobtain Ġa Ġsingle Ġdominant Ġemotion . ĠThere Ġwas Ġa Ġnumber Ġof Ġterm Ġgroups Ġwith Ġopposite Ġemotion Ġdy ads , Ġpresented Ġin ĠTable ĠT AB REF 33 Ġ, but Ġas Ġthe Ġnumber Ġof Ġannotations Ġincreases , Ġemotional Ġagreement Ġoccurrences Ġ- comb ination Ġor Ġopposition - Ġdecreases . </s> In Ġtotal , Ġthe Ġlex icon Ġfeatures Ġ177 40 Ġannot ated Ġterms Ġwith Ġ3 Ġclasses Ġand Ġ11 Ġsub classes . The Ġdominant Ġclass Ġfor Ġ70 30 Ġterms Ġwas Ġemotion , Ġ191 Ġintens ifying , Ġ68 01 Ġnone , Ġand Ġ37 18 Ġin Ġsome Ġform Ġof Ġsubclass Ġagreement . ĠLex icon Ġterms Ġare Ġmainly Ġjoy Ġannot ated , Ġand Ġemotional Ġagreement Ġis Ġprevalent Ġin Ġ10 % Ġof Ġthe Ġterms . ĠOnly Ġ21 % Ġof Ġtotal Ġterms Ġhave Ġa Ġsubclass Ġagreement . </s> Rel iability </s> Single Ġannotation Ġreliability Ġagreement Ġis Ġthe Ġdegree Ġof Ġagreement Ġbetween Ġannot ators , Ġfor Ġterm Ġgroups Ġthat Ġhave Ġannotation Ġmajority Ġin Ġexactly Ġone Ġsub Ġclass . ĠIn Ġour Ġlex icon , Ġsingle Ġannotation Ġreliability Ġagreement Ġwas Ġlow , Ġmainly Ġdue Ġto Ġthe Ġlow Ġnumber Ġof Ġannot ators Ġfor Ġeach Ġterm Ġgroup Ġin Ġrelation Ġto Ġthe Ġhigh Ġnumber Ġof Ġpossible Ġcategories . </s> Based Ġon ĠFle iss ĠKappa ĠB IB REF 30 Ġ( sim ply Ġreferred Ġas Ġk ), Ġand Ġas Ġseen Ġin ĠTable ĠT AB REF 35 Ġ, Ġterm Ġgroups Ġwith Ġ2 Ġannotations Ġhad Ġthe Ġlowest Ġreliability Ġagreement , Ġwhile Ġterm Ġgroups Ġwith Ġ6 Ġannotations Ġthe Ġhighest Ġreliability Ġagreement . ĠAs Ġthe Ġnumber Ġof Ġannot ators Ġrises , Ġthe Ġnumber Ġof Ġpossible Ġagreement Ġperm utations Ġincreases Ġbut Ġthe Ġnumber Ġof Ġmajor Ġannot ated Ġsub classes Ġdecreases . ĠMore Ġannot ators Ġhave Ġa Ġpositive Ġeffect Ġin Ġboth Ġk Ġand Ġcertainty Ġof Ġclassification . </s> As Ġwe Ġrestrict Ġour Ġlex icon Ġto Ġemotions , Ġreliability Ġincreases Ġfor Ġany Ġnumber Ġof Ġannot ators Ġexcept Ġtwo . ĠThis Ġis Ġexplained Ġby Ġthe Ġdecrease Ġin Ġthe Ġnumber Ġof Ġpossible Ġcategories . ĠWhen Ġwe Ġrestrict Ġour Ġanalysis Ġon Ġemotion Ġrelated Ġannotation Ġthe Ġprobability Ġfor Ġagreement Ġin Ġannotations Ġincreases , Ġresulting Ġin Ġa Ġhigh Ġemotional Ġk . ĠThe Ġbest Ġway Ġto Ġincrease Ġk Ġis Ġto Ġprovide Ġadditional Ġannotations Ġthat Ġwill Ġeventually Ġconverge Ġto Ġa Ġmajority Ġclass Ġor Ġa Ġlimited Ġgroup Ġof Ġclasses . </s> C rowd Ġand Ġexperts Ġcomparison </s> We Ġperform Ġa Ġdirect Ġcomparison Ġof Ġexpert Ġand Ġcrowd Ġcontributors , Ġfor Ġ1000 Ġterm Ġgroups Ġbased Ġon Ġthe Ġnumber Ġof Ġtotal Ġannotations ( 200 Ġterm Ġgroups Ġwith Ġ2 Ġtotal Ġannotations , Ġ200 Ġterm Ġgroups Ġwith Ġ3 Ġtotal Ġannotations , Ġand Ġso Ġon Ġup Ġto Ġterm Ġgroups Ġwith Ġ6 Ġtotal Ġannotations ). ĠThe Ġexperts Ġare Ġtwo ĠPh . D . Ġlingu ists , Ġwhile Ġthe Ġcrowd Ġis Ġmade Ġup Ġof Ġrandom Ġhigh Ġquality Ġcontributors Ġthat Ġchoose Ġto Ġparticipate Ġin Ġthe Ġtask . ĠAs Ġa Ġreference , Ġthe Ġcost Ġof Ġhiring Ġtwo Ġexperts Ġis Ġequal Ġto Ġthe Ġcost Ġof Ġemploying Ġnineteen Ġcontributors Ġin ĠCrowd flower . </s> E val u ators Ġwere Ġgiven Ġa Ġsummary Ġof Ġthe Ġannotations Ġreceived Ġfor Ġthe Ġterm Ġgroup Ġin Ġthe Ġform Ġof : The Ġterm Ġgroup Ġ" ine quality Ġinequ ity " Ġreceived Ġannotations Ġas Ġ50 . 0 % Ġsadness , Ġ33 . 33 % Ġdisgust , Ġ16 . 67 % Ġanger . ĠThen , Ġthey Ġwere Ġasked Ġto Ġevaluate Ġon Ġa Ġscale Ġfrom Ġ1 Ġto Ġ5 , Ġhow Ġvalid Ġthese Ġannotations Ġwere Ġconsidered . </s> The Ġsummary Ġof Ġthe Ġevaluation Ġfor Ġboth Ġexperts Ġand Ġcrowd Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 36 Ġ. ĠThe Ġfirst Ġgraph Ġpresents Ġthe Ġvalidity Ġover Ġthe Ġnumber Ġof Ġannotations Ġin Ġthe Ġmain Ġclass Ġof Ġthe Ġterm Ġgroup . ĠAlthough Ġthis Ġinformation Ġis Ġhidden Ġfrom Ġthe Ġeval u ators , Ġhigh Ġannot ational Ġagreement Ġresults Ġin Ġhigh Ġevaluation Ġscores . ĠBoth Ġexperts Ġand Ġthe Ġcrowd Ġfollow Ġthat Ġpositive Ġtrend . ĠCrowd Ġcontributors Ġare Ġmore Ġstrict Ġin Ġtheir Ġevaluations , Ġbut Ġafter Ġfour Ġannotations Ġwe Ġobserve Ġa Ġsignificant Ġvalidity Ġincrease Ġon Ġboth Ġcrowd Ġand Ġexperts . </s> Likewise , Ġthe Ġannotation Ġpercentage Ġfor Ġthe Ġmajority Ġclass Ġhas Ġa Ġpositive Ġinfluence Ġto Ġthe Ġevaluation Ġscore , Ġwith Ġthe Ġexception Ġof Ġ100 % Ġagreement , Ġsecond Ġgraph ĠFigure ĠFIG REF 36 Ġ. ĠThe Ġweighing Ġfactor Ġfor Ġterm Ġgroups Ġwith Ġ100 % Ġannotation Ġagreement Ġis Ġthe Ġreduced Ġnumber Ġof Ġtotal Ġannotations , Ġas Ġthe Ġmean Ġnumber Ġof Ġtotal Ġannotations Ġdrops Ġabruptly Ġon Ġthe Ġ100 %, Ġand Ġtotal Ġagreement Ġis Ġmore Ġfrequent Ġin Ġterm Ġgroups Ġwith Ġlow Ġnumber Ġof Ġtotal Ġannotations . ĠIt 's Ġworth Ġnoting Ġthat Ġcertain Ġpercentages Ġcan Ġonly Ġoccur Ġon Ġspecific Ġnumber Ġof Ġtotal Ġannotations , Ġe . g . Ġ17 % Ġand Ġ83 % Ġcan Ġonly Ġoccur Ġwhen Ġthe Ġnumber Ġof Ġtotal Ġannotations Ġis Ġsix . </s> In Ġemotion Ġannotations , Ġas Ġseen Ġon Ġthe Ġthird Ġgraph Ġof ĠFigure ĠFIG REF 36 Ġcrowd Ġand Ġexperts Ġfollow Ġa Ġsimilar Ġevaluation Ġpattern . ĠAnt icip ation Ġand Ġjoy Ġhad Ġthe Ġexact Ġsame Ġevaluation , Ġwhile Ġevery Ġother Ġemotion Ġand Ġstop Ġwords Ġwere Ġevaluated Ġlower Ġfrom Ġthe Ġcrowd . ĠThe Ġonly Ġsub classes Ġevaluated Ġhigher Ġfrom Ġthe Ġcrowd Ġwere Ġintens ifiers Ġand Ġneg ators , Ġwith Ġa Ġsignificant Ġdifference Ġin Ġthe Ġevaluations Ġfor Ġthe Ġlatter . ĠSection Ġ6 . 3 Ġprovides Ġa Ġmore Ġdetailed Ġevaluation Ġfor Ġterm Ġgroups Ġthat Ġreceived Ġat Ġleast Ġone Ġannotation Ġas Ġintens ifiers Ġor Ġneg ators . </s> The Ġfinal Ġgraph Ġin ĠFigure ĠFIG REF 36 Ġpresents Ġa Ġclear Ġnegative Ġcorrelation Ġof Ġsubclass Ġagreement Ġand Ġevaluation Ġscores . ĠThe Ġhighest Ġnumber Ġof Ġsub classes Ġthat Ġdo Ġnot Ġaffect Ġevaluation Ġscores Ġis Ġthree , Ġabove Ġthat Ġthere Ġis Ġa Ġsteady Ġdecline Ġof Ġthe Ġevaluation Ġscores , Ġfor Ġboth Ġthe Ġcrowd Ġand Ġthe Ġexperts . </s> The Ġevaluation Ġresults Ġprovide Ġsome Ġkey Ġinsights Ġin Ġthe Ġimportance Ġof Ġthe Ġnumber Ġof Ġannotations . ĠThe Ġevaluation Ġscores Ġstart Ġto Ġimprove Ġafter Ġfour Ġannotations . ĠAn not ational Ġagreement Ġand Ġmajority Ġvoting Ġare Ġless Ġimportant . ĠSub class Ġagreement Ġhas Ġa Ġnegative Ġeffect Ġon Ġthree Ġor Ġmore Ġsub classes . ĠMost Ġimportantly Ġand Ġcompared Ġto Ġexperts , Ġthe Ġcrowd Ġis Ġa Ġstricter Ġeval u ator Ġwith Ġsignificantly Ġlower Ġcosts , Ġand Ġhigher Ġscal ability . ĠSince Ġstrict Ġevaluation Ġleads Ġto Ġhigher Ġquality Ġannotations , Ġthe Ġevaluation Ġcan Ġbe Ġperformed Ġby Ġthe Ġcrowd Ġinstead Ġof Ġexperts . ĠCrowd Ġcontributors Ġcan Ġbe Ġfound Ġin Ġhigh Ġnumbers Ġand Ġmultiple Ġplatforms , Ġcompared Ġto Ġexpert Ġlingu ists . </s> E val uation Ġof Ġintens ifiers Ġand Ġneg ators , Ġwas Ġalso Ġa Ġbatch Ġof Ġevaluation Ġand Ġannotation Ġtasks , Ġas Ġmentioned Ġin ĠSection Ġ6 . 2 . ĠHowever , Ġthe Ġdifference Ġwas Ġthat Ġnow Ġeval u ators Ġhad Ġto Ġanswer Ġif Ġa Ġterm Ġgroup Ġincluded Ġat Ġleast Ġone Ġvalid Ġintens ifier Ġor Ġneg ator . ĠThe Ġevaluation Ġwas Ġagain Ġperformed Ġby Ġexperts Ġand Ġthe Ġcrowd , Ġas Ġdescribed Ġin ĠSection Ġ6 . 2 . 1 . ĠBased Ġon Ġthe Ġannotations Ġreceived Ġin ĠP EL , Ġwe Ġused Ġ5 41 Ġterm Ġgroups Ġthat Ġhad Ġat Ġleast Ġone Ġannotation Ġin Ġany Ġof Ġthe Ġintens ifying Ġsub classes . ĠAlthough , Ġthe Ġparticular Ġselection Ġof Ġterm Ġgroups Ġis Ġstatistically Ġsignificant , Ġwe Ġexpect Ġrelatively Ġlow Ġevaluation Ġscores . ĠThat Ġis Ġbecause Ġthe Ġnumber Ġof Ġintens ifying Ġannotations Ġis Ġlow Ġin Ġmost Ġof Ġthe Ġselected Ġterm Ġgroups . </s> In ĠFigure ĠFIG REF 40 Ġ, Ġwe Ġdefine Ġvarying Ġlevels Ġof Ġagreement Ġon Ġthe Ġvalidity Ġof Ġthe Ġintens ifying Ġclass , Ġbased Ġon Ġthe Ġagreement Ġof Ġevaluations . ĠFor Ġthe Ġexperts Ġgroup , Ġlow Ġagreement Ġrefers Ġto Ġterm Ġgroups Ġthat Ġreceived Ġat Ġleast Ġone Ġout Ġof Ġtwo Ġevaluations Ġas Ġvalid , Ġwhile Ġhigh Ġagreement Ġrequires Ġthe Ġevaluation Ġagreement Ġof Ġboth Ġexperts . ĠSimilarly Ġfor Ġthe Ġcrowd , Ġlow Ġagreement Ġrefers Ġto Ġa Ġminimum Ġof Ġtwo Ġvalid Ġevaluations , Ġmid Ġagreement Ġcorresponds Ġto Ġat Ġleast Ġthree , Ġand Ġhigh Ġagreement Ġrequires Ġan Ġabsolute Ġagreement Ġof Ġall Ġfour Ġeval u ators . </s> Experts Ġare Ġfar Ġmore Ġstrict Ġthan Ġthe Ġcrowd Ġin Ġthe Ġevaluation Ġof Ġintens ifiers Ġand Ġneg ators . ĠWhen Ġthe Ġvalidity Ġagreement Ġis Ġlow Ġon Ġboth Ġevaluation Ġgroups , Ġthe Ġaverage Ġvalid Ġterm Ġgroup Ġdifference Ġis Ġmore Ġthan Ġ40 %, Ġbut Ġthe Ġhigh Ġvalidity Ġagreement Ġthe Ġdifference Ġis Ġjust Ġ5 . 33 %. ĠWhen Ġhigh Ġagreement Ġevaluation Ġis Ġapplied , Ġthe Ġcrowd Ġand Ġexpert Ġevaluations Ġare Ġalmost Ġidentical . ĠThe Ġnumber Ġof Ġcrowd Ġevaluations Ġis Ġthe Ġfactor Ġthat Ġprovides Ġa Ġdegree Ġof Ġfreedom Ġin Ġthe Ġevaluation Ġstrict ness . </s> Lim itations </s> Lex icon Ġacquisition Ġis Ġa Ġcomplex Ġtask Ġthat Ġincludes Ġa Ġmixture Ġof Ġobjective Ġand Ġsubjective Ġtasks . ĠWhile Ġannotation Ġof Ġemotions Ġis Ġmore Ġsubjective , Ġannotation Ġof Ġlinguistic Ġelements Ġ( such Ġas Ġstop Ġwords , Ġemotion Ġshift Ġterms , Ġintens ifiers Ġetc .) Ġis Ġpurely Ġobjective . ĠWe Ġpresented Ġa Ġnovel Ġwork Ġflow Ġthat Ġprovides Ġquality Ġresults Ġfor Ġboth Ġsubjective Ġand Ġobjective Ġtasks . </s> Sub comp onents Ġof Ġthe Ġlex icon Ġacquisition Ġcould Ġbe Ġimproved Ġon Ġan Ġindividual Ġbasis . ĠSpell Ġcheck Ġcan Ġinclude Ġspelling Ġrecommendations , Ġfiltering Ġcould Ġincorporate Ġrewarding Ġand Ġpenalties , Ġevaluation Ġprocess Ġcan Ġinclude Ġexperts Ġand Ġso Ġon . </s> C rowd Ġdiversity Ġin Ġthe Ġannotation Ġand Ġevaluation Ġprocess Ġis Ġanother Ġlimiting Ġfactor . ĠIdeally Ġwe Ġwould Ġprefer Ġa Ġfixed Ġnumber Ġof Ġindividuals , Ġeach Ġto Ġannot ate Ġand Ġevaluate Ġthe Ġwhole Ġcorpus . ĠHowever , Ġthe Ġuniform ity Ġof Ġexpert Ġjudgement Ġis Ġreplaced Ġwith Ġthe Ġdiversity Ġand Ġmass Ġof Ġcontributors . </s> The Ġcorpus Ġmay Ġbe Ġlimiting Ġthe Ġterm Ġgroups Ġin Ġthe Ġlex icon Ġto Ġspecific Ġdomain - specific Ġsubjects . ĠCompar isons Ġwith Ġexisting Ġlex icons , Ġsuch Ġas ĠN RC ĠB IB REF 21 Ġindicate Ġa Ġmoderate Ġoverlap Ġwith Ġ40 % Ġcommon Ġterms . ĠAdditionally , Ġthe Ġnumber Ġof Ġannotations Ġfor Ġa Ġnumber Ġof Ġterm Ġgroups Ġis Ġrelatively Ġlow . ĠHowever , Ġthe Ġbatch Ġtask Ġof Ġevaluation Ġand Ġannotation Ġprovided Ġalmost Ġten Ġthousand Ġannotations , Ġand Ġincreased Ġthe Ġmean Ġnumber Ġof Ġannotations Ġfrom Ġ2 . 3 Ġto Ġ3 . 2 . </s> Conclusion Ġand Ġfuture Ġwork </s> We Ġdemonstrated Ġthat Ġthe Ġcrowd Ġis Ġcapable Ġof Ġproducing Ġand Ġevaluating Ġa Ġquality Ġpure Ġemotion Ġlex icon Ġwithout Ġgold Ġstandards . ĠOur Ġwork - flow Ġis Ġun super vised , Ġsignificantly Ġlower Ġcosts , Ġand Ġimproves Ġscal ability . ĠThere Ġare Ġhowever , Ġvarious Ġparameters Ġthat Ġshould Ġbe Ġtaken Ġinto Ġaccount . ĠSp am Ġis Ġvery Ġcommon Ġand Ġquality Ġassessment Ġpost - annot ations Ġshould Ġbe Ġimplemented . </s> Our Ġapproach Ġrequired Ġworkers Ġto Ġlabel Ġterm Ġgroups Ġas Ġemotion , Ġintens ifiers , Ġand Ġstop Ġwords . ĠAgreement Ġis Ġnot Ġnecessary Ġand Ġmulti Ġemotional Ġterm Ġgroups , Ġwith Ġup Ġto Ġthree Ġemotions , Ġare Ġconsidered Ġequally Ġvalid Ġto Ġsingle Ġemotion Ġterm Ġgroups . ĠThe Ġhardest Ġtask Ġfor Ġthe Ġcrowd Ġproved Ġto Ġbe Ġthe Ġclassification Ġof Ġintens ifiers Ġand Ġneg ators , Ġprobably Ġbecause Ġit Ġrequired Ġa Ġcertain Ġlevel Ġof Ġobject ivity Ġwhich Ġcontradicted Ġthe Ġoverall Ġsubject ivity Ġof Ġthe Ġemotional Ġannotation Ġtask . ĠBased Ġon Ġthe Ġevaluation Ġof Ġterm Ġgroups Ġand Ġthe Ġresults Ġfrom Ġthe Ġassessment , Ġas Ġthe Ġnumber Ġof Ġoverall Ġannot ators Ġrises Ġthe Ġnumber Ġof Ġvalid Ġannotations Ġincreases Ġproportion ally . ĠThis Ġindicates Ġthe Ġimportance Ġof Ġa Ġcritical Ġmass Ġin Ġlex icon Ġacquisition Ġtasks . </s> St em ming Ġreduced Ġtime Ġand Ġcosts Ġrequirements , Ġwith Ġminimal Ġemotional Ġand Ġsubclass Ġagreement . ĠCosts Ġwere Ġreduced Ġby Ġ45 %, Ġand Ġmulti - em otion Ġclassification Ġwas Ġlower Ġthan Ġ10 %. ĠTerm Ġgroups Ġdid Ġnot Ġcreate Ġconfusion Ġamongst Ġworkers , Ġand Ġonly Ġa Ġsmall Ġfraction Ġof Ġterm Ġgroups Ġhad Ġsubclass Ġagreement . ĠOn Ġthe Ġcontrary , Ġincluding Ġthe Ġstem Ġand Ġdescription Ġin Ġthe Ġtask Ġconfused Ġworkers , Ġand Ġwere Ġexcluded Ġfrom Ġthe Ġinterface . ĠWe Ġtested Ġseveral Ġinterface Ġdesigns , Ġand Ġthe Ġone Ġthat Ġworked Ġbest Ġhad Ġminimal Ġinstructions . ĠLex icon Ġacquisition Ġinterfaces Ġin Ġpaid Ġmicro - task Ġenvironments Ġshould Ġbe Ġfurther Ġstudied , Ġwith Ġregards Ġto Ġvarious Ġother Ġcontribution Ġincentives . </s> The Ġcrowd Ġis Ġas Ġcapable Ġof Ġevaluating Ġlex icons , Ġas Ġexperts . ĠL ingu istic Ġelement Ġevaluation Ġcan Ġbe Ġefficiently Ġcrowds ourced , Ġand Ġthe Ġevaluation Ġof Ġemotional Ġor Ġnon Ġemotional Ġelements Ġcan Ġbe Ġas Ġstrict Ġas Ġneeded . ĠThe Ġnumber Ġof Ġeval u ators Ġplays Ġa Ġkey Ġrole Ġin Ġboth Ġemotional Ġand Ġlinguistic Ġevaluations . ĠThe Ġcrowd Ġis Ġstrict Ġon Ġemotional Ġevaluations , Ġwhile Ġthe Ġexperts Ġare Ġstrict Ġin Ġlinguistic Ġevaluations . ĠHowever , Ġa Ġhigh Ġnumber Ġof Ġcrowd Ġevaluations Ġbroad ens Ġthe Ġstrict ness Ġfreedom , Ġwith Ġa Ġsmall Ġfraction Ġof Ġthe Ġexperts ' Ġhiring Ġcosts . ĠDepending Ġon Ġthe Ġnumber Ġof Ġevaluations , Ġvarying Ġlevels Ġof Ġevaluation Ġagreement Ġcan Ġbe Ġimplemented . </s> Our Ġlong Ġterm Ġgoal Ġis Ġto Ġcreate Ġa Ġvoluntary Ġplatform Ġfor Ġpure Ġemotion Ġlex icon Ġacquisition , Ġto Ġfurther Ġstudy Ġthe Ġeffects Ġof Ġcritical Ġmass Ġin Ġlex icon Ġacquisition . ĠIn Ġshort Ġterm , Ġwe Ġwill Ġperform Ġthe Ġexact Ġsame Ġcrowds ourcing Ġtask Ġin Ġa Ġvoluntary Ġplatform , ĠCrowd 4 U Ġor Ġsimilar Ġplatforms , Ġto Ġstudy Ġthe Ġeffect Ġof Ġmonetary Ġand Ġcontribution Ġincentives Ġin Ġpure Ġemotion Ġsentiment Ġannotation . ĠIn Ġparallel , Ġwe Ġwill Ġperform Ġa Ġqualitative Ġanalysis Ġwith Ġregards Ġto Ġunderstanding Ġof Ġintens ifiers Ġand Ġneg ators , Ġto Ġcreate Ġthe Ġoptimal Ġset Ġof Ġinstructions Ġand Ġexamples . ĠFinally , Ġwe Ġare Ġconsidering Ġhow Ġwe Ġcan Ġextend Ġthe Ġapproach Ġto Ġvarious Ġother Ġlinguistic Ġelements , Ġsuch Ġas Ġwords Ġthat Ġsplit Ġthe Ġsentence , Ġwords Ġthat Ġindicate Ġmore Ġimportant Ġparts Ġof Ġa Ġsentence Ġand Ġso Ġon . </s> We Ġbelieve Ġthat Ġbeyond Ġpol arity Ġsentiment Ġanalysis Ġcan Ġenhance Ġand Ġextend Ġsimple Ġpol arity Ġbased Ġapplications . ĠSent iment Ġanalysis Ġin Ġmarketing , Ġpolitics , Ġhealth Ġmonitoring , Ġonline Ġsocial Ġnetworks , Ġand Ġevaluation Ġprocesses Ġwould Ġbenefit Ġfrom Ġa Ġcrowds ourced Ġpure Ġemotion Ġlex icon . </s> Acknowled gement </s> This Ġwork Ġwas Ġsupported Ġby Ġthe ĠEU Ġproject Ġ" Q R OW D Ġ- ĠBecause ĠBig ĠData ĠIntegration Ġis ĠHuman ly ĠPossible ".