BIBREF17, BIBREF18, TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, Stanford NLP NER BIBREF21
EXTRACTIVE
BIBREF23, BIBREF17, BIBREF18, BIBREF19, BIBREF24, BIBREF25, BIBREF26
EXTRACTIVE
Among commercial NLP toolkits (e.g., BIBREF14, BIBREF15, BIBREF16), we selected BIBREF17 and BIBREF18 for our experiments, which, to the best of our knowledge, are the only publicly accessible commercial APIs for the task of entity-level sentiment analysis that is agnostic to the text domain. We also report results of TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, and Stanford NLP NER BIBREF21.
We conducted two sets of experiments. In the first set, we used BIBREF23, BIBREF17, and BIBREF18, for entity-level sentiment analysis; in the second set, BIBREF17, BIBREF19, BIBREF24, BIBREF25, and BIBREF26, BIBREF18 for named-entity recognition.
2002.04181
Introduction </s> As Ġsocial Ġmedia , Ġspecially ĠTwitter , Ġtakes Ġon Ġan Ġinfluential Ġrole Ġin Ġpresidential Ġelections Ġin Ġthe ĠU . S ., Ġnatural Ġlanguage Ġprocessing Ġof Ġpolitical Ġtweets ĠB IB REF 0 Ġhas Ġthe Ġpotential Ġto Ġhelp Ġwith Ġnow casting Ġand Ġforecasting Ġof Ġelection Ġresults Ġas Ġwell Ġas Ġidentifying Ġthe Ġmain Ġissues Ġwith Ġa Ġcandidate ĠâĢĵ Ġtasks Ġof Ġmuch Ġinterest Ġto Ġjournalists , Ġpolitical Ġscientists , Ġand Ġcampaign Ġorganizers ĠB IB REF 1 . ĠAs Ġa Ġmethodology Ġto Ġobtain Ġtraining Ġdata Ġfor Ġa Ġmachine Ġlearning Ġsystem Ġthat Ġanaly zes Ġpolitical Ġtweets , ĠB IB REF 2 Ġdevised Ġa Ġcrowds ourcing Ġscheme Ġwith Ġvariable Ġcrowd worker Ġnumbers Ġbased Ġon Ġthe Ġdifficulty Ġof Ġthe Ġannotation Ġtask . ĠThey Ġprovided Ġa Ġdataset Ġof Ġtweets Ġwhere Ġthe Ġsentiments Ġtowards Ġpolitical Ġcandidates Ġwere Ġlabeled Ġboth Ġby Ġexperts Ġin Ġpolitical Ġcommunication Ġand Ġby Ġcrowd workers Ġwho Ġwere Ġlikely Ġnot Ġdomain Ġexperts . ĠB IB REF 2 Ġrevealed Ġthat Ġcrowd workers Ġcan Ġmatch Ġexpert Ġperformance Ġrelatively Ġaccurately Ġand Ġin Ġa Ġbudget - efficient Ġmanner . ĠGiven Ġthis Ġresult , Ġthe Ġauthors Ġenvisioned Ġfuture Ġwork Ġin Ġwhich Ġground truth Ġlabels Ġwould Ġbe Ġcrowds ourced Ġfor Ġa Ġlarge Ġnumber Ġof Ġtweets Ġand Ġthen Ġused Ġto Ġdesign Ġan Ġautomated ĠN LP Ġtool Ġfor Ġpolitical Ġtweet Ġanalysis . </s> The Ġquestion Ġwe Ġaddress Ġhere Ġis : ĠHow Ġaccurate Ġare Ġexisting ĠN LP Ġtools Ġfor Ġpolitical Ġtweet Ġanalysis ? ĠThese Ġtools Ġwould Ġprovide Ġa Ġbaseline Ġperformance Ġthat Ġany Ġnew Ġmachine Ġlearning Ġsystem Ġfor Ġpolitical Ġtweet Ġanalysis Ġwould Ġcompete Ġagainst . ĠWe Ġhere Ġexplore Ġwhether Ġexisting ĠN LP Ġsystems Ġcan Ġanswer Ġthe Ġquestions Ġ" What Ġsentiment ?" Ġand Ġ" T ow ards Ġwhom ?" Ġaccurately Ġfor Ġthe Ġdataset Ġof Ġpolitical Ġtweets Ġprovided Ġby ĠB IB REF 2 . ĠIn Ġour Ġanalysis , Ġwe Ġinclude ĠN LP Ġtools Ġwith Ġpublicly - available ĠAPIs , Ġeven Ġif Ġthe Ġtools Ġwere Ġnot Ġspecifically Ġdesigned Ġfor Ġshort Ġtexts Ġlike Ġtweets , Ġand , Ġin Ġparticular , Ġpolitical Ġtweets . </s> Our Ġexperiments Ġreveal Ġthat Ġthe Ġtask Ġof Ġentity - level Ġsentiment Ġanalysis Ġis Ġdifficult Ġfor Ġexisting Ġtools Ġto Ġanswer Ġaccurately Ġwhile Ġthe Ġrecognition Ġof Ġthe Ġentity , Ġhere , Ġwhich Ġpolitician , Ġwas Ġeasier . </s> N LP ĠTool k its </s> N LP Ġtool k its Ġtypically Ġhave Ġthe Ġfollowing Ġcapabilities : Ġtoken ization , Ġpart - of - speech Ġ( Po S ) Ġtagging , Ġchunk ing , Ġnamed Ġentity Ġrecognition Ġand Ġsentiment Ġanalysis . ĠIn Ġa Ġstudy Ġby ĠB IB REF 3 , Ġit Ġis Ġshown Ġthat Ġthe Ġwell - known ĠN LP Ġtool k its ĠNL T K ĠB IB REF 4 , ĠStanford ĠCore N LP ĠB IB REF 5 , Ġand ĠTwitter N LP ĠB IB REF 6 Ġhave Ġtoken ization , ĠPo S Ġtagging Ġand ĠN ER Ġmodules Ġin Ġtheir Ġpipelines . ĠThere Ġare Ġtwo Ġmain Ġapproaches Ġfor ĠN ER : Ġ( 1 ) Ġrule - based Ġand Ġ( 2 ) Ġstatistical Ġor Ġmachine Ġlearning Ġbased . ĠThe Ġmost Ġubiquitous Ġalgorithms Ġfor Ġsequence Ġtagging Ġuse ĠHidden ĠMark ov ĠModels ĠB IB REF 7 , ĠMaximum ĠEnt ropy ĠMark ov ĠModels ĠB IB REF 7 , ĠB IB REF 8 , Ġor ĠCond itional ĠRandom ĠFields ĠB IB REF 9 . ĠRecent Ġworks ĠB IB REF 10 , ĠB IB REF 11 Ġhave Ġused Ġrecurrent Ġneural Ġnetworks Ġwith Ġattention Ġmodules Ġfor ĠN ER . </s> Sent iment Ġdetection Ġtools Ġlike ĠSent i Strength ĠB IB REF 12 Ġand ĠTens i Strength ĠB IB REF 13 Ġare Ġrule - based Ġtools , Ġrelying Ġon Ġvarious Ġdiction aries Ġof Ġemot icons , Ġsl angs , Ġidi oms , Ġand Ġironic Ġphrases , Ġand Ġset Ġof Ġrules Ġthat Ġcan Ġdetect Ġthe Ġsentiment Ġof Ġa Ġsentence Ġoverall Ġor Ġa Ġtargeted Ġsentiment . ĠGiven Ġa Ġlist Ġof Ġkeywords , ĠTens i Strength Ġ( similar Ġto ĠSent i Strength ) Ġreports Ġthe Ġsentiment Ġtowards Ġselected Ġentities Ġin Ġa Ġsentence , Ġbased Ġon Ġfive Ġlevels Ġof Ġrelaxation Ġand Ġfive Ġlevels Ġof Ġstress . </s> Among Ġcommercial ĠN LP Ġtool k its Ġ( e . g ., ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 ), Ġwe Ġselected ĠB IB REF 17 Ġand ĠB IB REF 18 Ġfor Ġour Ġexperiments , Ġwhich , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġare Ġthe Ġonly Ġpublicly Ġaccessible Ġcommercial ĠAPIs Ġfor Ġthe Ġtask Ġof Ġentity - level Ġsentiment Ġanalysis Ġthat Ġis Ġag nostic Ġto Ġthe Ġtext Ġdomain . ĠWe Ġalso Ġreport Ġresults Ġof ĠTens i Strength ĠB IB REF 13 , ĠTwitter N LP ĠB IB REF 6 , ĠB IB REF 19 , ĠC og Comp - N LP ĠB IB REF 20 , Ġand ĠStanford ĠN LP ĠN ER ĠB IB REF 21 . </s> Dat as et Ġand ĠAnalysis ĠMethod ology </s> We Ġused Ġthe Ġ1 , 000 - t weet Ġdataset Ġby ĠB IB REF 2 Ġthat Ġcontains Ġthe Ġnamed - ent ities Ġlabels Ġand Ġentity - level Ġsentiments Ġfor Ġeach Ġof Ġthe Ġfour Ġ2016 Ġpresidential Ġprimary Ġcandidates ĠBernie ĠSanders , ĠDonald ĠTrump , ĠHillary ĠClinton , Ġand ĠTed ĠCruz , Ġprovided Ġby Ġcrowd workers , Ġand Ġby Ġexperts Ġin Ġpolitical Ġcommunication , Ġwhose Ġlabels Ġare Ġconsidered Ġground truth . ĠThe Ġcrowd workers Ġwere Ġlocated Ġin Ġthe ĠUS Ġand Ġhired Ġon Ġthe ĠB IB REF 22 Ġplatform . ĠFor Ġthe Ġtask Ġof Ġentity - level Ġsentiment Ġanalysis , Ġa Ġ3 - scale Ġrating Ġof Ġ" negative ," Ġ" neutral ," Ġand Ġ" positive " Ġwas Ġused Ġby Ġthe Ġannot ators . </s> BI B REF 2 Ġproposed Ġa Ġdecision Ġtree Ġapproach Ġfor Ġcomputing Ġthe Ġnumber Ġof Ġcrowd workers Ġwho Ġshould Ġanalyze Ġa Ġtweet Ġbased Ġon Ġthe Ġdifficulty Ġof Ġthe Ġtask . ĠTwe ets Ġare Ġlabeled Ġby Ġ2 , Ġ3 , Ġ5 , Ġor Ġ7 Ġworkers Ġbased Ġon Ġthe Ġdifficulty Ġof Ġthe Ġtask Ġand Ġthe Ġlevel Ġof Ġdisagreement Ġbetween Ġthe Ġcrowd workers . ĠThe Ġmodel Ġcomp utes Ġthe Ġnumber Ġof Ġworkers Ġbased Ġon Ġhow Ġlong Ġa Ġtweet Ġis , Ġthe Ġpresence Ġof Ġa Ġlink Ġin Ġa Ġtweet , Ġand Ġthe Ġnumber Ġof Ġpresent Ġsarc asm Ġsignals . ĠS arc asm Ġis Ġoften Ġused Ġin Ġpolitical Ġtweets Ġand Ġcauses Ġdisagreement Ġbetween Ġthe Ġcrowd workers . ĠThe Ġtweets Ġthat Ġare Ġdeemed Ġto Ġbe Ġsarcastic Ġby Ġthe Ġdecision Ġtree Ġmodel , Ġare Ġexpected Ġto Ġbe Ġmore Ġdifficult Ġto Ġannot ate , Ġand Ġhence Ġare Ġallocated Ġmore Ġcrowd workers Ġto Ġwork Ġon . </s> We Ġconducted Ġtwo Ġsets Ġof Ġexperiments . ĠIn Ġthe Ġfirst Ġset , Ġwe Ġused ĠB IB REF 23 , ĠB IB REF 17 , Ġand ĠB IB REF 18 , Ġfor Ġentity - level Ġsentiment Ġanalysis ; Ġin Ġthe Ġsecond Ġset , ĠB IB REF 17 , ĠB IB REF 19 , ĠB IB REF 24 , ĠB IB REF 25 , Ġand ĠB IB REF 26 , ĠB IB REF 18 Ġfor Ġnamed - entity Ġrecognition . </s> In Ġthe Ġexperiments Ġthat Ġwe Ġconducted Ġwith ĠTwitter N LP Ġfor Ġnamed - entity Ġrecognition , Ġwe Ġworked Ġwith Ġthe Ġdefault Ġvalues Ġof Ġthe Ġmodel . ĠFurthermore , Ġwe Ġselected Ġthe Ġ3 - class ĠStanford ĠN ER Ġmodel , Ġwhich Ġuses Ġthe Ġclasses ĠâĢ ľ person , âĢ Ŀ ĠâĢ ľ organ ization , âĢ Ŀ Ġand ĠâĢ ľ location âĢ Ŀ Ġbecause Ġit Ġresulted Ġin Ġhigher Ġaccuracy Ġcompared Ġto Ġthe Ġ7 - class Ġmodel . ĠFor ĠC og Comp - N LP ĠN ER Ġwe Ġused ĠOnt on otes Ġ5 . 0 ĠN ER Ġmodel ĠB IB REF 27 . ĠFor Ġspa Cy ĠN ER Ġwe Ġused Ġthe Ġ` en _ core _ web _ l g ' Ġmodel . </s> We Ġreport Ġthe Ġexperimental Ġresults Ġfor Ġour Ġtwo Ġtasks Ġin Ġterms Ġof Ġthe Ġcorrect Ġclassification Ġrate Ġ( CC R ). ĠFor Ġsentiment Ġanalysis , Ġwe Ġhave Ġa Ġthree - class Ġproblem Ġ( positive , Ġnegative , Ġand Ġneutral ), Ġwhere Ġthe Ġclasses Ġare Ġmutually Ġexclusive . ĠThe ĠC CR , Ġaveraged Ġfor Ġa Ġset Ġof Ġtweets , Ġis Ġdefined Ġto Ġbe Ġthe Ġnumber Ġof Ġcorrectly - pred icted Ġsentiments Ġover Ġthe Ġnumber Ġof Ġground truth Ġsentiments Ġin Ġthese Ġtweets . ĠFor ĠN ER , Ġwe Ġconsider Ġthat Ġeach Ġtweet Ġmay Ġreference Ġup Ġto Ġfour Ġcandidates , Ġi . e ., Ġtargeted Ġentities . ĠThe ĠC CR , Ġaveraged Ġfor Ġa Ġset Ġof Ġtweets , Ġis Ġthe Ġnumber Ġof Ġcorrectly Ġpredicted Ġentities Ġ( cand idates ) Ġover Ġthe Ġnumber Ġof Ġground truth Ġentities Ġ( cand idates ) Ġin Ġthis Ġset . </s> Results Ġand ĠDiscussion </s> The Ġdataset Ġof Ġ1 , 000 Ġrandomly Ġselected Ġtweets Ġcontains Ġmore Ġthan Ġtwice Ġas Ġmany Ġtweets Ġabout ĠTrump Ġthan Ġabout Ġthe Ġother Ġcandidates . ĠIn Ġthe Ġnamed - entity Ġrecognition Ġexperiment , Ġthe Ġaverage ĠC CR Ġof Ġcrowd workers Ġwas Ġ98 . 6 %, Ġwhile Ġthe ĠC CR Ġof Ġthe Ġautomated Ġsystems Ġranged Ġfrom Ġ77 . 2 % Ġto Ġ96 . 7 %. ĠFor Ġfour Ġof Ġthe Ġautomated Ġsystems , Ġdetecting Ġthe Ġentity ĠTrump Ġwas Ġmore Ġdifficult Ġthan Ġthe Ġother Ġentities Ġ( e . g ., Ġspa Cy Ġ72 . 7 % Ġfor Ġthe Ġentity ĠTrump Ġvs . Ġabove Ġ91 % Ġfor Ġthe Ġother Ġentities ). ĠAn Ġexample Ġof Ġincorrect ĠN ER Ġis Ġshown Ġin ĠFigure ĠFIG REF 1 Ġtop . ĠThe Ġdifficulties Ġthe Ġautomated Ġtools Ġhad Ġin ĠN ER Ġmay Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat Ġthe Ġtools Ġwere Ġnot Ġtrained Ġon Ġtweets , Ġexcept Ġfor ĠTwitter N LP , Ġwhich Ġwas Ġnot Ġin Ġactive Ġdevelopment Ġwhen Ġthe Ġdata Ġwas Ġcreated ĠB IB REF 1 . </s> In Ġthe Ġsentiment Ġanalysis Ġexperiments , Ġwe Ġfound Ġthat Ġa Ġtweet Ġmay Ġcontain Ġmultiple Ġsentiments . ĠThe Ġground truth Ġlabels Ġcontain Ġ210 Ġpositive Ġsentiments , Ġ5 21 Ġneutral Ġsentiments , Ġand Ġ305 Ġnegative Ġsentiments Ġto Ġthe Ġcandidates . ĠWe Ġmeasured Ġthe ĠC CR , Ġacross Ġall Ġtweets , Ġto Ġbe Ġ31 . 7 % Ġfor ĠRos ette ĠText ĠAnalytics , Ġ43 . 2 % Ġfor ĠGoogle ĠCloud , Ġ44 . 2 % Ġfor ĠTens i Strength , Ġand Ġ74 . 7 % Ġfor Ġthe Ġcrowd workers . ĠThis Ġmeans Ġthe Ġdifference Ġbetween Ġthe Ġperformance Ġof Ġthe Ġtools Ġand Ġthe Ġcrowd workers Ġis Ġsignificant ĠâĢĵ Ġmore Ġthan Ġ30 Ġpercent Ġpoints . </s> C rowd workers Ġcorrectly Ġidentified Ġ62 % Ġof Ġthe Ġneutral , Ġ85 % Ġof Ġthe Ġpositive , Ġand Ġ92 % Ġof Ġthe Ġnegative Ġsentiments . ĠGoogle ĠCloud Ġcorrectly Ġidentified Ġ88 % Ġof Ġthe Ġneutral Ġsentiments , Ġbut Ġonly Ġ3 % Ġof Ġthe Ġpositive , Ġand Ġ19 % Ġof Ġthe Ġnegative Ġsentiments . ĠTens i Strength Ġcorrectly Ġidentified Ġ87 . 2 % Ġof Ġthe Ġneutral Ġsentiments , Ġbut Ġ10 . 5 % Ġof Ġthe Ġpositive , Ġand Ġ8 . 1 % Ġof Ġthe Ġnegative Ġsentiments . ĠRos ette ĠText ĠAnalytics Ġcorrectly Ġidentified Ġ22 . 7 % Ġof Ġneutral Ġsentiments , Ġ38 . 1 % Ġof Ġnegative Ġsentiments Ġand Ġ40 . 9 % Ġof Ġpositive Ġsentiments . ĠThe Ġlowest Ġand Ġhighest ĠC CR Ġper tains Ġto Ġtweets Ġabout ĠTrump Ġand ĠSanders Ġfor Ġboth ĠGoogle ĠCloud Ġand ĠTens i Strength , ĠTrump Ġand ĠClinton Ġfor ĠRos ette ĠText ĠAnalytics , Ġand ĠClinton Ġand ĠCruz Ġfor Ġcrowd workers . ĠAn Ġexample Ġof Ġincorrect ĠE LS Ġanalysis Ġis Ġshown Ġin ĠFigure ĠFIG REF 1 Ġbottom . </s> Con clusions Ġand ĠFuture ĠWork </s> Our Ġresults Ġshow Ġthat Ġexisting ĠN LP Ġsystems Ġcannot Ġaccurately Ġperform Ġsentiment Ġanalysis Ġof Ġpolitical Ġtweets Ġin Ġthe Ġdataset Ġwe Ġexperimented Ġwith . ĠLab eling Ġby Ġhumans , Ġeven Ġnon - ex pert Ġcrowd workers , Ġyields Ġaccuracy Ġresults Ġthat Ġare Ġwell Ġabove Ġthe Ġresults Ġof Ġexisting Ġautomated ĠN LP Ġsystems . ĠIn Ġfuture Ġwork Ġwe Ġwill Ġtherefore Ġuse Ġa Ġcrowd worker - label ed Ġdataset Ġto Ġtrain Ġa Ġnew Ġmachine - learning Ġbased ĠN LP Ġsystem Ġfor Ġtweet Ġanalysis . ĠWe Ġwill Ġensure Ġthat Ġthe Ġtraining Ġdata Ġis Ġbalanced Ġamong Ġclasses . ĠOur Ġplan Ġis Ġto Ġuse Ġstate - of - the - art Ġdeep Ġneural Ġnetworks Ġand Ġcompare Ġtheir Ġperformance Ġfor Ġentity - level Ġsentiment Ġanalysis Ġof Ġpolitical Ġtweets . </s> Acknowled gments </s> Part ial Ġsupport Ġof Ġthis Ġwork Ġby Ġthe ĠHar iri ĠInstitute Ġfor ĠComputing Ġand ĠComput ational ĠScience Ġ& ĠEngineering Ġat ĠBoston ĠUniversity Ġ( to ĠL . G .) Ġand Ġa ĠGoogle ĠFaculty ĠResearch ĠAward Ġ( to ĠM . B . Ġand ĠL . G .) Ġis Ġgrate fully Ġacknowledged . ĠAdditionally , Ġwe Ġwould Ġlike Ġto Ġthank ĠDaniel ĠKh ash abi Ġfor Ġhis Ġhelp Ġin Ġrunning Ġthe ĠC og Comp - N LP ĠPython ĠAPI Ġand ĠMike ĠThe l wal Ġfor Ġhis Ġhelp Ġwith ĠTens i Strength . ĠWe Ġare Ġalso Ġgrateful Ġto Ġthe ĠStanford ĠN LP Ġgroup Ġfor Ġclar ifying Ġsome Ġof Ġthe Ġquestions Ġwe Ġhad Ġwith Ġregards Ġto Ġthe ĠStanford ĠN ER Ġtool .
Introduction As social media, specially Twitter, takes on an influential role in presidential elections in the U.S., natural language processing of political tweets BIBREF0 has the potential to help with nowcasting and forecasting of election results as well as identifying the main issues with a candidate – tasks of much interest to journalists, political scientists, and campaign organizers BIBREF1. As a methodology to obtain training data for a machine learning system that analyzes political tweets, BIBREF2 devised a crowdsourcing scheme with variable crowdworker numbers based on the difficulty of the annotation task. They provided a dataset of tweets where the sentiments towards political candidates were labeled both by experts in political communication and by crowdworkers who were likely not domain experts. BIBREF2 revealed that crowdworkers can match expert performance relatively accurately and in a budget-efficient manner. Given this result, the authors envisioned future work in which groundtruth labels would be crowdsourced for a large number of tweets and then used to design an automated NLP tool for political tweet analysis. The question we address here is: How accurate are existing NLP tools for political tweet analysis? These tools would provide a baseline performance that any new machine learning system for political tweet analysis would compete against. We here explore whether existing NLP systems can answer the questions "What sentiment?" and "Towards whom?" accurately for the dataset of political tweets provided by BIBREF2. In our analysis, we include NLP tools with publicly-available APIs, even if the tools were not specifically designed for short texts like tweets, and, in particular, political tweets. Our experiments reveal that the task of entity-level sentiment analysis is difficult for existing tools to answer accurately while the recognition of the entity, here, which politician, was easier. NLP Toolkits NLP toolkits typically have the following capabilities: tokenization, part-of-speech (PoS) tagging, chunking, named entity recognition and sentiment analysis. In a study by BIBREF3, it is shown that the well-known NLP toolkits NLTK BIBREF4, Stanford CoreNLP BIBREF5, and TwitterNLP BIBREF6 have tokenization, PoS tagging and NER modules in their pipelines. There are two main approaches for NER: (1) rule-based and (2) statistical or machine learning based. The most ubiquitous algorithms for sequence tagging use Hidden Markov Models BIBREF7, Maximum Entropy Markov Models BIBREF7, BIBREF8, or Conditional Random Fields BIBREF9. Recent works BIBREF10, BIBREF11 have used recurrent neural networks with attention modules for NER. Sentiment detection tools like SentiStrength BIBREF12 and TensiStrength BIBREF13 are rule-based tools, relying on various dictionaries of emoticons, slangs, idioms, and ironic phrases, and set of rules that can detect the sentiment of a sentence overall or a targeted sentiment. Given a list of keywords, TensiStrength (similar to SentiStrength) reports the sentiment towards selected entities in a sentence, based on five levels of relaxation and five levels of stress. Among commercial NLP toolkits (e.g., BIBREF14, BIBREF15, BIBREF16), we selected BIBREF17 and BIBREF18 for our experiments, which, to the best of our knowledge, are the only publicly accessible commercial APIs for the task of entity-level sentiment analysis that is agnostic to the text domain. We also report results of TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, and Stanford NLP NER BIBREF21. Dataset and Analysis Methodology We used the 1,000-tweet dataset by BIBREF2 that contains the named-entities labels and entity-level sentiments for each of the four 2016 presidential primary candidates Bernie Sanders, Donald Trump, Hillary Clinton, and Ted Cruz, provided by crowdworkers, and by experts in political communication, whose labels are considered groundtruth. The crowdworkers were located in the US and hired on the BIBREF22 platform. For the task of entity-level sentiment analysis, a 3-scale rating of "negative," "neutral," and "positive" was used by the annotators. BIBREF2 proposed a decision tree approach for computing the number of crowdworkers who should analyze a tweet based on the difficulty of the task. Tweets are labeled by 2, 3, 5, or 7 workers based on the difficulty of the task and the level of disagreement between the crowdworkers. The model computes the number of workers based on how long a tweet is, the presence of a link in a tweet, and the number of present sarcasm signals. Sarcasm is often used in political tweets and causes disagreement between the crowdworkers. The tweets that are deemed to be sarcastic by the decision tree model, are expected to be more difficult to annotate, and hence are allocated more crowdworkers to work on. We conducted two sets of experiments. In the first set, we used BIBREF23, BIBREF17, and BIBREF18, for entity-level sentiment analysis; in the second set, BIBREF17, BIBREF19, BIBREF24, BIBREF25, and BIBREF26, BIBREF18 for named-entity recognition. In the experiments that we conducted with TwitterNLP for named-entity recognition, we worked with the default values of the model. Furthermore, we selected the 3-class Stanford NER model, which uses the classes “person,” “organization,” and “location” because it resulted in higher accuracy compared to the 7-class model. For CogComp-NLP NER we used Ontonotes 5.0 NER model BIBREF27. For spaCy NER we used the `en_core_web_lg' model. We report the experimental results for our two tasks in terms of the correct classification rate (CCR). For sentiment analysis, we have a three-class problem (positive, negative, and neutral), where the classes are mutually exclusive. The CCR, averaged for a set of tweets, is defined to be the number of correctly-predicted sentiments over the number of groundtruth sentiments in these tweets. For NER, we consider that each tweet may reference up to four candidates, i.e., targeted entities. The CCR, averaged for a set of tweets, is the number of correctly predicted entities (candidates) over the number of groundtruth entities (candidates) in this set. Results and Discussion The dataset of 1,000 randomly selected tweets contains more than twice as many tweets about Trump than about the other candidates. In the named-entity recognition experiment, the average CCR of crowdworkers was 98.6%, while the CCR of the automated systems ranged from 77.2% to 96.7%. For four of the automated systems, detecting the entity Trump was more difficult than the other entities (e.g., spaCy 72.7% for the entity Trump vs. above 91% for the other entities). An example of incorrect NER is shown in Figure FIGREF1 top. The difficulties the automated tools had in NER may be explained by the fact that the tools were not trained on tweets, except for TwitterNLP, which was not in active development when the data was created BIBREF1. In the sentiment analysis experiments, we found that a tweet may contain multiple sentiments. The groundtruth labels contain 210 positive sentiments, 521 neutral sentiments, and 305 negative sentiments to the candidates. We measured the CCR, across all tweets, to be 31.7% for Rosette Text Analytics, 43.2% for Google Cloud, 44.2% for TensiStrength, and 74.7% for the crowdworkers. This means the difference between the performance of the tools and the crowdworkers is significant – more than 30 percent points. Crowdworkers correctly identified 62% of the neutral, 85% of the positive, and 92% of the negative sentiments. Google Cloud correctly identified 88% of the neutral sentiments, but only 3% of the positive, and 19% of the negative sentiments. TensiStrength correctly identified 87.2% of the neutral sentiments, but 10.5% of the positive, and 8.1% of the negative sentiments. Rosette Text Analytics correctly identified 22.7% of neutral sentiments, 38.1% of negative sentiments and 40.9% of positive sentiments. The lowest and highest CCR pertains to tweets about Trump and Sanders for both Google Cloud and TensiStrength, Trump and Clinton for Rosette Text Analytics, and Clinton and Cruz for crowdworkers. An example of incorrect ELS analysis is shown in Figure FIGREF1 bottom. Conclusions and Future Work Our results show that existing NLP systems cannot accurately perform sentiment analysis of political tweets in the dataset we experimented with. Labeling by humans, even non-expert crowdworkers, yields accuracy results that are well above the results of existing automated NLP systems. In future work we will therefore use a crowdworker-labeled dataset to train a new machine-learning based NLP system for tweet analysis. We will ensure that the training data is balanced among classes. Our plan is to use state-of-the-art deep neural networks and compare their performance for entity-level sentiment analysis of political tweets. Acknowledgments Partial support of this work by the Hariri Institute for Computing and Computational Science & Engineering at Boston University (to L.G.) and a Google Faculty Research Award (to M.B. and L.G.) is gratefully acknowledged. Additionally, we would like to thank Daniel Khashabi for his help in running the CogComp-NLP Python API and Mike Thelwal for his help with TensiStrength. We are also grateful to the Stanford NLP group for clarifying some of the questions we had with regards to the Stanford NER tool.
Which toolkits do they use?
9b7655d39c7a19a23eb8944568eb5618042b9026
Which Ġtool k its Ġdo Ġthey Ġuse ?
BI B REF 17 , ĠB IB REF 18 , ĠTens i Strength ĠB IB REF 13 , ĠTwitter N LP ĠB IB REF 6 , ĠB IB REF 19 , ĠC og Comp - N LP ĠB IB REF 20 , ĠStanford ĠN LP ĠN ER ĠB IB REF 21
<s> Which Ġtool k its Ġdo Ġthey Ġuse ? </s> Introduction </s> As Ġsocial Ġmedia , Ġspecially ĠTwitter , Ġtakes Ġon Ġan Ġinfluential Ġrole Ġin Ġpresidential Ġelections Ġin Ġthe ĠU . S ., Ġnatural Ġlanguage Ġprocessing Ġof Ġpolitical Ġtweets ĠB IB REF 0 Ġhas Ġthe Ġpotential Ġto Ġhelp Ġwith Ġnow casting Ġand Ġforecasting Ġof Ġelection Ġresults Ġas Ġwell Ġas Ġidentifying Ġthe Ġmain Ġissues Ġwith Ġa Ġcandidate ĠâĢĵ Ġtasks Ġof Ġmuch Ġinterest Ġto Ġjournalists , Ġpolitical Ġscientists , Ġand Ġcampaign Ġorganizers ĠB IB REF 1 . ĠAs Ġa Ġmethodology Ġto Ġobtain Ġtraining Ġdata Ġfor Ġa Ġmachine Ġlearning Ġsystem Ġthat Ġanaly zes Ġpolitical Ġtweets , ĠB IB REF 2 Ġdevised Ġa Ġcrowds ourcing Ġscheme Ġwith Ġvariable Ġcrowd worker Ġnumbers Ġbased Ġon Ġthe Ġdifficulty Ġof Ġthe Ġannotation Ġtask . ĠThey Ġprovided Ġa Ġdataset Ġof Ġtweets Ġwhere Ġthe Ġsentiments Ġtowards Ġpolitical Ġcandidates Ġwere Ġlabeled Ġboth Ġby Ġexperts Ġin Ġpolitical Ġcommunication Ġand Ġby Ġcrowd workers Ġwho Ġwere Ġlikely Ġnot Ġdomain Ġexperts . ĠB IB REF 2 Ġrevealed Ġthat Ġcrowd workers Ġcan Ġmatch Ġexpert Ġperformance Ġrelatively Ġaccurately Ġand Ġin Ġa Ġbudget - efficient Ġmanner . ĠGiven Ġthis Ġresult , Ġthe Ġauthors Ġenvisioned Ġfuture Ġwork Ġin Ġwhich Ġground truth Ġlabels Ġwould Ġbe Ġcrowds ourced Ġfor Ġa Ġlarge Ġnumber Ġof Ġtweets Ġand Ġthen Ġused Ġto Ġdesign Ġan Ġautomated ĠN LP Ġtool Ġfor Ġpolitical Ġtweet Ġanalysis . </s> The Ġquestion Ġwe Ġaddress Ġhere Ġis : ĠHow Ġaccurate Ġare Ġexisting ĠN LP Ġtools Ġfor Ġpolitical Ġtweet Ġanalysis ? ĠThese Ġtools Ġwould Ġprovide Ġa Ġbaseline Ġperformance Ġthat Ġany Ġnew Ġmachine Ġlearning Ġsystem Ġfor Ġpolitical Ġtweet Ġanalysis Ġwould Ġcompete Ġagainst . ĠWe Ġhere Ġexplore Ġwhether Ġexisting ĠN LP Ġsystems Ġcan Ġanswer Ġthe Ġquestions Ġ" What Ġsentiment ?" Ġand Ġ" T ow ards Ġwhom ?" Ġaccurately Ġfor Ġthe Ġdataset Ġof Ġpolitical Ġtweets Ġprovided Ġby ĠB IB REF 2 . ĠIn Ġour Ġanalysis , Ġwe Ġinclude ĠN LP Ġtools Ġwith Ġpublicly - available ĠAPIs , Ġeven Ġif Ġthe Ġtools Ġwere Ġnot Ġspecifically Ġdesigned Ġfor Ġshort Ġtexts Ġlike Ġtweets , Ġand , Ġin Ġparticular , Ġpolitical Ġtweets . </s> Our Ġexperiments Ġreveal Ġthat Ġthe Ġtask Ġof Ġentity - level Ġsentiment Ġanalysis Ġis Ġdifficult Ġfor Ġexisting Ġtools Ġto Ġanswer Ġaccurately Ġwhile Ġthe Ġrecognition Ġof Ġthe Ġentity , Ġhere , Ġwhich Ġpolitician , Ġwas Ġeasier . </s> N LP ĠTool k its </s> N LP Ġtool k its Ġtypically Ġhave Ġthe Ġfollowing Ġcapabilities : Ġtoken ization , Ġpart - of - speech Ġ( Po S ) Ġtagging , Ġchunk ing , Ġnamed Ġentity Ġrecognition Ġand Ġsentiment Ġanalysis . ĠIn Ġa Ġstudy Ġby ĠB IB REF 3 , Ġit Ġis Ġshown Ġthat Ġthe Ġwell - known ĠN LP Ġtool k its ĠNL T K ĠB IB REF 4 , ĠStanford ĠCore N LP ĠB IB REF 5 , Ġand ĠTwitter N LP ĠB IB REF 6 Ġhave Ġtoken ization , ĠPo S Ġtagging Ġand ĠN ER Ġmodules Ġin Ġtheir Ġpipelines . ĠThere Ġare Ġtwo Ġmain Ġapproaches Ġfor ĠN ER : Ġ( 1 ) Ġrule - based Ġand Ġ( 2 ) Ġstatistical Ġor Ġmachine Ġlearning Ġbased . ĠThe Ġmost Ġubiquitous Ġalgorithms Ġfor Ġsequence Ġtagging Ġuse ĠHidden ĠMark ov ĠModels ĠB IB REF 7 , ĠMaximum ĠEnt ropy ĠMark ov ĠModels ĠB IB REF 7 , ĠB IB REF 8 , Ġor ĠCond itional ĠRandom ĠFields ĠB IB REF 9 . ĠRecent Ġworks ĠB IB REF 10 , ĠB IB REF 11 Ġhave Ġused Ġrecurrent Ġneural Ġnetworks Ġwith Ġattention Ġmodules Ġfor ĠN ER . </s> Sent iment Ġdetection Ġtools Ġlike ĠSent i Strength ĠB IB REF 12 Ġand ĠTens i Strength ĠB IB REF 13 Ġare Ġrule - based Ġtools , Ġrelying Ġon Ġvarious Ġdiction aries Ġof Ġemot icons , Ġsl angs , Ġidi oms , Ġand Ġironic Ġphrases , Ġand Ġset Ġof Ġrules Ġthat Ġcan Ġdetect Ġthe Ġsentiment Ġof Ġa Ġsentence Ġoverall Ġor Ġa Ġtargeted Ġsentiment . ĠGiven Ġa Ġlist Ġof Ġkeywords , ĠTens i Strength Ġ( similar Ġto ĠSent i Strength ) Ġreports Ġthe Ġsentiment Ġtowards Ġselected Ġentities Ġin Ġa Ġsentence , Ġbased Ġon Ġfive Ġlevels Ġof Ġrelaxation Ġand Ġfive Ġlevels Ġof Ġstress . </s> Among Ġcommercial ĠN LP Ġtool k its Ġ( e . g ., ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 ), Ġwe Ġselected ĠB IB REF 17 Ġand ĠB IB REF 18 Ġfor Ġour Ġexperiments , Ġwhich , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġare Ġthe Ġonly Ġpublicly Ġaccessible Ġcommercial ĠAPIs Ġfor Ġthe Ġtask Ġof Ġentity - level Ġsentiment Ġanalysis Ġthat Ġis Ġag nostic Ġto Ġthe Ġtext Ġdomain . ĠWe Ġalso Ġreport Ġresults Ġof ĠTens i Strength ĠB IB REF 13 , ĠTwitter N LP ĠB IB REF 6 , ĠB IB REF 19 , ĠC og Comp - N LP ĠB IB REF 20 , Ġand ĠStanford ĠN LP ĠN ER ĠB IB REF 21 . </s> Dat as et Ġand ĠAnalysis ĠMethod ology </s> We Ġused Ġthe Ġ1 , 000 - t weet Ġdataset Ġby ĠB IB REF 2 Ġthat Ġcontains Ġthe Ġnamed - ent ities Ġlabels Ġand Ġentity - level Ġsentiments Ġfor Ġeach Ġof Ġthe Ġfour Ġ2016 Ġpresidential Ġprimary Ġcandidates ĠBernie ĠSanders , ĠDonald ĠTrump , ĠHillary ĠClinton , Ġand ĠTed ĠCruz , Ġprovided Ġby Ġcrowd workers , Ġand Ġby Ġexperts Ġin Ġpolitical Ġcommunication , Ġwhose Ġlabels Ġare Ġconsidered Ġground truth . ĠThe Ġcrowd workers Ġwere Ġlocated Ġin Ġthe ĠUS Ġand Ġhired Ġon Ġthe ĠB IB REF 22 Ġplatform . ĠFor Ġthe Ġtask Ġof Ġentity - level Ġsentiment Ġanalysis , Ġa Ġ3 - scale Ġrating Ġof Ġ" negative ," Ġ" neutral ," Ġand Ġ" positive " Ġwas Ġused Ġby Ġthe Ġannot ators . </s> BI B REF 2 Ġproposed Ġa Ġdecision Ġtree Ġapproach Ġfor Ġcomputing Ġthe Ġnumber Ġof Ġcrowd workers Ġwho Ġshould Ġanalyze Ġa Ġtweet Ġbased Ġon Ġthe Ġdifficulty Ġof Ġthe Ġtask . ĠTwe ets Ġare Ġlabeled Ġby Ġ2 , Ġ3 , Ġ5 , Ġor Ġ7 Ġworkers Ġbased Ġon Ġthe Ġdifficulty Ġof Ġthe Ġtask Ġand Ġthe Ġlevel Ġof Ġdisagreement Ġbetween Ġthe Ġcrowd workers . ĠThe Ġmodel Ġcomp utes Ġthe Ġnumber Ġof Ġworkers Ġbased Ġon Ġhow Ġlong Ġa Ġtweet Ġis , Ġthe Ġpresence Ġof Ġa Ġlink Ġin Ġa Ġtweet , Ġand Ġthe Ġnumber Ġof Ġpresent Ġsarc asm Ġsignals . ĠS arc asm Ġis Ġoften Ġused Ġin Ġpolitical Ġtweets Ġand Ġcauses Ġdisagreement Ġbetween Ġthe Ġcrowd workers . ĠThe Ġtweets Ġthat Ġare Ġdeemed Ġto Ġbe Ġsarcastic Ġby Ġthe Ġdecision Ġtree Ġmodel , Ġare Ġexpected Ġto Ġbe Ġmore Ġdifficult Ġto Ġannot ate , Ġand Ġhence Ġare Ġallocated Ġmore Ġcrowd workers Ġto Ġwork Ġon . </s> We Ġconducted Ġtwo Ġsets Ġof Ġexperiments . ĠIn Ġthe Ġfirst Ġset , Ġwe Ġused ĠB IB REF 23 , ĠB IB REF 17 , Ġand ĠB IB REF 18 , Ġfor Ġentity - level Ġsentiment Ġanalysis ; Ġin Ġthe Ġsecond Ġset , ĠB IB REF 17 , ĠB IB REF 19 , ĠB IB REF 24 , ĠB IB REF 25 , Ġand ĠB IB REF 26 , ĠB IB REF 18 Ġfor Ġnamed - entity Ġrecognition . </s> In Ġthe Ġexperiments Ġthat Ġwe Ġconducted Ġwith ĠTwitter N LP Ġfor Ġnamed - entity Ġrecognition , Ġwe Ġworked Ġwith Ġthe Ġdefault Ġvalues Ġof Ġthe Ġmodel . ĠFurthermore , Ġwe Ġselected Ġthe Ġ3 - class ĠStanford ĠN ER Ġmodel , Ġwhich Ġuses Ġthe Ġclasses ĠâĢ ľ person , âĢ Ŀ ĠâĢ ľ organ ization , âĢ Ŀ Ġand ĠâĢ ľ location âĢ Ŀ Ġbecause Ġit Ġresulted Ġin Ġhigher Ġaccuracy Ġcompared Ġto Ġthe Ġ7 - class Ġmodel . ĠFor ĠC og Comp - N LP ĠN ER Ġwe Ġused ĠOnt on otes Ġ5 . 0 ĠN ER Ġmodel ĠB IB REF 27 . ĠFor Ġspa Cy ĠN ER Ġwe Ġused Ġthe Ġ` en _ core _ web _ l g ' Ġmodel . </s> We Ġreport Ġthe Ġexperimental Ġresults Ġfor Ġour Ġtwo Ġtasks Ġin Ġterms Ġof Ġthe Ġcorrect Ġclassification Ġrate Ġ( CC R ). ĠFor Ġsentiment Ġanalysis , Ġwe Ġhave Ġa Ġthree - class Ġproblem Ġ( positive , Ġnegative , Ġand Ġneutral ), Ġwhere Ġthe Ġclasses Ġare Ġmutually Ġexclusive . ĠThe ĠC CR , Ġaveraged Ġfor Ġa Ġset Ġof Ġtweets , Ġis Ġdefined Ġto Ġbe Ġthe Ġnumber Ġof Ġcorrectly - pred icted Ġsentiments Ġover Ġthe Ġnumber Ġof Ġground truth Ġsentiments Ġin Ġthese Ġtweets . ĠFor ĠN ER , Ġwe Ġconsider Ġthat Ġeach Ġtweet Ġmay Ġreference Ġup Ġto Ġfour Ġcandidates , Ġi . e ., Ġtargeted Ġentities . ĠThe ĠC CR , Ġaveraged Ġfor Ġa Ġset Ġof Ġtweets , Ġis Ġthe Ġnumber Ġof Ġcorrectly Ġpredicted Ġentities Ġ( cand idates ) Ġover Ġthe Ġnumber Ġof Ġground truth Ġentities Ġ( cand idates ) Ġin Ġthis Ġset . </s> Results Ġand ĠDiscussion </s> The Ġdataset Ġof Ġ1 , 000 Ġrandomly Ġselected Ġtweets Ġcontains Ġmore Ġthan Ġtwice Ġas Ġmany Ġtweets Ġabout ĠTrump Ġthan Ġabout Ġthe Ġother Ġcandidates . ĠIn Ġthe Ġnamed - entity Ġrecognition Ġexperiment , Ġthe Ġaverage ĠC CR Ġof Ġcrowd workers Ġwas Ġ98 . 6 %, Ġwhile Ġthe ĠC CR Ġof Ġthe Ġautomated Ġsystems Ġranged Ġfrom Ġ77 . 2 % Ġto Ġ96 . 7 %. ĠFor Ġfour Ġof Ġthe Ġautomated Ġsystems , Ġdetecting Ġthe Ġentity ĠTrump Ġwas Ġmore Ġdifficult Ġthan Ġthe Ġother Ġentities Ġ( e . g ., Ġspa Cy Ġ72 . 7 % Ġfor Ġthe Ġentity ĠTrump Ġvs . Ġabove Ġ91 % Ġfor Ġthe Ġother Ġentities ). ĠAn Ġexample Ġof Ġincorrect ĠN ER Ġis Ġshown Ġin ĠFigure ĠFIG REF 1 Ġtop . ĠThe Ġdifficulties Ġthe Ġautomated Ġtools Ġhad Ġin ĠN ER Ġmay Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat Ġthe Ġtools Ġwere Ġnot Ġtrained Ġon Ġtweets , Ġexcept Ġfor ĠTwitter N LP , Ġwhich Ġwas Ġnot Ġin Ġactive Ġdevelopment Ġwhen Ġthe Ġdata Ġwas Ġcreated ĠB IB REF 1 . </s> In Ġthe Ġsentiment Ġanalysis Ġexperiments , Ġwe Ġfound Ġthat Ġa Ġtweet Ġmay Ġcontain Ġmultiple Ġsentiments . ĠThe Ġground truth Ġlabels Ġcontain Ġ210 Ġpositive Ġsentiments , Ġ5 21 Ġneutral Ġsentiments , Ġand Ġ305 Ġnegative Ġsentiments Ġto Ġthe Ġcandidates . ĠWe Ġmeasured Ġthe ĠC CR , Ġacross Ġall Ġtweets , Ġto Ġbe Ġ31 . 7 % Ġfor ĠRos ette ĠText ĠAnalytics , Ġ43 . 2 % Ġfor ĠGoogle ĠCloud , Ġ44 . 2 % Ġfor ĠTens i Strength , Ġand Ġ74 . 7 % Ġfor Ġthe Ġcrowd workers . ĠThis Ġmeans Ġthe Ġdifference Ġbetween Ġthe Ġperformance Ġof Ġthe Ġtools Ġand Ġthe Ġcrowd workers Ġis Ġsignificant ĠâĢĵ Ġmore Ġthan Ġ30 Ġpercent Ġpoints . </s> C rowd workers Ġcorrectly Ġidentified Ġ62 % Ġof Ġthe Ġneutral , Ġ85 % Ġof Ġthe Ġpositive , Ġand Ġ92 % Ġof Ġthe Ġnegative Ġsentiments . ĠGoogle ĠCloud Ġcorrectly Ġidentified Ġ88 % Ġof Ġthe Ġneutral Ġsentiments , Ġbut Ġonly Ġ3 % Ġof Ġthe Ġpositive , Ġand Ġ19 % Ġof Ġthe Ġnegative Ġsentiments . ĠTens i Strength Ġcorrectly Ġidentified Ġ87 . 2 % Ġof Ġthe Ġneutral Ġsentiments , Ġbut Ġ10 . 5 % Ġof Ġthe Ġpositive , Ġand Ġ8 . 1 % Ġof Ġthe Ġnegative Ġsentiments . ĠRos ette ĠText ĠAnalytics Ġcorrectly Ġidentified Ġ22 . 7 % Ġof Ġneutral Ġsentiments , Ġ38 . 1 % Ġof Ġnegative Ġsentiments Ġand Ġ40 . 9 % Ġof Ġpositive Ġsentiments . ĠThe Ġlowest Ġand Ġhighest ĠC CR Ġper tains Ġto Ġtweets Ġabout ĠTrump Ġand ĠSanders Ġfor Ġboth ĠGoogle ĠCloud Ġand ĠTens i Strength , ĠTrump Ġand ĠClinton Ġfor ĠRos ette ĠText ĠAnalytics , Ġand ĠClinton Ġand ĠCruz Ġfor Ġcrowd workers . ĠAn Ġexample Ġof Ġincorrect ĠE LS Ġanalysis Ġis Ġshown Ġin ĠFigure ĠFIG REF 1 Ġbottom . </s> Con clusions Ġand ĠFuture ĠWork </s> Our Ġresults Ġshow Ġthat Ġexisting ĠN LP Ġsystems Ġcannot Ġaccurately Ġperform Ġsentiment Ġanalysis Ġof Ġpolitical Ġtweets Ġin Ġthe Ġdataset Ġwe Ġexperimented Ġwith . ĠLab eling Ġby Ġhumans , Ġeven Ġnon - ex pert Ġcrowd workers , Ġyields Ġaccuracy Ġresults Ġthat Ġare Ġwell Ġabove Ġthe Ġresults Ġof Ġexisting Ġautomated ĠN LP Ġsystems . ĠIn Ġfuture Ġwork Ġwe Ġwill Ġtherefore Ġuse Ġa Ġcrowd worker - label ed Ġdataset Ġto Ġtrain Ġa Ġnew Ġmachine - learning Ġbased ĠN LP Ġsystem Ġfor Ġtweet Ġanalysis . ĠWe Ġwill Ġensure Ġthat Ġthe Ġtraining Ġdata Ġis Ġbalanced Ġamong Ġclasses . ĠOur Ġplan Ġis Ġto Ġuse Ġstate - of - the - art Ġdeep Ġneural Ġnetworks Ġand Ġcompare Ġtheir Ġperformance Ġfor Ġentity - level Ġsentiment Ġanalysis Ġof Ġpolitical Ġtweets . </s> Acknowled gments </s> Part ial Ġsupport Ġof Ġthis Ġwork Ġby Ġthe ĠHar iri ĠInstitute Ġfor ĠComputing Ġand ĠComput ational ĠScience Ġ& ĠEngineering Ġat ĠBoston ĠUniversity Ġ( to ĠL . G .) Ġand Ġa ĠGoogle ĠFaculty ĠResearch ĠAward Ġ( to ĠM . B . Ġand ĠL . G .) Ġis Ġgrate fully Ġacknowledged . ĠAdditionally , Ġwe Ġwould Ġlike Ġto Ġthank ĠDaniel ĠKh ash abi Ġfor Ġhis Ġhelp Ġin Ġrunning Ġthe ĠC og Comp - N LP ĠPython ĠAPI Ġand ĠMike ĠThe l wal Ġfor Ġhis Ġhelp Ġwith ĠTens i Strength . ĠWe Ġare Ġalso Ġgrateful Ġto Ġthe ĠStanford ĠN LP Ġgroup Ġfor Ġclar ifying Ġsome Ġof Ġthe Ġquestions Ġwe Ġhad Ġwith Ġregards Ġto Ġthe ĠStanford ĠN ER Ġtool .