 high-quality datasets  from SemEval-2016 “Sentiment Analysis in Twitter” task
ABSTRACTIVE
 SemEval-2016 “Sentiment Analysis in Twitter”
EXTRACTIVE
Ternary and fine-grained sentiment classification were part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF16 . We use the high-quality datasets the challenge organizers released. The dataset for fine-grained classification is split in training, development, development_test and test parts. In the rest, we refer to these splits as train, development and test, where train is composed by the training and the development instances. Table TABREF7 presents an overview of the data. As discussed in BIBREF16 and illustrated in the Table, the fine-grained dataset is highly unbalanced and skewed towards the positive sentiment: only INLINEFORM0 of the training examples are labeled with one of the negative classes.
Ternary and fine-grained sentiment classification were part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF16 . We use the high-quality datasets the challenge organizers released. The dataset for fine-grained classification is split in training, development, development_test and test parts. In the rest, we refer to these splits as train, development and test, where train is composed by the training and the development instances. Table TABREF7 presents an overview of the data. As discussed in BIBREF16 and illustrated in the Table, the fine-grained dataset is highly unbalanced and skewed towards the positive sentiment: only INLINEFORM0 of the training examples are labeled with one of the negative classes.
1707.03569
Introduction </s> Aut omatic Ġclassification Ġof Ġsentiment Ġhas Ġmainly Ġfocused Ġon Ġcategor izing Ġtweets Ġin Ġeither Ġtwo Ġ( binary Ġsentiment Ġanalysis ) Ġor Ġthree Ġ( tern ary Ġsentiment Ġanalysis ) Ġcategories ĠB IB REF 0 Ġ. ĠIn Ġthis Ġwork Ġwe Ġstudy Ġthe Ġproblem Ġof Ġfine - gr ained Ġsentiment Ġclassification Ġwhere Ġtweets Ġare Ġclassified Ġaccording Ġto Ġa Ġfive - point Ġscale Ġranging Ġfrom ĠVery Neg ative Ġto ĠVery Pos itive . ĠTo Ġillustrate Ġthis , ĠTable ĠT AB REF 3 Ġpresents Ġexamples Ġof Ġtweets Ġassociated Ġwith Ġeach Ġof Ġthese Ġcategories . ĠFive - point Ġscales Ġare Ġwidely Ġadopted Ġin Ġreview Ġsites Ġlike ĠAmazon Ġand ĠTrip Ad visor , Ġwhere Ġa Ġuser 's Ġsentiment Ġis Ġordered Ġwith Ġrespect Ġto Ġits Ġintensity . ĠFrom Ġa Ġsentiment Ġanalysis Ġperspective , Ġthis Ġdefines Ġa Ġclassification Ġproblem Ġwith Ġfive Ġcategories . ĠIn Ġparticular , ĠSebast iani Ġet Ġal . ĠB IB REF 1 Ġdefined Ġsuch Ġclassification Ġproblems Ġwhose Ġcategories Ġare Ġexplicitly Ġordered Ġto Ġbe Ġord inal Ġclassification Ġproblems . ĠTo Ġaccount Ġfor Ġthe Ġordering Ġof Ġthe Ġcategories , Ġlearners Ġare Ġpenal ized Ġaccording Ġto Ġhow Ġfar Ġfrom Ġthe Ġtrue Ġclass Ġtheir Ġpredictions Ġare . </s> Although Ġconsidering Ġdifferent Ġscales , Ġthe Ġvarious Ġsettings Ġof Ġsentiment Ġclassification Ġare Ġrelated . ĠFirst , Ġone Ġmay Ġuse Ġthe Ġsame Ġfeature Ġextraction Ġand Ġengineering Ġapproaches Ġto Ġrepresent Ġthe Ġtext Ġspans Ġsuch Ġas Ġword Ġmembership Ġin Ġlex icons , Ġmorph o - sy nt actic Ġstatistics Ġlike Ġpunct uation Ġor Ġelong ated Ġword Ġcounts ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠSecond , Ġone Ġwould Ġexpect Ġthat Ġknowledge Ġfrom Ġone Ġtask Ġcan Ġbe Ġtransf ered Ġto Ġthe Ġothers Ġand Ġthis Ġwould Ġbenefit Ġthe Ġperformance . ĠKnowing Ġthat Ġa Ġtweet Ġis ĠâĢ ľ Pos itive âĢ Ŀ Ġin Ġthe Ġter n ary Ġsetting Ġnarrow s Ġthe Ġclassification Ġdecision Ġbetween Ġthe ĠVery Pos itive Ġand ĠPositive Ġcategories Ġin Ġthe Ġfine - gr ained Ġsetting . ĠFrom Ġa Ġresearch Ġperspective Ġthis Ġraises Ġthe Ġquestion Ġof Ġwhether Ġand Ġhow Ġone Ġmay Ġbenefit Ġwhen Ġtackling Ġsuch Ġrelated Ġtasks Ġand Ġhow Ġone Ġcan Ġtransfer Ġknowledge Ġfrom Ġone Ġtask Ġto Ġanother Ġduring Ġthe Ġtraining Ġphase . </s> Our Ġfocus Ġin Ġthis Ġwork Ġis Ġto Ġexploit Ġthe Ġrelation Ġbetween Ġthe Ġsentiment Ġclassification Ġsettings Ġand Ġdemonstrate Ġthe Ġbenefits Ġstemming Ġfrom Ġcombining Ġthem . ĠTo Ġthis Ġend , Ġwe Ġpropose Ġto Ġformulate Ġthe Ġdifferent Ġclassification Ġproblems Ġas Ġa Ġmultit ask Ġlearning Ġproblem Ġand Ġjointly Ġlearn Ġthem . ĠMult it ask Ġlearning ĠB IB REF 4 Ġhas Ġshown Ġgreat Ġpotential Ġin Ġvarious Ġdomains Ġand Ġits Ġbenefits Ġhave Ġbeen Ġempir ically Ġvalidated ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġusing Ġdifferent Ġtypes Ġof Ġdata Ġand Ġlearning Ġapproaches . ĠAn Ġimportant Ġbenefit Ġof Ġmultit ask Ġlearning Ġis Ġthat Ġit Ġprovides Ġan Ġelegant Ġway Ġto Ġaccess Ġresources Ġdeveloped Ġfor Ġsimilar Ġtasks . ĠBy Ġjointly Ġlearning Ġcorrelated Ġtasks , Ġthe Ġamount Ġof Ġusable Ġdata Ġincreases . ĠFor Ġinstance , Ġwhile Ġfor Ġter n ary Ġclassification Ġone Ġcan Ġlabel Ġdata Ġusing Ġdistant Ġsupervision Ġwith Ġemot icons ĠB IB REF 9 Ġ, Ġthere Ġis Ġno Ġstraightforward Ġway Ġto Ġdo Ġso Ġfor Ġthe Ġfine - gr ained Ġproblem . ĠHowever , Ġthe Ġlatter Ġcan Ġbenefit Ġindirectly , Ġif Ġthe Ġter n ary Ġand Ġfine - gr ained Ġtasks Ġare Ġlearned Ġjointly . </s> The Ġresearch Ġquestion Ġthat Ġthe Ġpaper Ġattempts Ġto Ġanswer Ġis Ġthe Ġfollowing : ĠCan Ġtwitter Ġsentiment Ġclassification Ġproblems , Ġand Ġfine - gr ained Ġsentiment Ġclassification Ġin Ġparticular , Ġbenefit Ġfrom Ġmultit ask Ġlearning ? ĠTo Ġanswer Ġthe Ġquestion , Ġthe Ġpaper Ġbrings Ġthe Ġfollowing Ġtwo Ġmain Ġcontributions : Ġ( i ) Ġwe Ġshow Ġhow Ġjointly Ġlearning Ġthe Ġter n ary Ġand Ġfine - gr ained Ġsentiment Ġclassification Ġproblems Ġin Ġa Ġmultit ask Ġsetting Ġimproves Ġthe Ġstate - of - the - art Ġperformance , Ġand Ġ( ii ) Ġwe Ġdemonstrate Ġthat Ġrecurrent Ġneural Ġnetworks Ġoutper form Ġmodels Ġpreviously Ġproposed Ġwithout Ġaccess Ġto Ġhuge Ġcorpor a Ġwhile Ġbeing Ġflexible Ġto Ġincorporate Ġdifferent Ġsources Ġof Ġdata . </s> Mult it ask ĠLearning Ġfor ĠTwitter ĠSent iment ĠClassification </s> In Ġhis Ġwork , ĠCar uana ĠB IB REF 4 Ġproposed Ġa Ġmultit ask Ġapproach Ġin Ġwhich Ġa Ġlear ner Ġtakes Ġadvantage Ġof Ġthe Ġmultipl icity Ġof Ġinter dependent Ġtasks Ġwhile Ġjointly Ġlearning Ġthem . ĠThe Ġintuition Ġis Ġthat Ġif Ġthe Ġtasks Ġare Ġcorrelated , Ġthe Ġlear ner Ġcan Ġlearn Ġa Ġmodel Ġjointly Ġfor Ġthem Ġwhile Ġtaking Ġinto Ġaccount Ġthe Ġshared Ġinformation Ġwhich Ġis Ġexpected Ġto Ġimprove Ġits Ġgeneral ization Ġability . ĠPeople Ġexpress Ġtheir Ġopinions Ġonline Ġon Ġvarious Ġsubjects Ġ( events , Ġproducts .. ), Ġon Ġseveral Ġlanguages Ġand Ġin Ġseveral Ġstyles Ġ( t we ets , Ġparagraph - sized Ġreviews .. ), Ġand Ġit Ġis Ġexactly Ġthis Ġvariety Ġthat Ġmotiv ates Ġthe Ġmultit ask Ġapproaches . ĠSpecifically Ġfor ĠTwitter Ġfor Ġinstance , Ġthe Ġdifferent Ġsettings Ġof Ġclassification Ġlike Ġbinary , Ġter n ary Ġand Ġfine - gr ained Ġare Ġcorrelated Ġsince Ġtheir Ġdifference Ġlies Ġin Ġthe Ġsentiment Ġgran ularity Ġof Ġthe Ġclasses Ġwhich Ġincreases Ġwhile Ġmoving Ġfrom Ġbinary Ġto Ġfine - gr ained Ġproblems . </s> There Ġare Ġtwo Ġmain Ġdecisions Ġto Ġbe Ġmade Ġin Ġour Ġapproach : Ġthe Ġlearning Ġalgorithm , Ġwhich Ġlearns Ġa Ġdecision Ġfunction , Ġand Ġthe Ġdata Ġrepresentation . ĠWith Ġrespect Ġto Ġthe Ġformer , Ġneural Ġnetworks Ġare Ġparticularly Ġsuitable Ġas Ġone Ġcan Ġdesign Ġarchitectures Ġwith Ġdifferent Ġproperties Ġand Ġarbitrary Ġcomplexity . ĠAlso , Ġas Ġtraining Ġneural Ġnetwork Ġusually Ġrelies Ġon Ġback - prop ag ation Ġof Ġerrors , Ġone Ġcan Ġhave Ġshared Ġparts Ġof Ġthe Ġnetwork Ġtrained Ġby Ġestimating Ġerrors Ġon Ġthe Ġjoint Ġtasks Ġand Ġothers Ġspecialized Ġfor Ġparticular Ġtasks . ĠCon cerning Ġthe Ġdata Ġrepresentation , Ġit Ġstrongly Ġdepends Ġon Ġthe Ġdata Ġtype Ġavailable . ĠFor Ġthe Ġtask Ġof Ġsentiment Ġclassification Ġof Ġtweets Ġwith Ġneural Ġnetworks , Ġdistributed Ġembed d ings Ġof Ġwords Ġhave Ġshown Ġgreat Ġpotential . ĠEmb edd ings Ġare Ġdefined Ġas Ġlow - dimensional , Ġdense Ġrepresentations Ġof Ġwords Ġthat Ġcan Ġbe Ġobtained Ġin Ġan Ġun super vised Ġfashion Ġby Ġtraining Ġon Ġlarge Ġquantities Ġof Ġtext ĠB IB REF 10 Ġ. </s> Con cerning Ġthe Ġneural Ġnetwork Ġarchitecture , Ġwe Ġfocus Ġon ĠRec urrent ĠNeural ĠNetworks Ġ( R NN s ) Ġthat Ġare Ġcapable Ġof Ġmodeling Ġshort - range Ġand Ġlong - range Ġdependencies Ġlike Ġthose Ġexhibited Ġin Ġsequence Ġdata Ġof Ġarbitrary Ġlength Ġlike Ġtext . ĠWhile Ġin Ġthe Ġtraditional Ġinformation Ġretrieval Ġparadigm Ġsuch Ġdependencies Ġare Ġcaptured Ġusing ĠIN LINE FORM 0 Ġ- gram s Ġand Ġskip - gram s , ĠR NN s Ġlearn Ġto Ġcapture Ġthem Ġautomatically ĠB IB REF 11 Ġ. ĠTo Ġcircumvent Ġthe Ġproblems Ġwith Ġcapturing Ġlong - range Ġdependencies Ġand Ġpreventing Ġgrad ients Ġfrom Ġvanishing , Ġthe Ġlong Ġshort - term Ġmemory Ġnetwork Ġ( L ST M ) Ġwas Ġproposed ĠB IB REF 12 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġuse Ġan Ġextended Ġversion Ġof ĠL ST M Ġcalled Ġbid irection al ĠL ST M Ġ( bi L ST M ). ĠWhile Ġstandard ĠL ST Ms Ġaccess Ġinformation Ġonly Ġfrom Ġthe Ġpast Ġ( pre vious Ġwords ), Ġbi L ST Ms Ġcapture Ġboth Ġpast Ġand Ġfuture Ġinformation Ġeffectively ĠB IB REF 13 Ġ, ĠB IB REF 11 Ġ. ĠThey Ġconsist Ġof Ġtwo ĠL ST M Ġnetworks , Ġfor Ġpropag ating Ġtext Ġforward Ġand Ġbackwards Ġwith Ġthe Ġgoal Ġbeing Ġto Ġcapture Ġthe Ġdependencies Ġbetter . ĠIndeed , Ġprevious Ġwork Ġon Ġmultit ask Ġlearning Ġshowed Ġthe Ġeffectiveness Ġof Ġbi L ST Ms Ġin Ġa Ġvariety Ġof Ġproblems : ĠB IB REF 14 Ġtackled Ġsequence Ġprediction , Ġwhile ĠB IB REF 6 Ġand ĠB IB REF 15 Ġused Ġbi L ST Ms Ġfor ĠNamed ĠEntity ĠRecogn ition Ġand Ġdependency Ġparsing Ġrespectively . </s> Figure ĠFIG REF 2 Ġpresents Ġthe Ġarchitecture Ġwe Ġuse Ġfor Ġmultit ask Ġlearning . ĠIn Ġthe Ġtop - left Ġof Ġthe Ġfigure Ġa Ġbi L ST M Ġnetwork Ġ( en closed Ġby Ġthe Ġdashed Ġline ) Ġis Ġfed Ġwith Ġembed d ings ĠIN LINE FORM 0 Ġthat Ġcorrespond Ġto Ġthe ĠIN LINE FORM 1 Ġwords Ġof Ġa Ġtoken ized Ġtweet . ĠNotice , Ġas Ġdiscussed Ġabove , Ġthe Ġbi L ST M Ġconsists Ġof Ġtwo ĠL ST Ms Ġthat Ġare Ġfed Ġwith Ġthe Ġword Ġsequence Ġforward Ġand Ġbackwards . ĠOn Ġtop Ġof Ġthe Ġbi L ST M Ġnetwork Ġone Ġ( or Ġmore ) Ġhidden Ġlayers ĠIN LINE FORM 2 Ġtransform Ġits Ġoutput . ĠThe Ġoutput Ġof ĠIN LINE FORM 3 Ġis Ġled Ġto Ġthe Ġsoft max Ġlayers Ġfor Ġthe Ġprediction Ġstep . ĠThere Ġare ĠIN LINE FORM 4 Ġsoft max Ġlayers Ġand Ġeach Ġis Ġused Ġfor Ġone Ġof Ġthe ĠIN LINE FORM 5 Ġtasks Ġof Ġthe Ġmultit ask Ġsetting . ĠIn Ġtasks Ġsuch Ġas Ġsentiment Ġclassification , Ġadditional Ġfeatures Ġlike Ġmembership Ġof Ġwords Ġin Ġsentiment Ġlex icons Ġor Ġcounts Ġof Ġelong ated / capital ized Ġwords Ġcan Ġbe Ġused Ġto Ġenrich Ġthe Ġrepresentation Ġof Ġtweets Ġbefore Ġthe Ġclassification Ġstep ĠB IB REF 3 Ġ. ĠThe Ġlower Ġpart Ġof Ġthe Ġnetwork Ġillustrates Ġhow Ġsuch Ġsources Ġof Ġinformation Ġcan Ġbe Ġincorporated Ġto Ġthe Ġprocess . ĠA Ġvector ĠâĢ ľ Additional ĠFeatures âĢ Ŀ Ġfor Ġeach Ġtweet Ġis Ġtransformed Ġfrom Ġthe Ġhidden Ġlayer ( s ) ĠIN LINE FORM 6 Ġand Ġthen Ġis Ġcombined Ġby Ġconc aten ation Ġwith Ġthe Ġtransformed Ġbi L ST M Ġoutput Ġin Ġthe ĠIN LINE FORM 7 Ġlayer . </s> Exper imental Ġsetup </s> Our Ġgoal Ġis Ġto Ġdemonstrate Ġhow Ġmultit ask Ġlearning Ġcan Ġbe Ġsuccessfully Ġapplied Ġon Ġthe Ġtask Ġof Ġsentiment Ġclassification Ġof Ġtweets . ĠThe Ġparticular ities Ġof Ġtweets Ġare Ġto Ġbe Ġshort Ġand Ġinformal Ġtext Ġspans . ĠThe Ġcommon Ġuse Ġof Ġabbrevi ations , Ġcreative Ġlanguage Ġetc ., Ġmakes Ġthe Ġsentiment Ġclassification Ġproblem Ġchallenging . ĠTo Ġvalidate Ġour Ġhypothesis , Ġthat Ġlearning Ġthe Ġtasks Ġjointly Ġcan Ġbenefit Ġthe Ġperformance , Ġwe Ġpropose Ġan Ġexperimental Ġsetting Ġwhere Ġthere Ġare Ġdata Ġfrom Ġtwo Ġdifferent Ġtwitter Ġsentiment Ġclassification Ġproblems : Ġa Ġfine - gr ained Ġand Ġa Ġter n ary . ĠWe Ġconsider Ġthe Ġfine - gr ained Ġtask Ġto Ġbe Ġour Ġprimary Ġtask Ġas Ġit Ġis Ġmore Ġchallenging Ġand Ġobtaining Ġbigger Ġdatasets , Ġe . g . Ġby Ġdistant Ġsupervision , Ġis Ġnot Ġstraightforward Ġand , Ġhence Ġwe Ġreport Ġthe Ġperformance Ġachieved Ġfor Ġthis Ġtask . </s> T ern ary Ġand Ġfine - gr ained Ġsentiment Ġclassification Ġwere Ġpart Ġof Ġthe ĠSem E val - 2016 ĠâĢ ľ Sent iment ĠAnalysis Ġin ĠTwitter âĢ Ŀ Ġtask ĠB IB REF 16 Ġ. ĠWe Ġuse Ġthe Ġhigh - quality Ġdatasets Ġthe Ġchallenge Ġorganizers Ġreleased . ĠThe Ġdataset Ġfor Ġfine - gr ained Ġclassification Ġis Ġsplit Ġin Ġtraining , Ġdevelopment , Ġdevelopment _ test Ġand Ġtest Ġparts . ĠIn Ġthe Ġrest , Ġwe Ġrefer Ġto Ġthese Ġsplits Ġas Ġtrain , Ġdevelopment Ġand Ġtest , Ġwhere Ġtrain Ġis Ġcomposed Ġby Ġthe Ġtraining Ġand Ġthe Ġdevelopment Ġinstances . ĠTable ĠT AB REF 7 Ġpresents Ġan Ġoverview Ġof Ġthe Ġdata . ĠAs Ġdiscussed Ġin ĠB IB REF 16 Ġand Ġillustrated Ġin Ġthe ĠTable , Ġthe Ġfine - gr ained Ġdataset Ġis Ġhighly Ġun balanced Ġand Ġskewed Ġtowards Ġthe Ġpositive Ġsentiment : Ġonly ĠIN LINE FORM 0 Ġof Ġthe Ġtraining Ġexamples Ġare Ġlabeled Ġwith Ġone Ġof Ġthe Ġnegative Ġclasses . </s> Feature Ġrepresentation ĠWe Ġreport Ġresults Ġusing Ġtwo Ġdifferent Ġfeature Ġsets . ĠThe Ġfirst Ġone , Ġdubbed Ġn bow , Ġis Ġa Ġneural Ġbag - of - words Ġthat Ġuses Ġtext Ġembed d ings Ġto Ġgenerate Ġlow - dimensional , Ġdense Ġrepresentations Ġof Ġthe Ġtweets . ĠTo Ġconstruct Ġthe Ġn bow Ġrepresentation , Ġgiven Ġthe Ġword Ġembed d ings Ġdictionary Ġwhere Ġeach Ġword Ġis Ġassociated Ġwith Ġa Ġvector , Ġwe Ġapply Ġthe Ġaverage Ġcompos itional Ġfunction Ġthat Ġaverages Ġthe Ġembed d ings Ġof Ġthe Ġwords Ġthat Ġcompose Ġa Ġtweet . ĠSimple Ġcompos itional Ġfunctions Ġlike Ġaverage Ġwere Ġshown Ġto Ġbe Ġrobust Ġand Ġefficient Ġin Ġprevious Ġwork ĠB IB REF 17 Ġ. ĠInstead Ġof Ġtraining Ġembed d ings Ġfrom Ġscratch , Ġwe Ġuse Ġthe Ġpre - trained Ġon Ġtweets ĠGl o Ve Ġembed d ings Ġof ĠB IB REF 10 Ġ. ĠIn Ġterms Ġof Ġresources Ġrequired , Ġusing Ġonly Ġn bow Ġis Ġefficient Ġas Ġit Ġdoes Ġnot Ġrequire Ġany Ġdomain Ġknowledge . ĠHowever , Ġprevious Ġresearch Ġon Ġsentiment Ġanalysis Ġshowed Ġthat Ġusing Ġextra Ġresources , Ġlike Ġsentiment Ġlex icons , Ġcan Ġbenefit Ġsignificantly Ġthe Ġperformance ĠB IB REF 3 Ġ, ĠB IB REF 2 Ġ. ĠTo Ġvalidate Ġthis Ġand Ġexamine Ġat Ġwhich Ġextent Ġneural Ġnetworks Ġand Ġmultit ask Ġlearning Ġbenefit Ġfrom Ġsuch Ġfeatures Ġwe Ġevaluate Ġthe Ġmodels Ġusing Ġan Ġaugmented Ġversion Ġof Ġn bow , Ġdubbed Ġn bow +. ĠThe Ġfeature Ġspace Ġof Ġthe Ġlatter , Ġis Ġaugmented Ġusing Ġ1 , 368 Ġextra Ġfeatures Ġconsisting Ġmostly Ġof Ġcounts Ġof Ġpunct uation Ġsymbols Ġ(' !? # @ '), Ġemot icons , Ġelong ated Ġwords Ġand Ġword Ġmembership Ġfeatures Ġin Ġseveral Ġsentiment Ġlex icons . ĠDue Ġto Ġspace Ġlimitations , Ġfor Ġa Ġcomplete Ġpresentation Ġof Ġthese Ġfeatures , Ġwe Ġrefer Ġthe Ġinterested Ġreader Ġto ĠB IB REF 2 Ġ, Ġwhose Ġopen Ġimplementation Ġwe Ġused Ġto Ġextract Ġthem . </s> E val uation Ġmeasure ĠTo Ġreproduce Ġthe Ġsetting Ġof Ġthe ĠSem E val Ġchallenges ĠB IB REF 16 Ġ, Ġwe Ġoptimize Ġour Ġsystems Ġusing Ġas Ġprimary Ġmeasure Ġthe Ġmacro - aver aged ĠMean ĠAbsolute ĠError Ġ( ĠIN LINE FORM 0 Ġ) Ġgiven Ġby : ĠIN LINE FORM 1 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġcategories , ĠIN LINE FORM 1 Ġis Ġthe Ġset Ġof Ġinstances Ġwhose Ġtrue Ġclass Ġis ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġis Ġthe Ġtrue Ġlabel Ġof Ġthe Ġinstance ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġthe Ġpredicted Ġlabel . ĠThe Ġmeasure Ġpenal izes Ġdecisions Ġfar Ġfrom Ġthe Ġtrue Ġones Ġand Ġis Ġmacro - aver aged Ġto Ġaccount Ġfor Ġthe Ġfact Ġthat Ġthe Ġdata Ġare Ġun balanced . ĠCom plement ary Ġto ĠIN LINE FORM 6 Ġ, Ġwe Ġreport Ġthe Ġperformance Ġachieved Ġon Ġthe Ġmicro - aver aged ĠIN LINE FORM 7 Ġmeasure , Ġwhich Ġis Ġa Ġcommonly Ġused Ġmeasure Ġfor Ġclassification . </s> The Ġmodels ĠTo Ġevaluate Ġthe Ġmultit ask Ġlearning Ġapproach , Ġwe Ġcompared Ġit Ġwith Ġseveral Ġother Ġmodels . ĠSupport ĠVector ĠMachines Ġ( S V Ms ) Ġare Ġmaximum Ġmargin Ġclassification Ġalgorithms Ġthat Ġhave Ġbeen Ġshown Ġto Ġachieve Ġcompetitive Ġperformance Ġin Ġseveral Ġtext Ġclassification Ġproblems ĠB IB REF 16 Ġ. ĠS VM ĠIN LINE FORM 0 Ġstands Ġfor Ġan ĠS VM Ġwith Ġlinear Ġkernel Ġand Ġan Ġone - vs - rest Ġapproach Ġfor Ġthe Ġmulti - class Ġproblem . ĠAlso , ĠS VM ĠIN LINE FORM 1 Ġis Ġan ĠS VM Ġwith Ġlinear Ġkernel Ġthat Ġemploys Ġthe Ġcram mer - s inger Ġstrategy ĠB IB REF 18 Ġfor Ġthe Ġmulti - class Ġproblem . ĠLog istic Ġregression Ġ( LR ) Ġis Ġanother Ġtype Ġof Ġlinear Ġclassification Ġmethod , Ġwith Ġprob abil istic Ġmotivation . ĠAgain , Ġwe Ġuse Ġtwo Ġtypes Ġof ĠLog istic ĠReg ression Ġdepending Ġon Ġthe Ġmulti - class Ġstrategy : ĠLR ĠIN LINE FORM 2 Ġthat Ġuses Ġan Ġone - vs - rest Ġapproach Ġand Ġmult in omial ĠLog istic ĠReg ression Ġalso Ġknown Ġas Ġthe ĠMax Ent Ġclass ifier Ġthat Ġuses Ġa Ġmult in omial Ġcriterion . </s> Both ĠSV Ms Ġand ĠL Rs Ġas Ġdiscussed Ġabove Ġtreat Ġthe Ġproblem Ġas Ġa Ġmulti - class Ġone , Ġwithout Ġconsidering Ġthe Ġordering Ġof Ġthe Ġclasses . ĠFor Ġthese Ġfour Ġmodels , Ġwe Ġtuned Ġthe Ġhyper - param eter ĠIN LINE FORM 0 Ġthat Ġcontrols Ġthe Ġimportance Ġof Ġthe ĠL ĠIN LINE FORM 1 Ġregular ization Ġpart Ġin Ġthe Ġoptimization Ġproblem Ġwith Ġgrid - search Ġover ĠIN LINE FORM 2 Ġusing Ġ10 - fold Ġcross - valid ation Ġin Ġthe Ġunion Ġof Ġthe Ġtraining Ġand Ġdevelopment Ġdata Ġand Ġthen Ġret rained Ġthe Ġmodels Ġwith Ġthe Ġselected Ġvalues . ĠAlso , Ġto Ġaccount Ġfor Ġthe Ġun balanced Ġclassification Ġproblem Ġwe Ġused Ġclass Ġweights Ġto Ġpenal ize Ġmore Ġthe Ġerrors Ġmade Ġon Ġthe Ġrare Ġclasses . ĠThese Ġweights Ġwere Ġin versely Ġproportional Ġto Ġthe Ġfrequency Ġof Ġeach Ġclass . ĠFor Ġthe Ġfour Ġmodels Ġwe Ġused Ġthe Ġimplementations Ġof ĠSci kit - learn ĠB IB REF 19 Ġ. </s> For Ġmultit ask Ġlearning Ġwe Ġuse Ġthe Ġarchitecture Ġshown Ġin ĠFigure ĠFIG REF 2 Ġ, Ġwhich Ġwe Ġimplemented Ġwith ĠKer as ĠB IB REF 20 Ġ. ĠThe Ġembed d ings Ġare Ġinitialized Ġwith Ġthe Ġ50 - dimensional ĠGl o Ve Ġembed d ings Ġwhile Ġthe Ġoutput Ġof Ġthe Ġbi L ST M Ġnetwork Ġis Ġset Ġto Ġdimension Ġ50 . ĠThe Ġactivation Ġfunction Ġof Ġthe Ġhidden Ġlayers Ġis Ġthe Ġhyper b olic Ġtang ent . ĠThe Ġweights Ġof Ġthe Ġlayers Ġwere Ġinitialized Ġfrom Ġa Ġuniform Ġdistribution , Ġscaled Ġas Ġdescribed Ġin ĠB IB REF 21 Ġ. ĠWe Ġused Ġthe ĠRoot ĠMean ĠSquare ĠProp ag ation Ġoptimization Ġmethod . ĠWe Ġused Ġdrop out Ġfor Ġregular izing Ġthe Ġnetwork . ĠWe Ġtrained Ġthe Ġnetwork Ġusing Ġbatches Ġof Ġ128 Ġexamples Ġas Ġfollows : Ġbefore Ġselecting Ġthe Ġbatch , Ġwe Ġperform Ġa ĠBern ou lli Ġtrial Ġwith Ġprobability ĠIN LINE FORM 0 Ġto Ġselect Ġthe Ġtask Ġto Ġtrain Ġfor . ĠWith Ġprobability ĠIN LINE FORM 1 Ġwe Ġpick Ġa Ġbatch Ġfor Ġthe Ġfine - gr ained Ġsentiment Ġclassification Ġproblem , Ġwhile Ġwith Ġprobability ĠIN LINE FORM 2 Ġwe Ġpick Ġa Ġbatch Ġfor Ġthe Ġter n ary Ġproblem . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 2 Ġ, Ġthe Ġerror Ġis Ġback prop ag ated Ġuntil Ġthe Ġembed d ings , Ġthat Ġwe Ġfine - t une Ġduring Ġthe Ġlearning Ġprocess . ĠNotice Ġalso Ġthat Ġthe Ġweights Ġof Ġthe Ġnetwork Ġuntil Ġthe Ġlayer ĠIN LINE FORM 3 Ġare Ġshared Ġand Ġtherefore Ġaffected Ġby Ġboth Ġtasks . </s> To Ġtune Ġthe Ġneural Ġnetwork Ġhyper - param eters Ġwe Ġused Ġ5 - fold Ġcross Ġvalidation . ĠWe Ġtuned Ġthe Ġprobability ĠIN LINE FORM 0 Ġof Ġdrop out Ġafter Ġthe Ġhidden Ġlayers ĠIN LINE FORM 1 Ġand Ġfor Ġthe Ġbi L ST M Ġfor ĠIN LINE FORM 2 Ġ, Ġthe Ġsize Ġof Ġthe Ġhidden Ġlayer ĠIN LINE FORM 3 Ġand Ġthe Ġprobability ĠIN LINE FORM 4 Ġof Ġthe ĠBern ou lli Ġtrials Ġfrom ĠIN LINE FORM 5 Ġ. ĠDuring Ġtraining , Ġwe Ġmonitor Ġthe Ġnetwork 's Ġperformance Ġon Ġthe Ġdevelopment Ġset Ġand Ġapply Ġearly Ġstopping Ġif Ġthe Ġperformance Ġon Ġthe Ġvalidation Ġset Ġdoes Ġnot Ġimprove Ġfor Ġ5 Ġconsecutive Ġepoch s . </s> Exper imental Ġresults ĠTable ĠT AB REF 9 Ġillustrates Ġthe Ġperformance Ġof Ġthe Ġmodels Ġfor Ġthe Ġdifferent Ġdata Ġrepresentations . ĠThe Ġupper Ġpart Ġof Ġthe ĠTable Ġsummarizes Ġthe Ġperformance Ġof Ġthe Ġbas elines . ĠThe Ġentry ĠâĢ ľ Bal ik as Ġet Ġal . âĢ Ŀ Ġstands Ġfor Ġthe Ġwinning Ġsystem Ġof Ġthe Ġ2016 Ġedition Ġof Ġthe Ġchallenge ĠB IB REF 2 Ġ, Ġwhich Ġto Ġthe Ġbest Ġof Ġour Ġknowledge Ġholds Ġthe Ġstate - of - the - art . ĠDue Ġto Ġthe Ġsto ch astic ity Ġof Ġtraining Ġthe Ġbi L ST M Ġmodels , Ġwe Ġrepeat Ġthe Ġexperiment Ġ10 Ġtimes Ġand Ġreport Ġthe Ġaverage Ġand Ġthe Ġstandard Ġdeviation Ġof Ġthe Ġperformance Ġachieved . </s> Several Ġobservations Ġcan Ġbe Ġmade Ġfrom Ġthe Ġtable . ĠFirst Ġnotice Ġthat , Ġoverall , Ġthe Ġbest Ġperformance Ġis Ġachieved Ġby Ġthe Ġneural Ġnetwork Ġarchitecture Ġthat Ġuses Ġmultit ask Ġlearning . ĠThis Ġentails Ġthat Ġthe Ġsystem Ġmakes Ġuse Ġof Ġthe Ġavailable Ġresources Ġefficiently Ġand Ġimproves Ġthe Ġstate - of - the - art Ġperformance . ĠIn Ġconjunction Ġwith Ġthe Ġfact Ġthat Ġwe Ġfound Ġthe Ġoptimal Ġprobability ĠIN LINE FORM 0 Ġ, Ġthis Ġhighlights Ġthe Ġbenefits Ġof Ġmultit ask Ġlearning Ġover Ġsingle Ġtask Ġlearning . ĠFurthermore , Ġas Ġdescribed Ġabove , Ġthe Ġneural Ġnetwork - based Ġmodels Ġhave Ġonly Ġaccess Ġto Ġthe Ġtraining Ġdata Ġas Ġthe Ġdevelopment Ġare Ġhold Ġfor Ġearly Ġstopping . ĠOn Ġthe Ġother Ġhand , Ġthe Ġbaseline Ġsystems Ġwere Ġret rained Ġon Ġthe Ġunion Ġof Ġthe Ġtrain Ġand Ġdevelopment Ġsets . ĠHence , Ġeven Ġwith Ġfewer Ġresources Ġavailable Ġfor Ġtraining Ġon Ġthe Ġfine - gr ained Ġproblem , Ġthe Ġneural Ġnetworks Ġoutper form Ġthe Ġbas elines . ĠWe Ġalso Ġhighlight Ġthe Ġpositive Ġeffect Ġof Ġthe Ġadditional Ġfeatures Ġthat Ġprevious Ġresearch Ġproposed . ĠAdding Ġthe Ġfeatures Ġboth Ġin Ġthe Ġbas elines Ġand Ġin Ġthe Ġbi L ST M - based Ġarchitectures Ġimproves Ġthe ĠIN LINE FORM 1 Ġscores Ġby Ġseveral Ġpoints . </s> Lastly , Ġwe Ġcompare Ġthe Ġperformance Ġof Ġthe Ġbaseline Ġsystems Ġwith Ġthe Ġperformance Ġof Ġthe Ġstate - of - the - art Ġsystem Ġof ĠB IB REF 2 Ġ. ĠWhile ĠB IB REF 2 Ġuses Ġn - gram s Ġ( and Ġcharacter - gram s ) Ġwith ĠIN LINE FORM 0 Ġ, Ġthe Ġbaseline Ġsystems Ġ( S V Ms , ĠL Rs ) Ġused Ġin Ġthis Ġwork Ġuse Ġthe Ġn bow + Ġrepresentation , Ġthat Ġrelies Ġon Ġun ig rams . ĠAlthough Ġthey Ġperform Ġon Ġpar , Ġthe Ġcompetitive Ġperformance Ġof Ġn bow Ġhighlights Ġthe Ġpotential Ġof Ġdistributed Ġrepresentations Ġfor Ġshort - text Ġclassification . ĠFurther , Ġincorporating Ġstructure Ġand Ġdistributed Ġrepresentations Ġleads Ġto Ġthe Ġgains Ġof Ġthe Ġbi L ST M Ġnetwork , Ġin Ġthe Ġmultit ask Ġand Ġsingle Ġtask Ġsetting . </s> Similar Ġobservations Ġcan Ġbe Ġdrawn Ġfrom ĠFigure ĠFIG REF 10 Ġthat Ġpresents Ġthe ĠIN LINE FORM 0 Ġscores . ĠAgain , Ġthe Ġbi L ST M Ġnetwork Ġwith Ġmultit ask Ġlearning Ġachieves Ġthe Ġbest Ġperformance . ĠIt Ġis Ġalso Ġto Ġbe Ġnoted Ġthat Ġalthough Ġthe Ġtwo Ġevaluation Ġmeasures Ġare Ġcorrelated Ġin Ġthe Ġsense Ġthat Ġthe Ġranking Ġof Ġthe Ġmodels Ġis Ġthe Ġsame , Ġsmall Ġdifferences Ġin Ġthe ĠIN LINE FORM 1 Ġhave Ġlarge Ġeffect Ġon Ġthe Ġscores Ġof Ġthe ĠIN LINE FORM 2 Ġmeasure . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġshowed Ġthat Ġby Ġjointly Ġlearning Ġthe Ġtasks Ġof Ġter n ary Ġand Ġfine - gr ained Ġclassification Ġwith Ġa Ġmultit ask Ġlearning Ġmodel , Ġone Ġcan Ġgreatly Ġimprove Ġthe Ġperformance Ġon Ġthe Ġsecond . ĠThis Ġopens Ġseveral Ġavenues Ġfor Ġfuture Ġresearch . ĠSince Ġsentiment Ġis Ġexpressed Ġin Ġdifferent Ġtextual Ġtypes Ġlike Ġtweets Ġand Ġparagraph - sized Ġreviews , Ġin Ġdifferent Ġlanguages Ġ( English , ĠGerman , Ġ.. ) Ġand Ġin Ġdifferent Ġgran ularity Ġlevels Ġ( binary , Ġter n ary , .. ) Ġone Ġcan Ġimagine Ġmultit ask Ġapproaches Ġthat Ġcould Ġbenefit Ġfrom Ġcombining Ġsuch Ġresources . ĠAlso , Ġwhile Ġwe Ġopted Ġfor Ġbi L ST M Ġnetworks Ġhere , Ġone Ġcould Ġuse Ġconv olution al Ġneural Ġnetworks Ġor Ġeven Ġtry Ġto Ġcombine Ġdifferent Ġtypes Ġof Ġnetworks Ġand Ġtasks Ġto Ġinvestigate Ġthe Ġperformance Ġeffect Ġof Ġmultit ask Ġlearning . ĠLastly , Ġwhile Ġour Ġapproach Ġmainly Ġrelied Ġon Ġthe Ġfoundations Ġof ĠB IB REF 4 Ġ, Ġthe Ġinternal Ġmechanisms Ġand Ġthe Ġtheoretical Ġguarantees Ġof Ġmultit ask Ġlearning Ġremain Ġto Ġbe Ġbetter Ġunderstood . </s> Acknowled gements </s> This Ġwork Ġis Ġpartially Ġsupported Ġby Ġthe ĠC IF RE ĠN Ġ28 / 2015 .
Introduction Automatic classification of sentiment has mainly focused on categorizing tweets in either two (binary sentiment analysis) or three (ternary sentiment analysis) categories BIBREF0 . In this work we study the problem of fine-grained sentiment classification where tweets are classified according to a five-point scale ranging from VeryNegative to VeryPositive. To illustrate this, Table TABREF3 presents examples of tweets associated with each of these categories. Five-point scales are widely adopted in review sites like Amazon and TripAdvisor, where a user's sentiment is ordered with respect to its intensity. From a sentiment analysis perspective, this defines a classification problem with five categories. In particular, Sebastiani et al. BIBREF1 defined such classification problems whose categories are explicitly ordered to be ordinal classification problems. To account for the ordering of the categories, learners are penalized according to how far from the true class their predictions are. Although considering different scales, the various settings of sentiment classification are related. First, one may use the same feature extraction and engineering approaches to represent the text spans such as word membership in lexicons, morpho-syntactic statistics like punctuation or elongated word counts BIBREF2 , BIBREF3 . Second, one would expect that knowledge from one task can be transfered to the others and this would benefit the performance. Knowing that a tweet is “Positive” in the ternary setting narrows the classification decision between the VeryPositive and Positive categories in the fine-grained setting. From a research perspective this raises the question of whether and how one may benefit when tackling such related tasks and how one can transfer knowledge from one task to another during the training phase. Our focus in this work is to exploit the relation between the sentiment classification settings and demonstrate the benefits stemming from combining them. To this end, we propose to formulate the different classification problems as a multitask learning problem and jointly learn them. Multitask learning BIBREF4 has shown great potential in various domains and its benefits have been empirically validated BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 using different types of data and learning approaches. An important benefit of multitask learning is that it provides an elegant way to access resources developed for similar tasks. By jointly learning correlated tasks, the amount of usable data increases. For instance, while for ternary classification one can label data using distant supervision with emoticons BIBREF9 , there is no straightforward way to do so for the fine-grained problem. However, the latter can benefit indirectly, if the ternary and fine-grained tasks are learned jointly. The research question that the paper attempts to answer is the following: Can twitter sentiment classification problems, and fine-grained sentiment classification in particular, benefit from multitask learning? To answer the question, the paper brings the following two main contributions: (i) we show how jointly learning the ternary and fine-grained sentiment classification problems in a multitask setting improves the state-of-the-art performance, and (ii) we demonstrate that recurrent neural networks outperform models previously proposed without access to huge corpora while being flexible to incorporate different sources of data. Multitask Learning for Twitter Sentiment Classification In his work, Caruana BIBREF4 proposed a multitask approach in which a learner takes advantage of the multiplicity of interdependent tasks while jointly learning them. The intuition is that if the tasks are correlated, the learner can learn a model jointly for them while taking into account the shared information which is expected to improve its generalization ability. People express their opinions online on various subjects (events, products..), on several languages and in several styles (tweets, paragraph-sized reviews..), and it is exactly this variety that motivates the multitask approaches. Specifically for Twitter for instance, the different settings of classification like binary, ternary and fine-grained are correlated since their difference lies in the sentiment granularity of the classes which increases while moving from binary to fine-grained problems. There are two main decisions to be made in our approach: the learning algorithm, which learns a decision function, and the data representation. With respect to the former, neural networks are particularly suitable as one can design architectures with different properties and arbitrary complexity. Also, as training neural network usually relies on back-propagation of errors, one can have shared parts of the network trained by estimating errors on the joint tasks and others specialized for particular tasks. Concerning the data representation, it strongly depends on the data type available. For the task of sentiment classification of tweets with neural networks, distributed embeddings of words have shown great potential. Embeddings are defined as low-dimensional, dense representations of words that can be obtained in an unsupervised fashion by training on large quantities of text BIBREF10 . Concerning the neural network architecture, we focus on Recurrent Neural Networks (RNNs) that are capable of modeling short-range and long-range dependencies like those exhibited in sequence data of arbitrary length like text. While in the traditional information retrieval paradigm such dependencies are captured using INLINEFORM0 -grams and skip-grams, RNNs learn to capture them automatically BIBREF11 . To circumvent the problems with capturing long-range dependencies and preventing gradients from vanishing, the long short-term memory network (LSTM) was proposed BIBREF12 . In this work, we use an extended version of LSTM called bidirectional LSTM (biLSTM). While standard LSTMs access information only from the past (previous words), biLSTMs capture both past and future information effectively BIBREF13 , BIBREF11 . They consist of two LSTM networks, for propagating text forward and backwards with the goal being to capture the dependencies better. Indeed, previous work on multitask learning showed the effectiveness of biLSTMs in a variety of problems: BIBREF14 tackled sequence prediction, while BIBREF6 and BIBREF15 used biLSTMs for Named Entity Recognition and dependency parsing respectively. Figure FIGREF2 presents the architecture we use for multitask learning. In the top-left of the figure a biLSTM network (enclosed by the dashed line) is fed with embeddings INLINEFORM0 that correspond to the INLINEFORM1 words of a tokenized tweet. Notice, as discussed above, the biLSTM consists of two LSTMs that are fed with the word sequence forward and backwards. On top of the biLSTM network one (or more) hidden layers INLINEFORM2 transform its output. The output of INLINEFORM3 is led to the softmax layers for the prediction step. There are INLINEFORM4 softmax layers and each is used for one of the INLINEFORM5 tasks of the multitask setting. In tasks such as sentiment classification, additional features like membership of words in sentiment lexicons or counts of elongated/capitalized words can be used to enrich the representation of tweets before the classification step BIBREF3 . The lower part of the network illustrates how such sources of information can be incorporated to the process. A vector “Additional Features” for each tweet is transformed from the hidden layer(s) INLINEFORM6 and then is combined by concatenation with the transformed biLSTM output in the INLINEFORM7 layer. Experimental setup Our goal is to demonstrate how multitask learning can be successfully applied on the task of sentiment classification of tweets. The particularities of tweets are to be short and informal text spans. The common use of abbreviations, creative language etc., makes the sentiment classification problem challenging. To validate our hypothesis, that learning the tasks jointly can benefit the performance, we propose an experimental setting where there are data from two different twitter sentiment classification problems: a fine-grained and a ternary. We consider the fine-grained task to be our primary task as it is more challenging and obtaining bigger datasets, e.g. by distant supervision, is not straightforward and, hence we report the performance achieved for this task. Ternary and fine-grained sentiment classification were part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF16 . We use the high-quality datasets the challenge organizers released. The dataset for fine-grained classification is split in training, development, development_test and test parts. In the rest, we refer to these splits as train, development and test, where train is composed by the training and the development instances. Table TABREF7 presents an overview of the data. As discussed in BIBREF16 and illustrated in the Table, the fine-grained dataset is highly unbalanced and skewed towards the positive sentiment: only INLINEFORM0 of the training examples are labeled with one of the negative classes. Feature representation We report results using two different feature sets. The first one, dubbed nbow, is a neural bag-of-words that uses text embeddings to generate low-dimensional, dense representations of the tweets. To construct the nbow representation, given the word embeddings dictionary where each word is associated with a vector, we apply the average compositional function that averages the embeddings of the words that compose a tweet. Simple compositional functions like average were shown to be robust and efficient in previous work BIBREF17 . Instead of training embeddings from scratch, we use the pre-trained on tweets GloVe embeddings of BIBREF10 . In terms of resources required, using only nbow is efficient as it does not require any domain knowledge. However, previous research on sentiment analysis showed that using extra resources, like sentiment lexicons, can benefit significantly the performance BIBREF3 , BIBREF2 . To validate this and examine at which extent neural networks and multitask learning benefit from such features we evaluate the models using an augmented version of nbow, dubbed nbow+. The feature space of the latter, is augmented using 1,368 extra features consisting mostly of counts of punctuation symbols ('!?#@'), emoticons, elongated words and word membership features in several sentiment lexicons. Due to space limitations, for a complete presentation of these features, we refer the interested reader to BIBREF2 , whose open implementation we used to extract them. Evaluation measure To reproduce the setting of the SemEval challenges BIBREF16 , we optimize our systems using as primary measure the macro-averaged Mean Absolute Error ( INLINEFORM0 ) given by: INLINEFORM1 where INLINEFORM0 is the number of categories, INLINEFORM1 is the set of instances whose true class is INLINEFORM2 , INLINEFORM3 is the true label of the instance INLINEFORM4 and INLINEFORM5 the predicted label. The measure penalizes decisions far from the true ones and is macro-averaged to account for the fact that the data are unbalanced. Complementary to INLINEFORM6 , we report the performance achieved on the micro-averaged INLINEFORM7 measure, which is a commonly used measure for classification. The models To evaluate the multitask learning approach, we compared it with several other models. Support Vector Machines (SVMs) are maximum margin classification algorithms that have been shown to achieve competitive performance in several text classification problems BIBREF16 . SVM INLINEFORM0 stands for an SVM with linear kernel and an one-vs-rest approach for the multi-class problem. Also, SVM INLINEFORM1 is an SVM with linear kernel that employs the crammer-singer strategy BIBREF18 for the multi-class problem. Logistic regression (LR) is another type of linear classification method, with probabilistic motivation. Again, we use two types of Logistic Regression depending on the multi-class strategy: LR INLINEFORM2 that uses an one-vs-rest approach and multinomial Logistic Regression also known as the MaxEnt classifier that uses a multinomial criterion. Both SVMs and LRs as discussed above treat the problem as a multi-class one, without considering the ordering of the classes. For these four models, we tuned the hyper-parameter INLINEFORM0 that controls the importance of the L INLINEFORM1 regularization part in the optimization problem with grid-search over INLINEFORM2 using 10-fold cross-validation in the union of the training and development data and then retrained the models with the selected values. Also, to account for the unbalanced classification problem we used class weights to penalize more the errors made on the rare classes. These weights were inversely proportional to the frequency of each class. For the four models we used the implementations of Scikit-learn BIBREF19 . For multitask learning we use the architecture shown in Figure FIGREF2 , which we implemented with Keras BIBREF20 . The embeddings are initialized with the 50-dimensional GloVe embeddings while the output of the biLSTM network is set to dimension 50. The activation function of the hidden layers is the hyperbolic tangent. The weights of the layers were initialized from a uniform distribution, scaled as described in BIBREF21 . We used the Root Mean Square Propagation optimization method. We used dropout for regularizing the network. We trained the network using batches of 128 examples as follows: before selecting the batch, we perform a Bernoulli trial with probability INLINEFORM0 to select the task to train for. With probability INLINEFORM1 we pick a batch for the fine-grained sentiment classification problem, while with probability INLINEFORM2 we pick a batch for the ternary problem. As shown in Figure FIGREF2 , the error is backpropagated until the embeddings, that we fine-tune during the learning process. Notice also that the weights of the network until the layer INLINEFORM3 are shared and therefore affected by both tasks. To tune the neural network hyper-parameters we used 5-fold cross validation. We tuned the probability INLINEFORM0 of dropout after the hidden layers INLINEFORM1 and for the biLSTM for INLINEFORM2 , the size of the hidden layer INLINEFORM3 and the probability INLINEFORM4 of the Bernoulli trials from INLINEFORM5 . During training, we monitor the network's performance on the development set and apply early stopping if the performance on the validation set does not improve for 5 consecutive epochs. Experimental results Table TABREF9 illustrates the performance of the models for the different data representations. The upper part of the Table summarizes the performance of the baselines. The entry “Balikas et al.” stands for the winning system of the 2016 edition of the challenge BIBREF2 , which to the best of our knowledge holds the state-of-the-art. Due to the stochasticity of training the biLSTM models, we repeat the experiment 10 times and report the average and the standard deviation of the performance achieved. Several observations can be made from the table. First notice that, overall, the best performance is achieved by the neural network architecture that uses multitask learning. This entails that the system makes use of the available resources efficiently and improves the state-of-the-art performance. In conjunction with the fact that we found the optimal probability INLINEFORM0 , this highlights the benefits of multitask learning over single task learning. Furthermore, as described above, the neural network-based models have only access to the training data as the development are hold for early stopping. On the other hand, the baseline systems were retrained on the union of the train and development sets. Hence, even with fewer resources available for training on the fine-grained problem, the neural networks outperform the baselines. We also highlight the positive effect of the additional features that previous research proposed. Adding the features both in the baselines and in the biLSTM-based architectures improves the INLINEFORM1 scores by several points. Lastly, we compare the performance of the baseline systems with the performance of the state-of-the-art system of BIBREF2 . While BIBREF2 uses n-grams (and character-grams) with INLINEFORM0 , the baseline systems (SVMs, LRs) used in this work use the nbow+ representation, that relies on unigrams. Although they perform on par, the competitive performance of nbow highlights the potential of distributed representations for short-text classification. Further, incorporating structure and distributed representations leads to the gains of the biLSTM network, in the multitask and single task setting. Similar observations can be drawn from Figure FIGREF10 that presents the INLINEFORM0 scores. Again, the biLSTM network with multitask learning achieves the best performance. It is also to be noted that although the two evaluation measures are correlated in the sense that the ranking of the models is the same, small differences in the INLINEFORM1 have large effect on the scores of the INLINEFORM2 measure. Conclusion In this paper, we showed that by jointly learning the tasks of ternary and fine-grained classification with a multitask learning model, one can greatly improve the performance on the second. This opens several avenues for future research. Since sentiment is expressed in different textual types like tweets and paragraph-sized reviews, in different languages (English, German, ..) and in different granularity levels (binary, ternary,..) one can imagine multitask approaches that could benefit from combining such resources. Also, while we opted for biLSTM networks here, one could use convolutional neural networks or even try to combine different types of networks and tasks to investigate the performance effect of multitask learning. Lastly, while our approach mainly relied on the foundations of BIBREF4 , the internal mechanisms and the theoretical guarantees of multitask learning remain to be better understood. Acknowledgements This work is partially supported by the CIFRE N 28/2015.
What dataset did they use?
876700622bd6811d903e65314ac75971bbe23dcc
What Ġdataset Ġdid Ġthey Ġuse ?
Ġhigh - quality Ġdatasets Ġ Ġfrom ĠSem E val - 2016 ĠâĢ ľ Sent iment ĠAnalysis Ġin ĠTwitter âĢ Ŀ Ġtask
<s> What Ġdataset Ġdid Ġthey Ġuse ? </s> Introduction </s> Aut omatic Ġclassification Ġof Ġsentiment Ġhas Ġmainly Ġfocused Ġon Ġcategor izing Ġtweets Ġin Ġeither Ġtwo Ġ( binary Ġsentiment Ġanalysis ) Ġor Ġthree Ġ( tern ary Ġsentiment Ġanalysis ) Ġcategories ĠB IB REF 0 Ġ. ĠIn Ġthis Ġwork Ġwe Ġstudy Ġthe Ġproblem Ġof Ġfine - gr ained Ġsentiment Ġclassification Ġwhere Ġtweets Ġare Ġclassified Ġaccording Ġto Ġa Ġfive - point Ġscale Ġranging Ġfrom ĠVery Neg ative Ġto ĠVery Pos itive . ĠTo Ġillustrate Ġthis , ĠTable ĠT AB REF 3 Ġpresents Ġexamples Ġof Ġtweets Ġassociated Ġwith Ġeach Ġof Ġthese Ġcategories . ĠFive - point Ġscales Ġare Ġwidely Ġadopted Ġin Ġreview Ġsites Ġlike ĠAmazon Ġand ĠTrip Ad visor , Ġwhere Ġa Ġuser 's Ġsentiment Ġis Ġordered Ġwith Ġrespect Ġto Ġits Ġintensity . ĠFrom Ġa Ġsentiment Ġanalysis Ġperspective , Ġthis Ġdefines Ġa Ġclassification Ġproblem Ġwith Ġfive Ġcategories . ĠIn Ġparticular , ĠSebast iani Ġet Ġal . ĠB IB REF 1 Ġdefined Ġsuch Ġclassification Ġproblems Ġwhose Ġcategories Ġare Ġexplicitly Ġordered Ġto Ġbe Ġord inal Ġclassification Ġproblems . ĠTo Ġaccount Ġfor Ġthe Ġordering Ġof Ġthe Ġcategories , Ġlearners Ġare Ġpenal ized Ġaccording Ġto Ġhow Ġfar Ġfrom Ġthe Ġtrue Ġclass Ġtheir Ġpredictions Ġare . </s> Although Ġconsidering Ġdifferent Ġscales , Ġthe Ġvarious Ġsettings Ġof Ġsentiment Ġclassification Ġare Ġrelated . ĠFirst , Ġone Ġmay Ġuse Ġthe Ġsame Ġfeature Ġextraction Ġand Ġengineering Ġapproaches Ġto Ġrepresent Ġthe Ġtext Ġspans Ġsuch Ġas Ġword Ġmembership Ġin Ġlex icons , Ġmorph o - sy nt actic Ġstatistics Ġlike Ġpunct uation Ġor Ġelong ated Ġword Ġcounts ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠSecond , Ġone Ġwould Ġexpect Ġthat Ġknowledge Ġfrom Ġone Ġtask Ġcan Ġbe Ġtransf ered Ġto Ġthe Ġothers Ġand Ġthis Ġwould Ġbenefit Ġthe Ġperformance . ĠKnowing Ġthat Ġa Ġtweet Ġis ĠâĢ ľ Pos itive âĢ Ŀ Ġin Ġthe Ġter n ary Ġsetting Ġnarrow s Ġthe Ġclassification Ġdecision Ġbetween Ġthe ĠVery Pos itive Ġand ĠPositive Ġcategories Ġin Ġthe Ġfine - gr ained Ġsetting . ĠFrom Ġa Ġresearch Ġperspective Ġthis Ġraises Ġthe Ġquestion Ġof Ġwhether Ġand Ġhow Ġone Ġmay Ġbenefit Ġwhen Ġtackling Ġsuch Ġrelated Ġtasks Ġand Ġhow Ġone Ġcan Ġtransfer Ġknowledge Ġfrom Ġone Ġtask Ġto Ġanother Ġduring Ġthe Ġtraining Ġphase . </s> Our Ġfocus Ġin Ġthis Ġwork Ġis Ġto Ġexploit Ġthe Ġrelation Ġbetween Ġthe Ġsentiment Ġclassification Ġsettings Ġand Ġdemonstrate Ġthe Ġbenefits Ġstemming Ġfrom Ġcombining Ġthem . ĠTo Ġthis Ġend , Ġwe Ġpropose Ġto Ġformulate Ġthe Ġdifferent Ġclassification Ġproblems Ġas Ġa Ġmultit ask Ġlearning Ġproblem Ġand Ġjointly Ġlearn Ġthem . ĠMult it ask Ġlearning ĠB IB REF 4 Ġhas Ġshown Ġgreat Ġpotential Ġin Ġvarious Ġdomains Ġand Ġits Ġbenefits Ġhave Ġbeen Ġempir ically Ġvalidated ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġusing Ġdifferent Ġtypes Ġof Ġdata Ġand Ġlearning Ġapproaches . ĠAn Ġimportant Ġbenefit Ġof Ġmultit ask Ġlearning Ġis Ġthat Ġit Ġprovides Ġan Ġelegant Ġway Ġto Ġaccess Ġresources Ġdeveloped Ġfor Ġsimilar Ġtasks . ĠBy Ġjointly Ġlearning Ġcorrelated Ġtasks , Ġthe Ġamount Ġof Ġusable Ġdata Ġincreases . ĠFor Ġinstance , Ġwhile Ġfor Ġter n ary Ġclassification Ġone Ġcan Ġlabel Ġdata Ġusing Ġdistant Ġsupervision Ġwith Ġemot icons ĠB IB REF 9 Ġ, Ġthere Ġis Ġno Ġstraightforward Ġway Ġto Ġdo Ġso Ġfor Ġthe Ġfine - gr ained Ġproblem . ĠHowever , Ġthe Ġlatter Ġcan Ġbenefit Ġindirectly , Ġif Ġthe Ġter n ary Ġand Ġfine - gr ained Ġtasks Ġare Ġlearned Ġjointly . </s> The Ġresearch Ġquestion Ġthat Ġthe Ġpaper Ġattempts Ġto Ġanswer Ġis Ġthe Ġfollowing : ĠCan Ġtwitter Ġsentiment Ġclassification Ġproblems , Ġand Ġfine - gr ained Ġsentiment Ġclassification Ġin Ġparticular , Ġbenefit Ġfrom Ġmultit ask Ġlearning ? ĠTo Ġanswer Ġthe Ġquestion , Ġthe Ġpaper Ġbrings Ġthe Ġfollowing Ġtwo Ġmain Ġcontributions : Ġ( i ) Ġwe Ġshow Ġhow Ġjointly Ġlearning Ġthe Ġter n ary Ġand Ġfine - gr ained Ġsentiment Ġclassification Ġproblems Ġin Ġa Ġmultit ask Ġsetting Ġimproves Ġthe Ġstate - of - the - art Ġperformance , Ġand Ġ( ii ) Ġwe Ġdemonstrate Ġthat Ġrecurrent Ġneural Ġnetworks Ġoutper form Ġmodels Ġpreviously Ġproposed Ġwithout Ġaccess Ġto Ġhuge Ġcorpor a Ġwhile Ġbeing Ġflexible Ġto Ġincorporate Ġdifferent Ġsources Ġof Ġdata . </s> Mult it ask ĠLearning Ġfor ĠTwitter ĠSent iment ĠClassification </s> In Ġhis Ġwork , ĠCar uana ĠB IB REF 4 Ġproposed Ġa Ġmultit ask Ġapproach Ġin Ġwhich Ġa Ġlear ner Ġtakes Ġadvantage Ġof Ġthe Ġmultipl icity Ġof Ġinter dependent Ġtasks Ġwhile Ġjointly Ġlearning Ġthem . ĠThe Ġintuition Ġis Ġthat Ġif Ġthe Ġtasks Ġare Ġcorrelated , Ġthe Ġlear ner Ġcan Ġlearn Ġa Ġmodel Ġjointly Ġfor Ġthem Ġwhile Ġtaking Ġinto Ġaccount Ġthe Ġshared Ġinformation Ġwhich Ġis Ġexpected Ġto Ġimprove Ġits Ġgeneral ization Ġability . ĠPeople Ġexpress Ġtheir Ġopinions Ġonline Ġon Ġvarious Ġsubjects Ġ( events , Ġproducts .. ), Ġon Ġseveral Ġlanguages Ġand Ġin Ġseveral Ġstyles Ġ( t we ets , Ġparagraph - sized Ġreviews .. ), Ġand Ġit Ġis Ġexactly Ġthis Ġvariety Ġthat Ġmotiv ates Ġthe Ġmultit ask Ġapproaches . ĠSpecifically Ġfor ĠTwitter Ġfor Ġinstance , Ġthe Ġdifferent Ġsettings Ġof Ġclassification Ġlike Ġbinary , Ġter n ary Ġand Ġfine - gr ained Ġare Ġcorrelated Ġsince Ġtheir Ġdifference Ġlies Ġin Ġthe Ġsentiment Ġgran ularity Ġof Ġthe Ġclasses Ġwhich Ġincreases Ġwhile Ġmoving Ġfrom Ġbinary Ġto Ġfine - gr ained Ġproblems . </s> There Ġare Ġtwo Ġmain Ġdecisions Ġto Ġbe Ġmade Ġin Ġour Ġapproach : Ġthe Ġlearning Ġalgorithm , Ġwhich Ġlearns Ġa Ġdecision Ġfunction , Ġand Ġthe Ġdata Ġrepresentation . ĠWith Ġrespect Ġto Ġthe Ġformer , Ġneural Ġnetworks Ġare Ġparticularly Ġsuitable Ġas Ġone Ġcan Ġdesign Ġarchitectures Ġwith Ġdifferent Ġproperties Ġand Ġarbitrary Ġcomplexity . ĠAlso , Ġas Ġtraining Ġneural Ġnetwork Ġusually Ġrelies Ġon Ġback - prop ag ation Ġof Ġerrors , Ġone Ġcan Ġhave Ġshared Ġparts Ġof Ġthe Ġnetwork Ġtrained Ġby Ġestimating Ġerrors Ġon Ġthe Ġjoint Ġtasks Ġand Ġothers Ġspecialized Ġfor Ġparticular Ġtasks . ĠCon cerning Ġthe Ġdata Ġrepresentation , Ġit Ġstrongly Ġdepends Ġon Ġthe Ġdata Ġtype Ġavailable . ĠFor Ġthe Ġtask Ġof Ġsentiment Ġclassification Ġof Ġtweets Ġwith Ġneural Ġnetworks , Ġdistributed Ġembed d ings Ġof Ġwords Ġhave Ġshown Ġgreat Ġpotential . ĠEmb edd ings Ġare Ġdefined Ġas Ġlow - dimensional , Ġdense Ġrepresentations Ġof Ġwords Ġthat Ġcan Ġbe Ġobtained Ġin Ġan Ġun super vised Ġfashion Ġby Ġtraining Ġon Ġlarge Ġquantities Ġof Ġtext ĠB IB REF 10 Ġ. </s> Con cerning Ġthe Ġneural Ġnetwork Ġarchitecture , Ġwe Ġfocus Ġon ĠRec urrent ĠNeural ĠNetworks Ġ( R NN s ) Ġthat Ġare Ġcapable Ġof Ġmodeling Ġshort - range Ġand Ġlong - range Ġdependencies Ġlike Ġthose Ġexhibited Ġin Ġsequence Ġdata Ġof Ġarbitrary Ġlength Ġlike Ġtext . ĠWhile Ġin Ġthe Ġtraditional Ġinformation Ġretrieval Ġparadigm Ġsuch Ġdependencies Ġare Ġcaptured Ġusing ĠIN LINE FORM 0 Ġ- gram s Ġand Ġskip - gram s , ĠR NN s Ġlearn Ġto Ġcapture Ġthem Ġautomatically ĠB IB REF 11 Ġ. ĠTo Ġcircumvent Ġthe Ġproblems Ġwith Ġcapturing Ġlong - range Ġdependencies Ġand Ġpreventing Ġgrad ients Ġfrom Ġvanishing , Ġthe Ġlong Ġshort - term Ġmemory Ġnetwork Ġ( L ST M ) Ġwas Ġproposed ĠB IB REF 12 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġuse Ġan Ġextended Ġversion Ġof ĠL ST M Ġcalled Ġbid irection al ĠL ST M Ġ( bi L ST M ). ĠWhile Ġstandard ĠL ST Ms Ġaccess Ġinformation Ġonly Ġfrom Ġthe Ġpast Ġ( pre vious Ġwords ), Ġbi L ST Ms Ġcapture Ġboth Ġpast Ġand Ġfuture Ġinformation Ġeffectively ĠB IB REF 13 Ġ, ĠB IB REF 11 Ġ. ĠThey Ġconsist Ġof Ġtwo ĠL ST M Ġnetworks , Ġfor Ġpropag ating Ġtext Ġforward Ġand Ġbackwards Ġwith Ġthe Ġgoal Ġbeing Ġto Ġcapture Ġthe Ġdependencies Ġbetter . ĠIndeed , Ġprevious Ġwork Ġon Ġmultit ask Ġlearning Ġshowed Ġthe Ġeffectiveness Ġof Ġbi L ST Ms Ġin Ġa Ġvariety Ġof Ġproblems : ĠB IB REF 14 Ġtackled Ġsequence Ġprediction , Ġwhile ĠB IB REF 6 Ġand ĠB IB REF 15 Ġused Ġbi L ST Ms Ġfor ĠNamed ĠEntity ĠRecogn ition Ġand Ġdependency Ġparsing Ġrespectively . </s> Figure ĠFIG REF 2 Ġpresents Ġthe Ġarchitecture Ġwe Ġuse Ġfor Ġmultit ask Ġlearning . ĠIn Ġthe Ġtop - left Ġof Ġthe Ġfigure Ġa Ġbi L ST M Ġnetwork Ġ( en closed Ġby Ġthe Ġdashed Ġline ) Ġis Ġfed Ġwith Ġembed d ings ĠIN LINE FORM 0 Ġthat Ġcorrespond Ġto Ġthe ĠIN LINE FORM 1 Ġwords Ġof Ġa Ġtoken ized Ġtweet . ĠNotice , Ġas Ġdiscussed Ġabove , Ġthe Ġbi L ST M Ġconsists Ġof Ġtwo ĠL ST Ms Ġthat Ġare Ġfed Ġwith Ġthe Ġword Ġsequence Ġforward Ġand Ġbackwards . ĠOn Ġtop Ġof Ġthe Ġbi L ST M Ġnetwork Ġone Ġ( or Ġmore ) Ġhidden Ġlayers ĠIN LINE FORM 2 Ġtransform Ġits Ġoutput . ĠThe Ġoutput Ġof ĠIN LINE FORM 3 Ġis Ġled Ġto Ġthe Ġsoft max Ġlayers Ġfor Ġthe Ġprediction Ġstep . ĠThere Ġare ĠIN LINE FORM 4 Ġsoft max Ġlayers Ġand Ġeach Ġis Ġused Ġfor Ġone Ġof Ġthe ĠIN LINE FORM 5 Ġtasks Ġof Ġthe Ġmultit ask Ġsetting . ĠIn Ġtasks Ġsuch Ġas Ġsentiment Ġclassification , Ġadditional Ġfeatures Ġlike Ġmembership Ġof Ġwords Ġin Ġsentiment Ġlex icons Ġor Ġcounts Ġof Ġelong ated / capital ized Ġwords Ġcan Ġbe Ġused Ġto Ġenrich Ġthe Ġrepresentation Ġof Ġtweets Ġbefore Ġthe Ġclassification Ġstep ĠB IB REF 3 Ġ. ĠThe Ġlower Ġpart Ġof Ġthe Ġnetwork Ġillustrates Ġhow Ġsuch Ġsources Ġof Ġinformation Ġcan Ġbe Ġincorporated Ġto Ġthe Ġprocess . ĠA Ġvector ĠâĢ ľ Additional ĠFeatures âĢ Ŀ Ġfor Ġeach Ġtweet Ġis Ġtransformed Ġfrom Ġthe Ġhidden Ġlayer ( s ) ĠIN LINE FORM 6 Ġand Ġthen Ġis Ġcombined Ġby Ġconc aten ation Ġwith Ġthe Ġtransformed Ġbi L ST M Ġoutput Ġin Ġthe ĠIN LINE FORM 7 Ġlayer . </s> Exper imental Ġsetup </s> Our Ġgoal Ġis Ġto Ġdemonstrate Ġhow Ġmultit ask Ġlearning Ġcan Ġbe Ġsuccessfully Ġapplied Ġon Ġthe Ġtask Ġof Ġsentiment Ġclassification Ġof Ġtweets . ĠThe Ġparticular ities Ġof Ġtweets Ġare Ġto Ġbe Ġshort Ġand Ġinformal Ġtext Ġspans . ĠThe Ġcommon Ġuse Ġof Ġabbrevi ations , Ġcreative Ġlanguage Ġetc ., Ġmakes Ġthe Ġsentiment Ġclassification Ġproblem Ġchallenging . ĠTo Ġvalidate Ġour Ġhypothesis , Ġthat Ġlearning Ġthe Ġtasks Ġjointly Ġcan Ġbenefit Ġthe Ġperformance , Ġwe Ġpropose Ġan Ġexperimental Ġsetting Ġwhere Ġthere Ġare Ġdata Ġfrom Ġtwo Ġdifferent Ġtwitter Ġsentiment Ġclassification Ġproblems : Ġa Ġfine - gr ained Ġand Ġa Ġter n ary . ĠWe Ġconsider Ġthe Ġfine - gr ained Ġtask Ġto Ġbe Ġour Ġprimary Ġtask Ġas Ġit Ġis Ġmore Ġchallenging Ġand Ġobtaining Ġbigger Ġdatasets , Ġe . g . Ġby Ġdistant Ġsupervision , Ġis Ġnot Ġstraightforward Ġand , Ġhence Ġwe Ġreport Ġthe Ġperformance Ġachieved Ġfor Ġthis Ġtask . </s> T ern ary Ġand Ġfine - gr ained Ġsentiment Ġclassification Ġwere Ġpart Ġof Ġthe ĠSem E val - 2016 ĠâĢ ľ Sent iment ĠAnalysis Ġin ĠTwitter âĢ Ŀ Ġtask ĠB IB REF 16 Ġ. ĠWe Ġuse Ġthe Ġhigh - quality Ġdatasets Ġthe Ġchallenge Ġorganizers Ġreleased . ĠThe Ġdataset Ġfor Ġfine - gr ained Ġclassification Ġis Ġsplit Ġin Ġtraining , Ġdevelopment , Ġdevelopment _ test Ġand Ġtest Ġparts . ĠIn Ġthe Ġrest , Ġwe Ġrefer Ġto Ġthese Ġsplits Ġas Ġtrain , Ġdevelopment Ġand Ġtest , Ġwhere Ġtrain Ġis Ġcomposed Ġby Ġthe Ġtraining Ġand Ġthe Ġdevelopment Ġinstances . ĠTable ĠT AB REF 7 Ġpresents Ġan Ġoverview Ġof Ġthe Ġdata . ĠAs Ġdiscussed Ġin ĠB IB REF 16 Ġand Ġillustrated Ġin Ġthe ĠTable , Ġthe Ġfine - gr ained Ġdataset Ġis Ġhighly Ġun balanced Ġand Ġskewed Ġtowards Ġthe Ġpositive Ġsentiment : Ġonly ĠIN LINE FORM 0 Ġof Ġthe Ġtraining Ġexamples Ġare Ġlabeled Ġwith Ġone Ġof Ġthe Ġnegative Ġclasses . </s> Feature Ġrepresentation ĠWe Ġreport Ġresults Ġusing Ġtwo Ġdifferent Ġfeature Ġsets . ĠThe Ġfirst Ġone , Ġdubbed Ġn bow , Ġis Ġa Ġneural Ġbag - of - words Ġthat Ġuses Ġtext Ġembed d ings Ġto Ġgenerate Ġlow - dimensional , Ġdense Ġrepresentations Ġof Ġthe Ġtweets . ĠTo Ġconstruct Ġthe Ġn bow Ġrepresentation , Ġgiven Ġthe Ġword Ġembed d ings Ġdictionary Ġwhere Ġeach Ġword Ġis Ġassociated Ġwith Ġa Ġvector , Ġwe Ġapply Ġthe Ġaverage Ġcompos itional Ġfunction Ġthat Ġaverages Ġthe Ġembed d ings Ġof Ġthe Ġwords Ġthat Ġcompose Ġa Ġtweet . ĠSimple Ġcompos itional Ġfunctions Ġlike Ġaverage Ġwere Ġshown Ġto Ġbe Ġrobust Ġand Ġefficient Ġin Ġprevious Ġwork ĠB IB REF 17 Ġ. ĠInstead Ġof Ġtraining Ġembed d ings Ġfrom Ġscratch , Ġwe Ġuse Ġthe Ġpre - trained Ġon Ġtweets ĠGl o Ve Ġembed d ings Ġof ĠB IB REF 10 Ġ. ĠIn Ġterms Ġof Ġresources Ġrequired , Ġusing Ġonly Ġn bow Ġis Ġefficient Ġas Ġit Ġdoes Ġnot Ġrequire Ġany Ġdomain Ġknowledge . ĠHowever , Ġprevious Ġresearch Ġon Ġsentiment Ġanalysis Ġshowed Ġthat Ġusing Ġextra Ġresources , Ġlike Ġsentiment Ġlex icons , Ġcan Ġbenefit Ġsignificantly Ġthe Ġperformance ĠB IB REF 3 Ġ, ĠB IB REF 2 Ġ. ĠTo Ġvalidate Ġthis Ġand Ġexamine Ġat Ġwhich Ġextent Ġneural Ġnetworks Ġand Ġmultit ask Ġlearning Ġbenefit Ġfrom Ġsuch Ġfeatures Ġwe Ġevaluate Ġthe Ġmodels Ġusing Ġan Ġaugmented Ġversion Ġof Ġn bow , Ġdubbed Ġn bow +. ĠThe Ġfeature Ġspace Ġof Ġthe Ġlatter , Ġis Ġaugmented Ġusing Ġ1 , 368 Ġextra Ġfeatures Ġconsisting Ġmostly Ġof Ġcounts Ġof Ġpunct uation Ġsymbols Ġ(' !? # @ '), Ġemot icons , Ġelong ated Ġwords Ġand Ġword Ġmembership Ġfeatures Ġin Ġseveral Ġsentiment Ġlex icons . ĠDue Ġto Ġspace Ġlimitations , Ġfor Ġa Ġcomplete Ġpresentation Ġof Ġthese Ġfeatures , Ġwe Ġrefer Ġthe Ġinterested Ġreader Ġto ĠB IB REF 2 Ġ, Ġwhose Ġopen Ġimplementation Ġwe Ġused Ġto Ġextract Ġthem . </s> E val uation Ġmeasure ĠTo Ġreproduce Ġthe Ġsetting Ġof Ġthe ĠSem E val Ġchallenges ĠB IB REF 16 Ġ, Ġwe Ġoptimize Ġour Ġsystems Ġusing Ġas Ġprimary Ġmeasure Ġthe Ġmacro - aver aged ĠMean ĠAbsolute ĠError Ġ( ĠIN LINE FORM 0 Ġ) Ġgiven Ġby : ĠIN LINE FORM 1 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġcategories , ĠIN LINE FORM 1 Ġis Ġthe Ġset Ġof Ġinstances Ġwhose Ġtrue Ġclass Ġis ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġis Ġthe Ġtrue Ġlabel Ġof Ġthe Ġinstance ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġthe Ġpredicted Ġlabel . ĠThe Ġmeasure Ġpenal izes Ġdecisions Ġfar Ġfrom Ġthe Ġtrue Ġones Ġand Ġis Ġmacro - aver aged Ġto Ġaccount Ġfor Ġthe Ġfact Ġthat Ġthe Ġdata Ġare Ġun balanced . ĠCom plement ary Ġto ĠIN LINE FORM 6 Ġ, Ġwe Ġreport Ġthe Ġperformance Ġachieved Ġon Ġthe Ġmicro - aver aged ĠIN LINE FORM 7 Ġmeasure , Ġwhich Ġis Ġa Ġcommonly Ġused Ġmeasure Ġfor Ġclassification . </s> The Ġmodels ĠTo Ġevaluate Ġthe Ġmultit ask Ġlearning Ġapproach , Ġwe Ġcompared Ġit Ġwith Ġseveral Ġother Ġmodels . ĠSupport ĠVector ĠMachines Ġ( S V Ms ) Ġare Ġmaximum Ġmargin Ġclassification Ġalgorithms Ġthat Ġhave Ġbeen Ġshown Ġto Ġachieve Ġcompetitive Ġperformance Ġin Ġseveral Ġtext Ġclassification Ġproblems ĠB IB REF 16 Ġ. ĠS VM ĠIN LINE FORM 0 Ġstands Ġfor Ġan ĠS VM Ġwith Ġlinear Ġkernel Ġand Ġan Ġone - vs - rest Ġapproach Ġfor Ġthe Ġmulti - class Ġproblem . ĠAlso , ĠS VM ĠIN LINE FORM 1 Ġis Ġan ĠS VM Ġwith Ġlinear Ġkernel Ġthat Ġemploys Ġthe Ġcram mer - s inger Ġstrategy ĠB IB REF 18 Ġfor Ġthe Ġmulti - class Ġproblem . ĠLog istic Ġregression Ġ( LR ) Ġis Ġanother Ġtype Ġof Ġlinear Ġclassification Ġmethod , Ġwith Ġprob abil istic Ġmotivation . ĠAgain , Ġwe Ġuse Ġtwo Ġtypes Ġof ĠLog istic ĠReg ression Ġdepending Ġon Ġthe Ġmulti - class Ġstrategy : ĠLR ĠIN LINE FORM 2 Ġthat Ġuses Ġan Ġone - vs - rest Ġapproach Ġand Ġmult in omial ĠLog istic ĠReg ression Ġalso Ġknown Ġas Ġthe ĠMax Ent Ġclass ifier Ġthat Ġuses Ġa Ġmult in omial Ġcriterion . </s> Both ĠSV Ms Ġand ĠL Rs Ġas Ġdiscussed Ġabove Ġtreat Ġthe Ġproblem Ġas Ġa Ġmulti - class Ġone , Ġwithout Ġconsidering Ġthe Ġordering Ġof Ġthe Ġclasses . ĠFor Ġthese Ġfour Ġmodels , Ġwe Ġtuned Ġthe Ġhyper - param eter ĠIN LINE FORM 0 Ġthat Ġcontrols Ġthe Ġimportance Ġof Ġthe ĠL ĠIN LINE FORM 1 Ġregular ization Ġpart Ġin Ġthe Ġoptimization Ġproblem Ġwith Ġgrid - search Ġover ĠIN LINE FORM 2 Ġusing Ġ10 - fold Ġcross - valid ation Ġin Ġthe Ġunion Ġof Ġthe Ġtraining Ġand Ġdevelopment Ġdata Ġand Ġthen Ġret rained Ġthe Ġmodels Ġwith Ġthe Ġselected Ġvalues . ĠAlso , Ġto Ġaccount Ġfor Ġthe Ġun balanced Ġclassification Ġproblem Ġwe Ġused Ġclass Ġweights Ġto Ġpenal ize Ġmore Ġthe Ġerrors Ġmade Ġon Ġthe Ġrare Ġclasses . ĠThese Ġweights Ġwere Ġin versely Ġproportional Ġto Ġthe Ġfrequency Ġof Ġeach Ġclass . ĠFor Ġthe Ġfour Ġmodels Ġwe Ġused Ġthe Ġimplementations Ġof ĠSci kit - learn ĠB IB REF 19 Ġ. </s> For Ġmultit ask Ġlearning Ġwe Ġuse Ġthe Ġarchitecture Ġshown Ġin ĠFigure ĠFIG REF 2 Ġ, Ġwhich Ġwe Ġimplemented Ġwith ĠKer as ĠB IB REF 20 Ġ. ĠThe Ġembed d ings Ġare Ġinitialized Ġwith Ġthe Ġ50 - dimensional ĠGl o Ve Ġembed d ings Ġwhile Ġthe Ġoutput Ġof Ġthe Ġbi L ST M Ġnetwork Ġis Ġset Ġto Ġdimension Ġ50 . ĠThe Ġactivation Ġfunction Ġof Ġthe Ġhidden Ġlayers Ġis Ġthe Ġhyper b olic Ġtang ent . ĠThe Ġweights Ġof Ġthe Ġlayers Ġwere Ġinitialized Ġfrom Ġa Ġuniform Ġdistribution , Ġscaled Ġas Ġdescribed Ġin ĠB IB REF 21 Ġ. ĠWe Ġused Ġthe ĠRoot ĠMean ĠSquare ĠProp ag ation Ġoptimization Ġmethod . ĠWe Ġused Ġdrop out Ġfor Ġregular izing Ġthe Ġnetwork . ĠWe Ġtrained Ġthe Ġnetwork Ġusing Ġbatches Ġof Ġ128 Ġexamples Ġas Ġfollows : Ġbefore Ġselecting Ġthe Ġbatch , Ġwe Ġperform Ġa ĠBern ou lli Ġtrial Ġwith Ġprobability ĠIN LINE FORM 0 Ġto Ġselect Ġthe Ġtask Ġto Ġtrain Ġfor . ĠWith Ġprobability ĠIN LINE FORM 1 Ġwe Ġpick Ġa Ġbatch Ġfor Ġthe Ġfine - gr ained Ġsentiment Ġclassification Ġproblem , Ġwhile Ġwith Ġprobability ĠIN LINE FORM 2 Ġwe Ġpick Ġa Ġbatch Ġfor Ġthe Ġter n ary Ġproblem . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 2 Ġ, Ġthe Ġerror Ġis Ġback prop ag ated Ġuntil Ġthe Ġembed d ings , Ġthat Ġwe Ġfine - t une Ġduring Ġthe Ġlearning Ġprocess . ĠNotice Ġalso Ġthat Ġthe Ġweights Ġof Ġthe Ġnetwork Ġuntil Ġthe Ġlayer ĠIN LINE FORM 3 Ġare Ġshared Ġand Ġtherefore Ġaffected Ġby Ġboth Ġtasks . </s> To Ġtune Ġthe Ġneural Ġnetwork Ġhyper - param eters Ġwe Ġused Ġ5 - fold Ġcross Ġvalidation . ĠWe Ġtuned Ġthe Ġprobability ĠIN LINE FORM 0 Ġof Ġdrop out Ġafter Ġthe Ġhidden Ġlayers ĠIN LINE FORM 1 Ġand Ġfor Ġthe Ġbi L ST M Ġfor ĠIN LINE FORM 2 Ġ, Ġthe Ġsize Ġof Ġthe Ġhidden Ġlayer ĠIN LINE FORM 3 Ġand Ġthe Ġprobability ĠIN LINE FORM 4 Ġof Ġthe ĠBern ou lli Ġtrials Ġfrom ĠIN LINE FORM 5 Ġ. ĠDuring Ġtraining , Ġwe Ġmonitor Ġthe Ġnetwork 's Ġperformance Ġon Ġthe Ġdevelopment Ġset Ġand Ġapply Ġearly Ġstopping Ġif Ġthe Ġperformance Ġon Ġthe Ġvalidation Ġset Ġdoes Ġnot Ġimprove Ġfor Ġ5 Ġconsecutive Ġepoch s . </s> Exper imental Ġresults ĠTable ĠT AB REF 9 Ġillustrates Ġthe Ġperformance Ġof Ġthe Ġmodels Ġfor Ġthe Ġdifferent Ġdata Ġrepresentations . ĠThe Ġupper Ġpart Ġof Ġthe ĠTable Ġsummarizes Ġthe Ġperformance Ġof Ġthe Ġbas elines . ĠThe Ġentry ĠâĢ ľ Bal ik as Ġet Ġal . âĢ Ŀ Ġstands Ġfor Ġthe Ġwinning Ġsystem Ġof Ġthe Ġ2016 Ġedition Ġof Ġthe Ġchallenge ĠB IB REF 2 Ġ, Ġwhich Ġto Ġthe Ġbest Ġof Ġour Ġknowledge Ġholds Ġthe Ġstate - of - the - art . ĠDue Ġto Ġthe Ġsto ch astic ity Ġof Ġtraining Ġthe Ġbi L ST M Ġmodels , Ġwe Ġrepeat Ġthe Ġexperiment Ġ10 Ġtimes Ġand Ġreport Ġthe Ġaverage Ġand Ġthe Ġstandard Ġdeviation Ġof Ġthe Ġperformance Ġachieved . </s> Several Ġobservations Ġcan Ġbe Ġmade Ġfrom Ġthe Ġtable . ĠFirst Ġnotice Ġthat , Ġoverall , Ġthe Ġbest Ġperformance Ġis Ġachieved Ġby Ġthe Ġneural Ġnetwork Ġarchitecture Ġthat Ġuses Ġmultit ask Ġlearning . ĠThis Ġentails Ġthat Ġthe Ġsystem Ġmakes Ġuse Ġof Ġthe Ġavailable Ġresources Ġefficiently Ġand Ġimproves Ġthe Ġstate - of - the - art Ġperformance . ĠIn Ġconjunction Ġwith Ġthe Ġfact Ġthat Ġwe Ġfound Ġthe Ġoptimal Ġprobability ĠIN LINE FORM 0 Ġ, Ġthis Ġhighlights Ġthe Ġbenefits Ġof Ġmultit ask Ġlearning Ġover Ġsingle Ġtask Ġlearning . ĠFurthermore , Ġas Ġdescribed Ġabove , Ġthe Ġneural Ġnetwork - based Ġmodels Ġhave Ġonly Ġaccess Ġto Ġthe Ġtraining Ġdata Ġas Ġthe Ġdevelopment Ġare Ġhold Ġfor Ġearly Ġstopping . ĠOn Ġthe Ġother Ġhand , Ġthe Ġbaseline Ġsystems Ġwere Ġret rained Ġon Ġthe Ġunion Ġof Ġthe Ġtrain Ġand Ġdevelopment Ġsets . ĠHence , Ġeven Ġwith Ġfewer Ġresources Ġavailable Ġfor Ġtraining Ġon Ġthe Ġfine - gr ained Ġproblem , Ġthe Ġneural Ġnetworks Ġoutper form Ġthe Ġbas elines . ĠWe Ġalso Ġhighlight Ġthe Ġpositive Ġeffect Ġof Ġthe Ġadditional Ġfeatures Ġthat Ġprevious Ġresearch Ġproposed . ĠAdding Ġthe Ġfeatures Ġboth Ġin Ġthe Ġbas elines Ġand Ġin Ġthe Ġbi L ST M - based Ġarchitectures Ġimproves Ġthe ĠIN LINE FORM 1 Ġscores Ġby Ġseveral Ġpoints . </s> Lastly , Ġwe Ġcompare Ġthe Ġperformance Ġof Ġthe Ġbaseline Ġsystems Ġwith Ġthe Ġperformance Ġof Ġthe Ġstate - of - the - art Ġsystem Ġof ĠB IB REF 2 Ġ. ĠWhile ĠB IB REF 2 Ġuses Ġn - gram s Ġ( and Ġcharacter - gram s ) Ġwith ĠIN LINE FORM 0 Ġ, Ġthe Ġbaseline Ġsystems Ġ( S V Ms , ĠL Rs ) Ġused Ġin Ġthis Ġwork Ġuse Ġthe Ġn bow + Ġrepresentation , Ġthat Ġrelies Ġon Ġun ig rams . ĠAlthough Ġthey Ġperform Ġon Ġpar , Ġthe Ġcompetitive Ġperformance Ġof Ġn bow Ġhighlights Ġthe Ġpotential Ġof Ġdistributed Ġrepresentations Ġfor Ġshort - text Ġclassification . ĠFurther , Ġincorporating Ġstructure Ġand Ġdistributed Ġrepresentations Ġleads Ġto Ġthe Ġgains Ġof Ġthe Ġbi L ST M Ġnetwork , Ġin Ġthe Ġmultit ask Ġand Ġsingle Ġtask Ġsetting . </s> Similar Ġobservations Ġcan Ġbe Ġdrawn Ġfrom ĠFigure ĠFIG REF 10 Ġthat Ġpresents Ġthe ĠIN LINE FORM 0 Ġscores . ĠAgain , Ġthe Ġbi L ST M Ġnetwork Ġwith Ġmultit ask Ġlearning Ġachieves Ġthe Ġbest Ġperformance . ĠIt Ġis Ġalso Ġto Ġbe Ġnoted Ġthat Ġalthough Ġthe Ġtwo Ġevaluation Ġmeasures Ġare Ġcorrelated Ġin Ġthe Ġsense Ġthat Ġthe Ġranking Ġof Ġthe Ġmodels Ġis Ġthe Ġsame , Ġsmall Ġdifferences Ġin Ġthe ĠIN LINE FORM 1 Ġhave Ġlarge Ġeffect Ġon Ġthe Ġscores Ġof Ġthe ĠIN LINE FORM 2 Ġmeasure . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġshowed Ġthat Ġby Ġjointly Ġlearning Ġthe Ġtasks Ġof Ġter n ary Ġand Ġfine - gr ained Ġclassification Ġwith Ġa Ġmultit ask Ġlearning Ġmodel , Ġone Ġcan Ġgreatly Ġimprove Ġthe Ġperformance Ġon Ġthe Ġsecond . ĠThis Ġopens Ġseveral Ġavenues Ġfor Ġfuture Ġresearch . ĠSince Ġsentiment Ġis Ġexpressed Ġin Ġdifferent Ġtextual Ġtypes Ġlike Ġtweets Ġand Ġparagraph - sized Ġreviews , Ġin Ġdifferent Ġlanguages Ġ( English , ĠGerman , Ġ.. ) Ġand Ġin Ġdifferent Ġgran ularity Ġlevels Ġ( binary , Ġter n ary , .. ) Ġone Ġcan Ġimagine Ġmultit ask Ġapproaches Ġthat Ġcould Ġbenefit Ġfrom Ġcombining Ġsuch Ġresources . ĠAlso , Ġwhile Ġwe Ġopted Ġfor Ġbi L ST M Ġnetworks Ġhere , Ġone Ġcould Ġuse Ġconv olution al Ġneural Ġnetworks Ġor Ġeven Ġtry Ġto Ġcombine Ġdifferent Ġtypes Ġof Ġnetworks Ġand Ġtasks Ġto Ġinvestigate Ġthe Ġperformance Ġeffect Ġof Ġmultit ask Ġlearning . ĠLastly , Ġwhile Ġour Ġapproach Ġmainly Ġrelied Ġon Ġthe Ġfoundations Ġof ĠB IB REF 4 Ġ, Ġthe Ġinternal Ġmechanisms Ġand Ġthe Ġtheoretical Ġguarantees Ġof Ġmultit ask Ġlearning Ġremain Ġto Ġbe Ġbetter Ġunderstood . </s> Acknowled gements </s> This Ġwork Ġis Ġpartially Ġsupported Ġby Ġthe ĠC IF RE ĠN Ġ28 / 2015 .