Galatasaray, Fenerbahçe
EXTRACTIVE
Galatasaray , Fenerbahçe 
EXTRACTIVE
We have decided to consider tweets about popular sports clubs as our domain for stance detection. Considerable amounts of tweets are being published for sports-related events at every instant. Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey. As is the case for the sentiment analysis tools, the outputs of the stance detection systems on a stream of tweets about these clubs can facilitate the use of the opinions of the football followers by these clubs.
We have decided to consider tweets about popular sports clubs as our domain for stance detection. Considerable amounts of tweets are being published for sports-related events at every instant. Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey. As is the case for the sentiment analysis tools, the outputs of the stance detection systems on a stream of tweets about these clubs can facilitate the use of the opinions of the football followers by these clubs.
1706.06894
Introduction </s> St ance Ġdetection Ġ( also Ġcalled Ġstance Ġidentification Ġor Ġstance Ġclassification ) Ġis Ġone Ġof Ġthe Ġconsiderably Ġrecent Ġresearch Ġtopics Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ). ĠIt Ġis Ġusually Ġdefined Ġas Ġa Ġclassification Ġproblem Ġwhere Ġfor Ġa Ġtext Ġand Ġtarget Ġpair , Ġthe Ġstance Ġof Ġthe Ġauthor Ġof Ġthe Ġtext Ġfor Ġthat Ġtarget Ġis Ġexpected Ġas Ġa Ġclassification Ġoutput Ġfrom Ġthe Ġset : Ġ{ F avor , ĠAgainst , ĠNeither } ĠB IB REF 0 Ġ. </s> St ance Ġdetection Ġis Ġusually Ġconsidered Ġas Ġa Ġsubt ask Ġof Ġsentiment Ġanalysis Ġ( op inion Ġmining ) ĠB IB REF 1 Ġtopic Ġin ĠN LP . ĠBoth Ġare Ġmostly Ġperformed Ġon Ġsocial Ġmedia Ġtexts , Ġparticularly Ġon Ġtweets , Ġhence Ġboth Ġare Ġimportant Ġcomponents Ġof Ġsocial Ġmedia Ġanalysis . ĠNevertheless , Ġin Ġsentiment Ġanalysis , Ġthe Ġsentiment Ġof Ġthe Ġauthor Ġof Ġa Ġpiece Ġof Ġtext Ġusually Ġas ĠPositive , ĠNegative , Ġand ĠNeutral Ġis Ġexplored Ġwhile Ġin Ġstance Ġdetection , Ġthe Ġstance Ġof Ġthe Ġauthor Ġof Ġthe Ġtext Ġfor Ġa Ġparticular Ġtarget Ġ( an Ġentity , Ġevent , Ġetc .) Ġeither Ġexplicitly Ġor Ġimplicitly Ġreferred Ġto Ġin Ġthe Ġtext Ġis Ġconsidered . ĠLike Ġsentiment Ġanalysis , Ġstance Ġdetection Ġsystems Ġcan Ġbe Ġvaluable Ġcomponents Ġof Ġinformation Ġretrieval Ġand Ġother Ġtext Ġanalysis Ġsystems ĠB IB REF 0 Ġ. </s> Previous Ġwork Ġon Ġstance Ġdetection Ġinclude ĠB IB REF 2 Ġwhere Ġa Ġstance Ġclass ifier Ġbased Ġon Ġsentiment Ġand Ġarguing Ġfeatures Ġis Ġproposed Ġin Ġaddition Ġto Ġan Ġarguing Ġlex icon Ġautomatically Ġcompiled . ĠThe Ġultimate Ġapproach Ġperforms Ġbetter Ġthan Ġdistribution - based Ġand Ġun i - gram - based Ġbaseline Ġsystems ĠB IB REF 2 Ġ. ĠIn ĠB IB REF 3 Ġ, Ġthe Ġauthors Ġshow Ġthat Ġthe Ġuse Ġof Ġdialogue Ġstructure Ġimproves Ġstance Ġdetection Ġin Ġon - line Ġdebates . ĠIn ĠB IB REF 4 Ġ, ĠHasan Ġand ĠNg Ġcarry Ġout Ġstance Ġdetection Ġexperiments Ġusing Ġdifferent Ġmachine Ġlearning Ġalgorithms , Ġtraining Ġdata Ġsets , Ġfeatures , Ġand Ġinter - post Ġconstraints Ġin Ġon - line Ġdebates , Ġand Ġdraw Ġinsightful Ġconclusions Ġbased Ġon Ġthese Ġexperiments . ĠFor Ġinstance , Ġthey Ġfind Ġthat Ġsequence Ġmodels Ġlike ĠHM Ms Ġperform Ġbetter Ġat Ġstance Ġdetection Ġwhen Ġcompared Ġwith Ġnon - sequence Ġmodels Ġlike ĠNa ive ĠBay es Ġ( NB ) ĠB IB REF 4 Ġ. ĠIn Ġanother Ġrelated Ġstudy ĠB IB REF 5 Ġ, Ġthe Ġauthors Ġconclude Ġthat Ġtopic - independent Ġfeatures Ġcan Ġbe Ġexploited Ġfor Ġdisagreement Ġdetection Ġin Ġon - line Ġdialog ues . ĠThe Ġemployed Ġfeatures Ġinclude Ġagreement , Ġcue Ġwords , Ġdenial , Ġhed ges , Ġduration , Ġpol arity , Ġand Ġpunct uation ĠB IB REF 5 Ġ. ĠSt ance Ġdetection Ġon Ġa Ġcorpus Ġof Ġstudent Ġessays Ġis Ġconsidered Ġin ĠB IB REF 6 Ġ. ĠAfter Ġusing Ġlingu istically - mot ivated Ġfeature Ġsets Ġtogether Ġwith Ġmult ival ued ĠNB Ġand ĠS VM Ġas Ġthe Ġlearning Ġmodels , Ġthe Ġauthors Ġconclude Ġthat Ġthey Ġoutper form Ġtwo Ġbaseline Ġapproaches ĠB IB REF 6 Ġ. ĠIn ĠB IB REF 7 Ġ, Ġthe Ġauthor Ġclaims Ġthat ĠWikipedia Ġcan Ġbe Ġused Ġto Ġdetermine Ġstances Ġabout Ġcontroversial Ġtopics Ġbased Ġon Ġtheir Ġprevious Ġwork Ġregarding Ġcontroversy Ġextraction Ġon Ġthe ĠWeb . </s> Among Ġmore Ġrecent Ġrelated Ġwork , Ġin ĠB IB REF 8 Ġstance Ġdetection Ġfor Ġunseen Ġtargets Ġis Ġstudied Ġand Ġbid irection al Ġconditional Ġencoding Ġis Ġemployed . ĠThe Ġauthors Ġstate Ġthat Ġtheir Ġapproach Ġachieves Ġstate - of - the Ġart Ġperformance Ġrates ĠB IB REF 8 Ġon ĠSem E val Ġ2016 ĠTwitter ĠSt ance ĠDetection Ġcorpus ĠB IB REF 0 Ġ. ĠIn ĠB IB REF 9 Ġ, Ġa Ġstance - community Ġdetection Ġapproach Ġcalled ĠSC IF NET Ġis Ġproposed . ĠSC IF NET Ġcreates Ġnetworks Ġof Ġpeople Ġwho Ġare Ġstance Ġtargets , Ġautomatically Ġfrom Ġthe Ġrelated Ġdocument Ġcollections ĠB IB REF 9 Ġusing Ġstance Ġexpansion Ġand Ġrefinement Ġtechniques Ġto Ġarrive Ġat Ġstance - co herent Ġnetworks . ĠA Ġtweet Ġdata Ġset Ġannot ated Ġwith Ġstance Ġinformation Ġregarding Ġsix Ġpred efined Ġtargets Ġis Ġproposed Ġin ĠB IB REF 10 Ġwhere Ġthis Ġdata Ġset Ġis Ġannot ated Ġthrough Ġcrowds ourcing . ĠThe Ġauthors Ġindicate Ġthat Ġthe Ġdata Ġset Ġis Ġalso Ġannot ated Ġwith Ġsentiment Ġinformation Ġin Ġaddition Ġto Ġstance , Ġso Ġit Ġcan Ġhelp Ġreveal Ġassociations Ġbetween Ġstance Ġand Ġsentiment ĠB IB REF 10 Ġ. ĠLastly , Ġin ĠB IB REF 0 Ġ, ĠSem E val Ġ2016 's Ġaforementioned Ġshared Ġtask Ġon ĠTwitter ĠSt ance ĠDetection Ġis Ġdescribed . ĠAlso Ġprovided Ġare Ġthe Ġresults Ġof Ġthe Ġevaluations Ġof Ġ19 Ġsystems Ġparticipating Ġin Ġtwo Ġsubt asks Ġ( one Ġwith Ġtraining Ġdata Ġset Ġprovided Ġand Ġthe Ġother Ġwithout Ġan Ġannot ated Ġdata Ġset ) Ġof Ġthe Ġshared Ġtask ĠB IB REF 0 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġtweet Ġdata Ġset Ġin ĠTurkish Ġannot ated Ġwith Ġstance Ġinformation , Ġwhere Ġthe Ġcorresponding Ġannotations Ġare Ġmade Ġpublicly Ġavailable . ĠThe Ġdomain Ġof Ġthe Ġtweets Ġcomprises Ġtwo Ġpopular Ġfootball Ġclubs Ġwhich Ġconstitute Ġthe Ġtargets Ġof Ġthe Ġtweets Ġincluded . ĠWe Ġalso Ġprovide Ġthe Ġevaluation Ġresults Ġof ĠS VM Ġclass ifiers Ġ( for Ġeach Ġtarget ) Ġon Ġthis Ġdata Ġset Ġusing Ġun ig ram , Ġbig ram , Ġand Ġhashtag Ġfeatures . </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthe Ġcurrent Ġstudy Ġis Ġthe Ġfirst Ġone Ġto Ġtarget Ġat Ġstance Ġdetection Ġin ĠTurkish Ġtweets . ĠTogether Ġwith Ġthe Ġprovided Ġannot ated Ġdata Ġset Ġand Ġthe Ġcorresponding Ġevaluations Ġwith Ġthe Ġaforementioned ĠS VM Ġclass ifiers Ġwhich Ġcan Ġbe Ġused Ġas Ġbaseline Ġsystems , Ġour Ġstudy Ġwill Ġhopefully Ġhelp Ġincrease Ġsocial Ġmedia Ġanalysis Ġstudies Ġon ĠTurkish Ġcontent . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows : ĠIn ĠSection ĠSEC REF 2 Ġ, Ġwe Ġdescribe Ġour Ġtweet Ġdata Ġset Ġannot ated Ġwith Ġthe Ġtarget Ġand Ġstance Ġinformation . ĠSection ĠSEC REF 3 Ġincludes Ġthe Ġdetails Ġof Ġour ĠS VM - based Ġstance Ġclass ifiers Ġand Ġtheir Ġevaluation Ġresults Ġwith Ġdiscussions . ĠSection ĠSEC REF 4 Ġincludes Ġfuture Ġresearch Ġtopics Ġbased Ġon Ġthe Ġcurrent Ġstudy , Ġand Ġfinally ĠSection ĠSEC REF 5 Ġconcludes Ġthe Ġpaper Ġwith Ġa Ġsummary . </s> A ĠSt ance ĠDetection ĠData ĠSet </s> We Ġhave Ġdecided Ġto Ġconsider Ġtweets Ġabout Ġpopular Ġsports Ġclubs Ġas Ġour Ġdomain Ġfor Ġstance Ġdetection . ĠConsider able Ġamounts Ġof Ġtweets Ġare Ġbeing Ġpublished Ġfor Ġsports - related Ġevents Ġat Ġevery Ġinstant . ĠHence Ġwe Ġhave Ġdetermined Ġour Ġtargets Ġas ĠGal at asar ay Ġ( name ly ĠTarget - 1 ) Ġand ĠF ener bah Ã§ e Ġ( name ly , ĠTarget - 2 ) Ġwhich Ġare Ġtwo Ġof Ġthe Ġmost Ġpopular Ġfootball Ġclubs Ġin ĠTurkey . ĠAs Ġis Ġthe Ġcase Ġfor Ġthe Ġsentiment Ġanalysis Ġtools , Ġthe Ġoutputs Ġof Ġthe Ġstance Ġdetection Ġsystems Ġon Ġa Ġstream Ġof Ġtweets Ġabout Ġthese Ġclubs Ġcan Ġfacilitate Ġthe Ġuse Ġof Ġthe Ġopinions Ġof Ġthe Ġfootball Ġfollowers Ġby Ġthese Ġclubs . </s> In Ġa Ġprevious Ġstudy Ġon Ġthe Ġidentification Ġof Ġpublic Ġhealth - related Ġtweets , Ġtwo Ġtweet Ġdata Ġsets Ġin ĠTurkish Ġ( each Ġset Ġcontaining Ġ1 Ġmillion Ġrandom Ġtweets ) Ġhave Ġbeen Ġcompiled Ġwhere Ġthese Ġsets Ġbelong Ġto Ġtwo Ġdifferent Ġperiods Ġof Ġ20 Ġconsecutive Ġdays ĠB IB REF 11 Ġ. ĠWe Ġhave Ġdecided Ġto Ġuse Ġone Ġof Ġthese Ġsets Ġ( cor respond ing Ġto Ġthe Ġperiod Ġbetween ĠAugust Ġ18 Ġand ĠSeptember Ġ6 , Ġ2015 ) Ġand Ġfirst ly Ġfiltered Ġthe Ġtweets Ġusing Ġthe Ġpossible Ġnames Ġused Ġto Ġrefer Ġto Ġthe Ġtarget Ġclubs . ĠThen , Ġwe Ġhave Ġannot ated Ġthe Ġstance Ġinformation Ġin Ġthe Ġtweets Ġfor Ġthese Ġtargets Ġas ĠFavor Ġor ĠAgainst . ĠWithin Ġthe Ġcourse Ġof Ġthis Ġstudy , Ġwe Ġhave Ġnot Ġconsidered Ġthose Ġtweets Ġin Ġwhich Ġthe Ġtarget Ġis Ġnot Ġexplicitly Ġmentioned , Ġas Ġour Ġinitial Ġfiltering Ġprocess Ġreveals . </s> For Ġthe Ġpurposes Ġof Ġthe Ġcurrent Ġstudy , Ġwe Ġhave Ġnot Ġannot ated Ġany Ġtweets Ġwith Ġthe ĠNeither Ġclass . ĠThis Ġstance Ġclass Ġand Ġeven Ġfiner - gr ained Ġclasses Ġcan Ġbe Ġconsidered Ġin Ġfurther Ġannotation Ġstudies . ĠWe Ġshould Ġalso Ġnote Ġthat Ġin Ġa Ġfew Ġtweets , Ġthe Ġtarget Ġof Ġthe Ġstance Ġwas Ġthe Ġmanagement Ġof Ġthe Ġclub Ġwhile Ġin Ġsome Ġothers Ġa Ġparticular Ġfootballer Ġof Ġthe Ġclub Ġis Ġpraised Ġor Ġcriticised . ĠStill , Ġwe Ġhave Ġconsidered Ġthe Ġclub Ġas Ġthe Ġtarget Ġof Ġthe Ġstance Ġin Ġall Ġof Ġthe Ġcases Ġand Ġcarried Ġout Ġour Ġannotations Ġaccordingly . </s> At Ġthe Ġend Ġof Ġthe Ġannotation Ġprocess , Ġwe Ġhave Ġannot ated Ġ700 Ġtweets , Ġwhere Ġ175 Ġtweets Ġare Ġin Ġfavor Ġof Ġand Ġ175 Ġtweets Ġare Ġagainst ĠTarget - 1 , Ġand Ġsimilarly Ġ175 Ġtweets Ġare Ġin Ġfavor Ġof Ġand Ġ175 Ġare Ġagainst ĠTarget - 2 . ĠHence , Ġour Ġdata Ġset Ġis Ġa Ġbalanced Ġone Ġalthough Ġit Ġis Ġcurrently Ġlimited Ġin Ġsize . ĠThe Ġcorresponding Ġstance Ġannotations Ġare Ġmade Ġpublicly Ġavailable Ġat Ġhttp :// c eng . met u . edu . tr / ĠIN LINE FORM 0 Ġe 12 03 29 / ĠTurkish _ St ance _ Det ection _ Tweet _ Dat as et . csv Ġin ĠComm a ĠSep ar ated ĠValues Ġ( CS V ) Ġformat . ĠThe Ġfile Ġcontains Ġthree Ġcolumns Ġwith Ġthe Ġcorresponding Ġheaders . ĠThe Ġfirst Ġcolumn Ġis Ġthe Ġtweet Ġid Ġof Ġthe Ġcorresponding Ġtweet , Ġthe Ġsecond Ġcolumn Ġcontains Ġthe Ġname Ġof Ġthe Ġstance Ġtarget , Ġand Ġthe Ġlast Ġcolumn Ġincludes Ġthe Ġstance Ġof Ġthe Ġtweet Ġfor Ġthe Ġtarget Ġas ĠFavor Ġor ĠAgainst . </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġpublicly - available Ġstance - annot ated Ġdata Ġset Ġfor ĠTurkish . ĠHence , Ġit Ġis Ġa Ġsignificant Ġresource Ġas Ġthere Ġis Ġa Ġscarcity Ġof Ġannot ated Ġdata Ġsets , Ġlinguistic Ġresources , Ġand ĠN LP Ġtools Ġavailable Ġfor ĠTurkish . ĠAdditionally , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġit Ġis Ġalso Ġsignificant Ġfor Ġbeing Ġthe Ġfirst Ġstance - annot ated Ġdata Ġset Ġincluding Ġsports - related Ġtweets , Ġas Ġprevious Ġstance Ġdetection Ġdata Ġsets Ġmostly Ġinclude Ġon - line Ġtexts Ġon Ġpolitical / ethical Ġissues . </s> St ance ĠDetection ĠExper iments ĠUsing ĠS VM ĠClass ifiers </s> It Ġis Ġemphasized Ġin Ġthe Ġrelated Ġliterature Ġthat Ġun ig ram - based Ġmethods Ġare Ġreliable Ġfor Ġthe Ġstance Ġdetection Ġtask ĠB IB REF 2 Ġand Ġsimilarly Ġun ig ram - based Ġmodels Ġhave Ġbeen Ġused Ġas Ġbaseline Ġmodels Ġin Ġstudies Ġsuch Ġas ĠB IB REF 0 Ġ. ĠIn Ġorder Ġto Ġbe Ġused Ġas Ġa Ġbaseline Ġand Ġreference Ġsystem Ġfor Ġfurther Ġstudies Ġon Ġstance Ġdetection Ġin ĠTurkish Ġtweets , Ġwe Ġhave Ġtrained Ġtwo ĠS VM Ġclass ifiers Ġ( one Ġfor Ġeach Ġtarget ) Ġusing Ġun ig rams Ġas Ġfeatures . ĠBefore Ġthe Ġextraction Ġof Ġun ig rams , Ġwe Ġhave Ġemployed Ġautomated Ġpre processing Ġto Ġfilter Ġout Ġthe Ġstop words Ġin Ġour Ġannot ated Ġdata Ġset Ġof Ġ700 Ġtweets . ĠThe Ġstop word Ġlist Ġused Ġis Ġthe Ġlist Ġpresented Ġin ĠB IB REF 12 Ġwhich , Ġin Ġturn , Ġis Ġthe Ġslightly Ġextended Ġversion Ġof Ġthe Ġstop word Ġlist Ġprovided Ġin ĠB IB REF 13 Ġ. </s> We Ġhave Ġused Ġthe ĠS VM Ġimplementation Ġavailable Ġin Ġthe ĠWe ka Ġdata Ġmining Ġapplication ĠB IB REF 14 Ġwhere Ġthis Ġparticular Ġimplementation Ġemploys Ġthe ĠSM O Ġalgorithm ĠB IB REF 15 Ġto Ġtrain Ġa Ġclass ifier Ġwith Ġa Ġlinear Ġkernel . ĠThe Ġ10 - fold Ġcross - valid ation Ġresults Ġof Ġthe Ġtwo Ġclass ifiers Ġare Ġprovided Ġin ĠTable ĠT AB REF 1 Ġusing Ġthe Ġmetrics Ġof Ġprecision , Ġrecall , Ġand ĠF - Measure . </s> The Ġevaluation Ġresults Ġare Ġquite Ġfavorable Ġfor Ġboth Ġtargets Ġand Ġparticularly Ġhigher Ġfor ĠTarget - 1 , Ġconsidering Ġthe Ġfact Ġthat Ġthey Ġare Ġthe Ġinitial Ġexperiments Ġon Ġthe Ġdata Ġset . ĠThe Ġperformance Ġof Ġthe Ġclass ifiers Ġis Ġbetter Ġfor Ġthe ĠFavor Ġclass Ġfor Ġboth Ġtargets Ġwhen Ġcompared Ġwith Ġthe Ġperformance Ġresults Ġfor Ġthe ĠAgainst Ġclass . ĠThis Ġoutcome Ġmay Ġbe Ġdue Ġto Ġthe Ġcommon Ġuse Ġof Ġsome Ġterms Ġwhen Ġexpressing Ġpositive Ġstance Ġtowards Ġsports Ġclubs Ġin ĠTurkish Ġtweets . ĠThe Ġsame Ġpercentage Ġof Ġcommon Ġterms Ġmay Ġnot Ġhave Ġbeen Ġobserved Ġin Ġtweets Ġduring Ġthe Ġexpression Ġof Ġnegative Ġstances Ġtowards Ġthe Ġtargets . ĠYet , Ġcompletely Ġthe Ġopposite Ġpattern Ġis Ġobserved Ġin Ġstance Ġdetection Ġresults Ġof Ġbaseline Ġsystems Ġgiven Ġin ĠB IB REF 0 Ġ, Ġi . e ., Ġbetter ĠF - Measure Ġrates Ġhave Ġbeen Ġobtained Ġfor Ġthe ĠAgainst Ġclass Ġwhen Ġcompared Ġwith Ġthe ĠFavor Ġclass ĠB IB REF 0 Ġ. ĠSome Ġof Ġthe Ġbaseline Ġsystems Ġreported Ġin ĠB IB REF 0 Ġare ĠS VM - based Ġsystems Ġusing Ġun ig rams Ġand Ġn gram s Ġas Ġfeatures Ġsimilar Ġto Ġour Ġstudy , Ġbut Ġtheir Ġdata Ġsets Ġinclude Ġall Ġthree Ġstance Ġclasses Ġof ĠFavor , ĠAgainst , Ġand ĠNeither , Ġwhile Ġour Ġdata Ġset Ġcomprises Ġonly Ġtweets Ġclassified Ġas Ġbelonging Ġto ĠFavor Ġor ĠAgainst Ġclasses . ĠAnother Ġdifference Ġis Ġthat Ġthe Ġdata Ġsets Ġin ĠB IB REF 0 Ġhave Ġbeen Ġdivided Ġinto Ġtraining Ġand Ġtest Ġsets , Ġwhile Ġin Ġour Ġstudy Ġwe Ġprovide Ġ10 - fold Ġcross - valid ation Ġresults Ġon Ġthe Ġwhole Ġdata Ġset . ĠOn Ġthe Ġother Ġhand , Ġwe Ġshould Ġalso Ġnote Ġthat ĠS VM - based Ġsentiment Ġanalysis Ġsystems Ġ( such Ġas Ġthose Ġgiven Ġin ĠB IB REF 16 Ġ) Ġhave Ġbeen Ġreported Ġto Ġachieve Ġbetter ĠF - Measure Ġrates Ġfor Ġthe ĠPositive Ġsentiment Ġclass Ġwhen Ġcompared Ġwith Ġthe Ġresults Ġobtained Ġfor Ġthe ĠNegative Ġclass . ĠTherefore , Ġour Ġevaluation Ġresults Ġfor Ġeach Ġstance Ġclass Ġseem Ġto Ġbe Ġin Ġline Ġwith Ġsuch Ġsentiment Ġanalysis Ġsystems . ĠYet , Ġfurther Ġexperiments Ġon Ġthe Ġextended Ġversions Ġof Ġour Ġdata Ġset Ġshould Ġbe Ġconducted Ġand Ġthe Ġresults Ġshould Ġagain Ġbe Ġcompared Ġwith Ġthe Ġstance Ġdetection Ġresults Ġgiven Ġin Ġthe Ġliterature . </s> We Ġhave Ġalso Ġevaluated ĠS VM Ġclass ifiers Ġwhich Ġuse Ġonly Ġbig rams Ġas Ġfeatures , Ġas Ġn gram - based Ġclass ifiers Ġhave Ġbeen Ġreported Ġto Ġperform Ġbetter Ġfor Ġthe Ġstance Ġdetection Ġproblem ĠB IB REF 0 Ġ. ĠHowever , Ġwe Ġhave Ġobserved Ġthat Ġusing Ġbig rams Ġas Ġthe Ġsole Ġfeatures Ġof Ġthe ĠS VM Ġclass ifiers Ġleads Ġto Ġquite Ġpoor Ġresults . ĠThis Ġobservation Ġmay Ġbe Ġdue Ġto Ġthe Ġrelatively Ġlimited Ġsize Ġof Ġthe Ġtweet Ġdata Ġset Ġemployed . ĠStill , Ġwe Ġcan Ġconclude Ġthat Ġun ig ram - based Ġfeatures Ġlead Ġto Ġsuperior Ġresults Ġcompared Ġto Ġthe Ġresults Ġobtained Ġusing Ġbig rams Ġas Ġfeatures , Ġbased Ġon Ġour Ġexperiments Ġon Ġour Ġdata Ġset . ĠYet , Ġn gram - based Ġfeatures Ġmay Ġbe Ġemployed Ġon Ġthe Ġextended Ġversions Ġof Ġthe Ġdata Ġset Ġto Ġverify Ġthis Ġconclusion Ġwithin Ġthe Ġcourse Ġof Ġfuture Ġwork . </s> With Ġan Ġintention Ġto Ġexploit Ġthe Ġcontribution Ġof Ġhashtag Ġuse Ġto Ġstance Ġdetection , Ġwe Ġhave Ġalso Ġused Ġthe Ġexistence Ġof Ġhasht ags Ġin Ġtweets Ġas Ġan Ġadditional Ġfeature Ġto Ġun ig rams . ĠThe Ġcorresponding Ġevaluation Ġresults Ġof Ġthe ĠS VM Ġclass ifiers Ġusing Ġun ig rams Ġtogether Ġthe Ġexistence Ġof Ġhasht ags Ġas Ġfeatures Ġare Ġprovided Ġin ĠTable ĠT AB REF 2 Ġ. </s> When Ġthe Ġresults Ġgiven Ġin ĠTable ĠT AB REF 2 Ġare Ġcompared Ġwith Ġthe Ġresults Ġin ĠTable ĠT AB REF 1 Ġ, Ġa Ġslight Ġdecrease Ġin ĠF - Measure Ġ( 0 . 5 %) Ġfor ĠTarget - 1 Ġis Ġobserved , Ġwhile Ġthe Ġoverall ĠF - Measure Ġvalue Ġfor ĠTarget - 2 Ġhas Ġincreased Ġby Ġ1 . 8 %. ĠAlthough Ġwe Ġcould Ġnot Ġderive Ġsound Ġconclusions Ġmainly Ġdue Ġto Ġthe Ġrelatively Ġsmall Ġsize Ġof Ġour Ġdata Ġset , Ġthe Ġincrease Ġin Ġthe Ġperformance Ġof Ġthe ĠS VM Ġclass ifier ĠTarget - 2 Ġis Ġan Ġencouraging Ġevidence Ġfor Ġthe Ġexploitation Ġof Ġhasht ags Ġin Ġa Ġstance Ġdetection Ġsystem . ĠWe Ġleave Ġother Ġways Ġof Ġexploiting Ġhasht ags Ġfor Ġstance Ġdetection Ġas Ġa Ġfuture Ġwork . </s> To Ġsum Ġup , Ġour Ġevaluation Ġresults Ġare Ġsignificant Ġas Ġreference Ġresults Ġto Ġbe Ġused Ġfor Ġcomparison Ġpurposes Ġand Ġprovides Ġevidence Ġfor Ġthe Ġutility Ġof Ġun ig ram - based Ġand Ġhashtag - related Ġfeatures Ġin ĠS VM Ġclass ifiers Ġfor Ġthe Ġstance Ġdetection Ġproblem Ġin ĠTurkish Ġtweets . </s> Future ĠProspect s </s> Future Ġwork Ġbased Ġon Ġthe Ġcurrent Ġstudy Ġincludes Ġthe Ġfollowing : </s> Conclusion </s> St ance Ġdetection Ġis Ġa Ġconsiderably Ġnew Ġresearch Ġarea Ġin Ġnatural Ġlanguage Ġprocessing Ġand Ġis Ġconsidered Ġwithin Ġthe Ġscope Ġof Ġthe Ġwell - stud ied Ġtopic Ġof Ġsentiment Ġanalysis . ĠIt Ġis Ġthe Ġdetection Ġof Ġstance Ġwithin Ġtext Ġtowards Ġa Ġtarget Ġwhich Ġmay Ġbe Ġexplicitly Ġspecified Ġin Ġthe Ġtext Ġor Ġnot . ĠIn Ġthis Ġstudy , Ġwe Ġpresent Ġa Ġstance - annot ated Ġtweet Ġdata Ġset Ġin ĠTurkish Ġwhere Ġthe Ġtargets Ġof Ġthe Ġannot ated Ġstances Ġare Ġtwo Ġpopular Ġsports Ġclubs Ġin ĠTurkey . ĠThe Ġcorresponding Ġannotations Ġare Ġmade Ġpublicly - available Ġfor Ġresearch Ġpurposes . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġstance Ġdetection Ġdata Ġset Ġfor Ġthe ĠTurkish Ġlanguage Ġand Ġalso Ġthe Ġfirst Ġsports - related Ġstance - annot ated Ġdata Ġset . ĠAlso Ġpresented Ġin Ġthis Ġstudy Ġare ĠS VM Ġclass ifiers Ġ( one Ġfor Ġeach Ġtarget ) Ġutilizing Ġun ig ram Ġand Ġbig ram Ġfeatures Ġin Ġaddition Ġto Ġusing Ġthe Ġexistence Ġof Ġhasht ags Ġas Ġanother Ġfeature . Ġ10 - fold Ġcross Ġvalidation Ġresults Ġof Ġthese Ġclass ifiers Ġare Ġpresented Ġwhich Ġcan Ġbe Ġused Ġas Ġreference Ġresults Ġby Ġprospective Ġsystems . ĠBoth Ġthe Ġannot ated Ġdata Ġset Ġand Ġthe Ġclass ifiers Ġwith Ġevaluations Ġare Ġsignificant Ġsince Ġthey Ġare Ġthe Ġinitial Ġcontributions Ġto Ġstance Ġdetection Ġproblem Ġin ĠTurkish Ġtweets .
Introduction Stance detection (also called stance identification or stance classification) is one of the considerably recent research topics in natural language processing (NLP). It is usually defined as a classification problem where for a text and target pair, the stance of the author of the text for that target is expected as a classification output from the set: {Favor, Against, Neither} BIBREF0 . Stance detection is usually considered as a subtask of sentiment analysis (opinion mining) BIBREF1 topic in NLP. Both are mostly performed on social media texts, particularly on tweets, hence both are important components of social media analysis. Nevertheless, in sentiment analysis, the sentiment of the author of a piece of text usually as Positive, Negative, and Neutral is explored while in stance detection, the stance of the author of the text for a particular target (an entity, event, etc.) either explicitly or implicitly referred to in the text is considered. Like sentiment analysis, stance detection systems can be valuable components of information retrieval and other text analysis systems BIBREF0 . Previous work on stance detection include BIBREF2 where a stance classifier based on sentiment and arguing features is proposed in addition to an arguing lexicon automatically compiled. The ultimate approach performs better than distribution-based and uni-gram-based baseline systems BIBREF2 . In BIBREF3 , the authors show that the use of dialogue structure improves stance detection in on-line debates. In BIBREF4 , Hasan and Ng carry out stance detection experiments using different machine learning algorithms, training data sets, features, and inter-post constraints in on-line debates, and draw insightful conclusions based on these experiments. For instance, they find that sequence models like HMMs perform better at stance detection when compared with non-sequence models like Naive Bayes (NB) BIBREF4 . In another related study BIBREF5 , the authors conclude that topic-independent features can be exploited for disagreement detection in on-line dialogues. The employed features include agreement, cue words, denial, hedges, duration, polarity, and punctuation BIBREF5 . Stance detection on a corpus of student essays is considered in BIBREF6 . After using linguistically-motivated feature sets together with multivalued NB and SVM as the learning models, the authors conclude that they outperform two baseline approaches BIBREF6 . In BIBREF7 , the author claims that Wikipedia can be used to determine stances about controversial topics based on their previous work regarding controversy extraction on the Web. Among more recent related work, in BIBREF8 stance detection for unseen targets is studied and bidirectional conditional encoding is employed. The authors state that their approach achieves state-of-the art performance rates BIBREF8 on SemEval 2016 Twitter Stance Detection corpus BIBREF0 . In BIBREF9 , a stance-community detection approach called SCIFNET is proposed. SCIFNET creates networks of people who are stance targets, automatically from the related document collections BIBREF9 using stance expansion and refinement techniques to arrive at stance-coherent networks. A tweet data set annotated with stance information regarding six predefined targets is proposed in BIBREF10 where this data set is annotated through crowdsourcing. The authors indicate that the data set is also annotated with sentiment information in addition to stance, so it can help reveal associations between stance and sentiment BIBREF10 . Lastly, in BIBREF0 , SemEval 2016's aforementioned shared task on Twitter Stance Detection is described. Also provided are the results of the evaluations of 19 systems participating in two subtasks (one with training data set provided and the other without an annotated data set) of the shared task BIBREF0 . In this paper, we present a tweet data set in Turkish annotated with stance information, where the corresponding annotations are made publicly available. The domain of the tweets comprises two popular football clubs which constitute the targets of the tweets included. We also provide the evaluation results of SVM classifiers (for each target) on this data set using unigram, bigram, and hashtag features. To the best of our knowledge, the current study is the first one to target at stance detection in Turkish tweets. Together with the provided annotated data set and the corresponding evaluations with the aforementioned SVM classifiers which can be used as baseline systems, our study will hopefully help increase social media analysis studies on Turkish content. The rest of the paper is organized as follows: In Section SECREF2 , we describe our tweet data set annotated with the target and stance information. Section SECREF3 includes the details of our SVM-based stance classifiers and their evaluation results with discussions. Section SECREF4 includes future research topics based on the current study, and finally Section SECREF5 concludes the paper with a summary. A Stance Detection Data Set We have decided to consider tweets about popular sports clubs as our domain for stance detection. Considerable amounts of tweets are being published for sports-related events at every instant. Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey. As is the case for the sentiment analysis tools, the outputs of the stance detection systems on a stream of tweets about these clubs can facilitate the use of the opinions of the football followers by these clubs. In a previous study on the identification of public health-related tweets, two tweet data sets in Turkish (each set containing 1 million random tweets) have been compiled where these sets belong to two different periods of 20 consecutive days BIBREF11 . We have decided to use one of these sets (corresponding to the period between August 18 and September 6, 2015) and firstly filtered the tweets using the possible names used to refer to the target clubs. Then, we have annotated the stance information in the tweets for these targets as Favor or Against. Within the course of this study, we have not considered those tweets in which the target is not explicitly mentioned, as our initial filtering process reveals. For the purposes of the current study, we have not annotated any tweets with the Neither class. This stance class and even finer-grained classes can be considered in further annotation studies. We should also note that in a few tweets, the target of the stance was the management of the club while in some others a particular footballer of the club is praised or criticised. Still, we have considered the club as the target of the stance in all of the cases and carried out our annotations accordingly. At the end of the annotation process, we have annotated 700 tweets, where 175 tweets are in favor of and 175 tweets are against Target-1, and similarly 175 tweets are in favor of and 175 are against Target-2. Hence, our data set is a balanced one although it is currently limited in size. The corresponding stance annotations are made publicly available at http://ceng.metu.edu.tr/ INLINEFORM0 e120329/ Turkish_Stance_Detection_Tweet_Dataset.csv in Comma Separated Values (CSV) format. The file contains three columns with the corresponding headers. The first column is the tweet id of the corresponding tweet, the second column contains the name of the stance target, and the last column includes the stance of the tweet for the target as Favor or Against. To the best of our knowledge, this is the first publicly-available stance-annotated data set for Turkish. Hence, it is a significant resource as there is a scarcity of annotated data sets, linguistic resources, and NLP tools available for Turkish. Additionally, to the best of our knowledge, it is also significant for being the first stance-annotated data set including sports-related tweets, as previous stance detection data sets mostly include on-line texts on political/ethical issues. Stance Detection Experiments Using SVM Classifiers It is emphasized in the related literature that unigram-based methods are reliable for the stance detection task BIBREF2 and similarly unigram-based models have been used as baseline models in studies such as BIBREF0 . In order to be used as a baseline and reference system for further studies on stance detection in Turkish tweets, we have trained two SVM classifiers (one for each target) using unigrams as features. Before the extraction of unigrams, we have employed automated preprocessing to filter out the stopwords in our annotated data set of 700 tweets. The stopword list used is the list presented in BIBREF12 which, in turn, is the slightly extended version of the stopword list provided in BIBREF13 . We have used the SVM implementation available in the Weka data mining application BIBREF14 where this particular implementation employs the SMO algorithm BIBREF15 to train a classifier with a linear kernel. The 10-fold cross-validation results of the two classifiers are provided in Table TABREF1 using the metrics of precision, recall, and F-Measure. The evaluation results are quite favorable for both targets and particularly higher for Target-1, considering the fact that they are the initial experiments on the data set. The performance of the classifiers is better for the Favor class for both targets when compared with the performance results for the Against class. This outcome may be due to the common use of some terms when expressing positive stance towards sports clubs in Turkish tweets. The same percentage of common terms may not have been observed in tweets during the expression of negative stances towards the targets. Yet, completely the opposite pattern is observed in stance detection results of baseline systems given in BIBREF0 , i.e., better F-Measure rates have been obtained for the Against class when compared with the Favor class BIBREF0 . Some of the baseline systems reported in BIBREF0 are SVM-based systems using unigrams and ngrams as features similar to our study, but their data sets include all three stance classes of Favor, Against, and Neither, while our data set comprises only tweets classified as belonging to Favor or Against classes. Another difference is that the data sets in BIBREF0 have been divided into training and test sets, while in our study we provide 10-fold cross-validation results on the whole data set. On the other hand, we should also note that SVM-based sentiment analysis systems (such as those given in BIBREF16 ) have been reported to achieve better F-Measure rates for the Positive sentiment class when compared with the results obtained for the Negative class. Therefore, our evaluation results for each stance class seem to be in line with such sentiment analysis systems. Yet, further experiments on the extended versions of our data set should be conducted and the results should again be compared with the stance detection results given in the literature. We have also evaluated SVM classifiers which use only bigrams as features, as ngram-based classifiers have been reported to perform better for the stance detection problem BIBREF0 . However, we have observed that using bigrams as the sole features of the SVM classifiers leads to quite poor results. This observation may be due to the relatively limited size of the tweet data set employed. Still, we can conclude that unigram-based features lead to superior results compared to the results obtained using bigrams as features, based on our experiments on our data set. Yet, ngram-based features may be employed on the extended versions of the data set to verify this conclusion within the course of future work. With an intention to exploit the contribution of hashtag use to stance detection, we have also used the existence of hashtags in tweets as an additional feature to unigrams. The corresponding evaluation results of the SVM classifiers using unigrams together the existence of hashtags as features are provided in Table TABREF2 . When the results given in Table TABREF2 are compared with the results in Table TABREF1 , a slight decrease in F-Measure (0.5%) for Target-1 is observed, while the overall F-Measure value for Target-2 has increased by 1.8%. Although we could not derive sound conclusions mainly due to the relatively small size of our data set, the increase in the performance of the SVM classifier Target-2 is an encouraging evidence for the exploitation of hashtags in a stance detection system. We leave other ways of exploiting hashtags for stance detection as a future work. To sum up, our evaluation results are significant as reference results to be used for comparison purposes and provides evidence for the utility of unigram-based and hashtag-related features in SVM classifiers for the stance detection problem in Turkish tweets. Future Prospects Future work based on the current study includes the following: Conclusion Stance detection is a considerably new research area in natural language processing and is considered within the scope of the well-studied topic of sentiment analysis. It is the detection of stance within text towards a target which may be explicitly specified in the text or not. In this study, we present a stance-annotated tweet data set in Turkish where the targets of the annotated stances are two popular sports clubs in Turkey. The corresponding annotations are made publicly-available for research purposes. To the best of our knowledge, this is the first stance detection data set for the Turkish language and also the first sports-related stance-annotated data set. Also presented in this study are SVM classifiers (one for each target) utilizing unigram and bigram features in addition to using the existence of hashtags as another feature. 10-fold cross validation results of these classifiers are presented which can be used as reference results by prospective systems. Both the annotated data set and the classifiers with evaluations are significant since they are the initial contributions to stance detection problem in Turkish tweets.
Which sports clubs are the targets?
ff6c9af28f0e2bb4fb6a69f124665f8ceb966fbc
Which Ġsports Ġclubs Ġare Ġthe Ġtargets ?
Gal at asar ay , ĠF ener bah Ã§ e
<s> Which Ġsports Ġclubs Ġare Ġthe Ġtargets ? </s> Introduction </s> St ance Ġdetection Ġ( also Ġcalled Ġstance Ġidentification Ġor Ġstance Ġclassification ) Ġis Ġone Ġof Ġthe Ġconsiderably Ġrecent Ġresearch Ġtopics Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ). ĠIt Ġis Ġusually Ġdefined Ġas Ġa Ġclassification Ġproblem Ġwhere Ġfor Ġa Ġtext Ġand Ġtarget Ġpair , Ġthe Ġstance Ġof Ġthe Ġauthor Ġof Ġthe Ġtext Ġfor Ġthat Ġtarget Ġis Ġexpected Ġas Ġa Ġclassification Ġoutput Ġfrom Ġthe Ġset : Ġ{ F avor , ĠAgainst , ĠNeither } ĠB IB REF 0 Ġ. </s> St ance Ġdetection Ġis Ġusually Ġconsidered Ġas Ġa Ġsubt ask Ġof Ġsentiment Ġanalysis Ġ( op inion Ġmining ) ĠB IB REF 1 Ġtopic Ġin ĠN LP . ĠBoth Ġare Ġmostly Ġperformed Ġon Ġsocial Ġmedia Ġtexts , Ġparticularly Ġon Ġtweets , Ġhence Ġboth Ġare Ġimportant Ġcomponents Ġof Ġsocial Ġmedia Ġanalysis . ĠNevertheless , Ġin Ġsentiment Ġanalysis , Ġthe Ġsentiment Ġof Ġthe Ġauthor Ġof Ġa Ġpiece Ġof Ġtext Ġusually Ġas ĠPositive , ĠNegative , Ġand ĠNeutral Ġis Ġexplored Ġwhile Ġin Ġstance Ġdetection , Ġthe Ġstance Ġof Ġthe Ġauthor Ġof Ġthe Ġtext Ġfor Ġa Ġparticular Ġtarget Ġ( an Ġentity , Ġevent , Ġetc .) Ġeither Ġexplicitly Ġor Ġimplicitly Ġreferred Ġto Ġin Ġthe Ġtext Ġis Ġconsidered . ĠLike Ġsentiment Ġanalysis , Ġstance Ġdetection Ġsystems Ġcan Ġbe Ġvaluable Ġcomponents Ġof Ġinformation Ġretrieval Ġand Ġother Ġtext Ġanalysis Ġsystems ĠB IB REF 0 Ġ. </s> Previous Ġwork Ġon Ġstance Ġdetection Ġinclude ĠB IB REF 2 Ġwhere Ġa Ġstance Ġclass ifier Ġbased Ġon Ġsentiment Ġand Ġarguing Ġfeatures Ġis Ġproposed Ġin Ġaddition Ġto Ġan Ġarguing Ġlex icon Ġautomatically Ġcompiled . ĠThe Ġultimate Ġapproach Ġperforms Ġbetter Ġthan Ġdistribution - based Ġand Ġun i - gram - based Ġbaseline Ġsystems ĠB IB REF 2 Ġ. ĠIn ĠB IB REF 3 Ġ, Ġthe Ġauthors Ġshow Ġthat Ġthe Ġuse Ġof Ġdialogue Ġstructure Ġimproves Ġstance Ġdetection Ġin Ġon - line Ġdebates . ĠIn ĠB IB REF 4 Ġ, ĠHasan Ġand ĠNg Ġcarry Ġout Ġstance Ġdetection Ġexperiments Ġusing Ġdifferent Ġmachine Ġlearning Ġalgorithms , Ġtraining Ġdata Ġsets , Ġfeatures , Ġand Ġinter - post Ġconstraints Ġin Ġon - line Ġdebates , Ġand Ġdraw Ġinsightful Ġconclusions Ġbased Ġon Ġthese Ġexperiments . ĠFor Ġinstance , Ġthey Ġfind Ġthat Ġsequence Ġmodels Ġlike ĠHM Ms Ġperform Ġbetter Ġat Ġstance Ġdetection Ġwhen Ġcompared Ġwith Ġnon - sequence Ġmodels Ġlike ĠNa ive ĠBay es Ġ( NB ) ĠB IB REF 4 Ġ. ĠIn Ġanother Ġrelated Ġstudy ĠB IB REF 5 Ġ, Ġthe Ġauthors Ġconclude Ġthat Ġtopic - independent Ġfeatures Ġcan Ġbe Ġexploited Ġfor Ġdisagreement Ġdetection Ġin Ġon - line Ġdialog ues . ĠThe Ġemployed Ġfeatures Ġinclude Ġagreement , Ġcue Ġwords , Ġdenial , Ġhed ges , Ġduration , Ġpol arity , Ġand Ġpunct uation ĠB IB REF 5 Ġ. ĠSt ance Ġdetection Ġon Ġa Ġcorpus Ġof Ġstudent Ġessays Ġis Ġconsidered Ġin ĠB IB REF 6 Ġ. ĠAfter Ġusing Ġlingu istically - mot ivated Ġfeature Ġsets Ġtogether Ġwith Ġmult ival ued ĠNB Ġand ĠS VM Ġas Ġthe Ġlearning Ġmodels , Ġthe Ġauthors Ġconclude Ġthat Ġthey Ġoutper form Ġtwo Ġbaseline Ġapproaches ĠB IB REF 6 Ġ. ĠIn ĠB IB REF 7 Ġ, Ġthe Ġauthor Ġclaims Ġthat ĠWikipedia Ġcan Ġbe Ġused Ġto Ġdetermine Ġstances Ġabout Ġcontroversial Ġtopics Ġbased Ġon Ġtheir Ġprevious Ġwork Ġregarding Ġcontroversy Ġextraction Ġon Ġthe ĠWeb . </s> Among Ġmore Ġrecent Ġrelated Ġwork , Ġin ĠB IB REF 8 Ġstance Ġdetection Ġfor Ġunseen Ġtargets Ġis Ġstudied Ġand Ġbid irection al Ġconditional Ġencoding Ġis Ġemployed . ĠThe Ġauthors Ġstate Ġthat Ġtheir Ġapproach Ġachieves Ġstate - of - the Ġart Ġperformance Ġrates ĠB IB REF 8 Ġon ĠSem E val Ġ2016 ĠTwitter ĠSt ance ĠDetection Ġcorpus ĠB IB REF 0 Ġ. ĠIn ĠB IB REF 9 Ġ, Ġa Ġstance - community Ġdetection Ġapproach Ġcalled ĠSC IF NET Ġis Ġproposed . ĠSC IF NET Ġcreates Ġnetworks Ġof Ġpeople Ġwho Ġare Ġstance Ġtargets , Ġautomatically Ġfrom Ġthe Ġrelated Ġdocument Ġcollections ĠB IB REF 9 Ġusing Ġstance Ġexpansion Ġand Ġrefinement Ġtechniques Ġto Ġarrive Ġat Ġstance - co herent Ġnetworks . ĠA Ġtweet Ġdata Ġset Ġannot ated Ġwith Ġstance Ġinformation Ġregarding Ġsix Ġpred efined Ġtargets Ġis Ġproposed Ġin ĠB IB REF 10 Ġwhere Ġthis Ġdata Ġset Ġis Ġannot ated Ġthrough Ġcrowds ourcing . ĠThe Ġauthors Ġindicate Ġthat Ġthe Ġdata Ġset Ġis Ġalso Ġannot ated Ġwith Ġsentiment Ġinformation Ġin Ġaddition Ġto Ġstance , Ġso Ġit Ġcan Ġhelp Ġreveal Ġassociations Ġbetween Ġstance Ġand Ġsentiment ĠB IB REF 10 Ġ. ĠLastly , Ġin ĠB IB REF 0 Ġ, ĠSem E val Ġ2016 's Ġaforementioned Ġshared Ġtask Ġon ĠTwitter ĠSt ance ĠDetection Ġis Ġdescribed . ĠAlso Ġprovided Ġare Ġthe Ġresults Ġof Ġthe Ġevaluations Ġof Ġ19 Ġsystems Ġparticipating Ġin Ġtwo Ġsubt asks Ġ( one Ġwith Ġtraining Ġdata Ġset Ġprovided Ġand Ġthe Ġother Ġwithout Ġan Ġannot ated Ġdata Ġset ) Ġof Ġthe Ġshared Ġtask ĠB IB REF 0 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġtweet Ġdata Ġset Ġin ĠTurkish Ġannot ated Ġwith Ġstance Ġinformation , Ġwhere Ġthe Ġcorresponding Ġannotations Ġare Ġmade Ġpublicly Ġavailable . ĠThe Ġdomain Ġof Ġthe Ġtweets Ġcomprises Ġtwo Ġpopular Ġfootball Ġclubs Ġwhich Ġconstitute Ġthe Ġtargets Ġof Ġthe Ġtweets Ġincluded . ĠWe Ġalso Ġprovide Ġthe Ġevaluation Ġresults Ġof ĠS VM Ġclass ifiers Ġ( for Ġeach Ġtarget ) Ġon Ġthis Ġdata Ġset Ġusing Ġun ig ram , Ġbig ram , Ġand Ġhashtag Ġfeatures . </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthe Ġcurrent Ġstudy Ġis Ġthe Ġfirst Ġone Ġto Ġtarget Ġat Ġstance Ġdetection Ġin ĠTurkish Ġtweets . ĠTogether Ġwith Ġthe Ġprovided Ġannot ated Ġdata Ġset Ġand Ġthe Ġcorresponding Ġevaluations Ġwith Ġthe Ġaforementioned ĠS VM Ġclass ifiers Ġwhich Ġcan Ġbe Ġused Ġas Ġbaseline Ġsystems , Ġour Ġstudy Ġwill Ġhopefully Ġhelp Ġincrease Ġsocial Ġmedia Ġanalysis Ġstudies Ġon ĠTurkish Ġcontent . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows : ĠIn ĠSection ĠSEC REF 2 Ġ, Ġwe Ġdescribe Ġour Ġtweet Ġdata Ġset Ġannot ated Ġwith Ġthe Ġtarget Ġand Ġstance Ġinformation . ĠSection ĠSEC REF 3 Ġincludes Ġthe Ġdetails Ġof Ġour ĠS VM - based Ġstance Ġclass ifiers Ġand Ġtheir Ġevaluation Ġresults Ġwith Ġdiscussions . ĠSection ĠSEC REF 4 Ġincludes Ġfuture Ġresearch Ġtopics Ġbased Ġon Ġthe Ġcurrent Ġstudy , Ġand Ġfinally ĠSection ĠSEC REF 5 Ġconcludes Ġthe Ġpaper Ġwith Ġa Ġsummary . </s> A ĠSt ance ĠDetection ĠData ĠSet </s> We Ġhave Ġdecided Ġto Ġconsider Ġtweets Ġabout Ġpopular Ġsports Ġclubs Ġas Ġour Ġdomain Ġfor Ġstance Ġdetection . ĠConsider able Ġamounts Ġof Ġtweets Ġare Ġbeing Ġpublished Ġfor Ġsports - related Ġevents Ġat Ġevery Ġinstant . ĠHence Ġwe Ġhave Ġdetermined Ġour Ġtargets Ġas ĠGal at asar ay Ġ( name ly ĠTarget - 1 ) Ġand ĠF ener bah Ã§ e Ġ( name ly , ĠTarget - 2 ) Ġwhich Ġare Ġtwo Ġof Ġthe Ġmost Ġpopular Ġfootball Ġclubs Ġin ĠTurkey . ĠAs Ġis Ġthe Ġcase Ġfor Ġthe Ġsentiment Ġanalysis Ġtools , Ġthe Ġoutputs Ġof Ġthe Ġstance Ġdetection Ġsystems Ġon Ġa Ġstream Ġof Ġtweets Ġabout Ġthese Ġclubs Ġcan Ġfacilitate Ġthe Ġuse Ġof Ġthe Ġopinions Ġof Ġthe Ġfootball Ġfollowers Ġby Ġthese Ġclubs . </s> In Ġa Ġprevious Ġstudy Ġon Ġthe Ġidentification Ġof Ġpublic Ġhealth - related Ġtweets , Ġtwo Ġtweet Ġdata Ġsets Ġin ĠTurkish Ġ( each Ġset Ġcontaining Ġ1 Ġmillion Ġrandom Ġtweets ) Ġhave Ġbeen Ġcompiled Ġwhere Ġthese Ġsets Ġbelong Ġto Ġtwo Ġdifferent Ġperiods Ġof Ġ20 Ġconsecutive Ġdays ĠB IB REF 11 Ġ. ĠWe Ġhave Ġdecided Ġto Ġuse Ġone Ġof Ġthese Ġsets Ġ( cor respond ing Ġto Ġthe Ġperiod Ġbetween ĠAugust Ġ18 Ġand ĠSeptember Ġ6 , Ġ2015 ) Ġand Ġfirst ly Ġfiltered Ġthe Ġtweets Ġusing Ġthe Ġpossible Ġnames Ġused Ġto Ġrefer Ġto Ġthe Ġtarget Ġclubs . ĠThen , Ġwe Ġhave Ġannot ated Ġthe Ġstance Ġinformation Ġin Ġthe Ġtweets Ġfor Ġthese Ġtargets Ġas ĠFavor Ġor ĠAgainst . ĠWithin Ġthe Ġcourse Ġof Ġthis Ġstudy , Ġwe Ġhave Ġnot Ġconsidered Ġthose Ġtweets Ġin Ġwhich Ġthe Ġtarget Ġis Ġnot Ġexplicitly Ġmentioned , Ġas Ġour Ġinitial Ġfiltering Ġprocess Ġreveals . </s> For Ġthe Ġpurposes Ġof Ġthe Ġcurrent Ġstudy , Ġwe Ġhave Ġnot Ġannot ated Ġany Ġtweets Ġwith Ġthe ĠNeither Ġclass . ĠThis Ġstance Ġclass Ġand Ġeven Ġfiner - gr ained Ġclasses Ġcan Ġbe Ġconsidered Ġin Ġfurther Ġannotation Ġstudies . ĠWe Ġshould Ġalso Ġnote Ġthat Ġin Ġa Ġfew Ġtweets , Ġthe Ġtarget Ġof Ġthe Ġstance Ġwas Ġthe Ġmanagement Ġof Ġthe Ġclub Ġwhile Ġin Ġsome Ġothers Ġa Ġparticular Ġfootballer Ġof Ġthe Ġclub Ġis Ġpraised Ġor Ġcriticised . ĠStill , Ġwe Ġhave Ġconsidered Ġthe Ġclub Ġas Ġthe Ġtarget Ġof Ġthe Ġstance Ġin Ġall Ġof Ġthe Ġcases Ġand Ġcarried Ġout Ġour Ġannotations Ġaccordingly . </s> At Ġthe Ġend Ġof Ġthe Ġannotation Ġprocess , Ġwe Ġhave Ġannot ated Ġ700 Ġtweets , Ġwhere Ġ175 Ġtweets Ġare Ġin Ġfavor Ġof Ġand Ġ175 Ġtweets Ġare Ġagainst ĠTarget - 1 , Ġand Ġsimilarly Ġ175 Ġtweets Ġare Ġin Ġfavor Ġof Ġand Ġ175 Ġare Ġagainst ĠTarget - 2 . ĠHence , Ġour Ġdata Ġset Ġis Ġa Ġbalanced Ġone Ġalthough Ġit Ġis Ġcurrently Ġlimited Ġin Ġsize . ĠThe Ġcorresponding Ġstance Ġannotations Ġare Ġmade Ġpublicly Ġavailable Ġat Ġhttp :// c eng . met u . edu . tr / ĠIN LINE FORM 0 Ġe 12 03 29 / ĠTurkish _ St ance _ Det ection _ Tweet _ Dat as et . csv Ġin ĠComm a ĠSep ar ated ĠValues Ġ( CS V ) Ġformat . ĠThe Ġfile Ġcontains Ġthree Ġcolumns Ġwith Ġthe Ġcorresponding Ġheaders . ĠThe Ġfirst Ġcolumn Ġis Ġthe Ġtweet Ġid Ġof Ġthe Ġcorresponding Ġtweet , Ġthe Ġsecond Ġcolumn Ġcontains Ġthe Ġname Ġof Ġthe Ġstance Ġtarget , Ġand Ġthe Ġlast Ġcolumn Ġincludes Ġthe Ġstance Ġof Ġthe Ġtweet Ġfor Ġthe Ġtarget Ġas ĠFavor Ġor ĠAgainst . </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġpublicly - available Ġstance - annot ated Ġdata Ġset Ġfor ĠTurkish . ĠHence , Ġit Ġis Ġa Ġsignificant Ġresource Ġas Ġthere Ġis Ġa Ġscarcity Ġof Ġannot ated Ġdata Ġsets , Ġlinguistic Ġresources , Ġand ĠN LP Ġtools Ġavailable Ġfor ĠTurkish . ĠAdditionally , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġit Ġis Ġalso Ġsignificant Ġfor Ġbeing Ġthe Ġfirst Ġstance - annot ated Ġdata Ġset Ġincluding Ġsports - related Ġtweets , Ġas Ġprevious Ġstance Ġdetection Ġdata Ġsets Ġmostly Ġinclude Ġon - line Ġtexts Ġon Ġpolitical / ethical Ġissues . </s> St ance ĠDetection ĠExper iments ĠUsing ĠS VM ĠClass ifiers </s> It Ġis Ġemphasized Ġin Ġthe Ġrelated Ġliterature Ġthat Ġun ig ram - based Ġmethods Ġare Ġreliable Ġfor Ġthe Ġstance Ġdetection Ġtask ĠB IB REF 2 Ġand Ġsimilarly Ġun ig ram - based Ġmodels Ġhave Ġbeen Ġused Ġas Ġbaseline Ġmodels Ġin Ġstudies Ġsuch Ġas ĠB IB REF 0 Ġ. ĠIn Ġorder Ġto Ġbe Ġused Ġas Ġa Ġbaseline Ġand Ġreference Ġsystem Ġfor Ġfurther Ġstudies Ġon Ġstance Ġdetection Ġin ĠTurkish Ġtweets , Ġwe Ġhave Ġtrained Ġtwo ĠS VM Ġclass ifiers Ġ( one Ġfor Ġeach Ġtarget ) Ġusing Ġun ig rams Ġas Ġfeatures . ĠBefore Ġthe Ġextraction Ġof Ġun ig rams , Ġwe Ġhave Ġemployed Ġautomated Ġpre processing Ġto Ġfilter Ġout Ġthe Ġstop words Ġin Ġour Ġannot ated Ġdata Ġset Ġof Ġ700 Ġtweets . ĠThe Ġstop word Ġlist Ġused Ġis Ġthe Ġlist Ġpresented Ġin ĠB IB REF 12 Ġwhich , Ġin Ġturn , Ġis Ġthe Ġslightly Ġextended Ġversion Ġof Ġthe Ġstop word Ġlist Ġprovided Ġin ĠB IB REF 13 Ġ. </s> We Ġhave Ġused Ġthe ĠS VM Ġimplementation Ġavailable Ġin Ġthe ĠWe ka Ġdata Ġmining Ġapplication ĠB IB REF 14 Ġwhere Ġthis Ġparticular Ġimplementation Ġemploys Ġthe ĠSM O Ġalgorithm ĠB IB REF 15 Ġto Ġtrain Ġa Ġclass ifier Ġwith Ġa Ġlinear Ġkernel . ĠThe Ġ10 - fold Ġcross - valid ation Ġresults Ġof Ġthe Ġtwo Ġclass ifiers Ġare Ġprovided Ġin ĠTable ĠT AB REF 1 Ġusing Ġthe Ġmetrics Ġof Ġprecision , Ġrecall , Ġand ĠF - Measure . </s> The Ġevaluation Ġresults Ġare Ġquite Ġfavorable Ġfor Ġboth Ġtargets Ġand Ġparticularly Ġhigher Ġfor ĠTarget - 1 , Ġconsidering Ġthe Ġfact Ġthat Ġthey Ġare Ġthe Ġinitial Ġexperiments Ġon Ġthe Ġdata Ġset . ĠThe Ġperformance Ġof Ġthe Ġclass ifiers Ġis Ġbetter Ġfor Ġthe ĠFavor Ġclass Ġfor Ġboth Ġtargets Ġwhen Ġcompared Ġwith Ġthe Ġperformance Ġresults Ġfor Ġthe ĠAgainst Ġclass . ĠThis Ġoutcome Ġmay Ġbe Ġdue Ġto Ġthe Ġcommon Ġuse Ġof Ġsome Ġterms Ġwhen Ġexpressing Ġpositive Ġstance Ġtowards Ġsports Ġclubs Ġin ĠTurkish Ġtweets . ĠThe Ġsame Ġpercentage Ġof Ġcommon Ġterms Ġmay Ġnot Ġhave Ġbeen Ġobserved Ġin Ġtweets Ġduring Ġthe Ġexpression Ġof Ġnegative Ġstances Ġtowards Ġthe Ġtargets . ĠYet , Ġcompletely Ġthe Ġopposite Ġpattern Ġis Ġobserved Ġin Ġstance Ġdetection Ġresults Ġof Ġbaseline Ġsystems Ġgiven Ġin ĠB IB REF 0 Ġ, Ġi . e ., Ġbetter ĠF - Measure Ġrates Ġhave Ġbeen Ġobtained Ġfor Ġthe ĠAgainst Ġclass Ġwhen Ġcompared Ġwith Ġthe ĠFavor Ġclass ĠB IB REF 0 Ġ. ĠSome Ġof Ġthe Ġbaseline Ġsystems Ġreported Ġin ĠB IB REF 0 Ġare ĠS VM - based Ġsystems Ġusing Ġun ig rams Ġand Ġn gram s Ġas Ġfeatures Ġsimilar Ġto Ġour Ġstudy , Ġbut Ġtheir Ġdata Ġsets Ġinclude Ġall Ġthree Ġstance Ġclasses Ġof ĠFavor , ĠAgainst , Ġand ĠNeither , Ġwhile Ġour Ġdata Ġset Ġcomprises Ġonly Ġtweets Ġclassified Ġas Ġbelonging Ġto ĠFavor Ġor ĠAgainst Ġclasses . ĠAnother Ġdifference Ġis Ġthat Ġthe Ġdata Ġsets Ġin ĠB IB REF 0 Ġhave Ġbeen Ġdivided Ġinto Ġtraining Ġand Ġtest Ġsets , Ġwhile Ġin Ġour Ġstudy Ġwe Ġprovide Ġ10 - fold Ġcross - valid ation Ġresults Ġon Ġthe Ġwhole Ġdata Ġset . ĠOn Ġthe Ġother Ġhand , Ġwe Ġshould Ġalso Ġnote Ġthat ĠS VM - based Ġsentiment Ġanalysis Ġsystems Ġ( such Ġas Ġthose Ġgiven Ġin ĠB IB REF 16 Ġ) Ġhave Ġbeen Ġreported Ġto Ġachieve Ġbetter ĠF - Measure Ġrates Ġfor Ġthe ĠPositive Ġsentiment Ġclass Ġwhen Ġcompared Ġwith Ġthe Ġresults Ġobtained Ġfor Ġthe ĠNegative Ġclass . ĠTherefore , Ġour Ġevaluation Ġresults Ġfor Ġeach Ġstance Ġclass Ġseem Ġto Ġbe Ġin Ġline Ġwith Ġsuch Ġsentiment Ġanalysis Ġsystems . ĠYet , Ġfurther Ġexperiments Ġon Ġthe Ġextended Ġversions Ġof Ġour Ġdata Ġset Ġshould Ġbe Ġconducted Ġand Ġthe Ġresults Ġshould Ġagain Ġbe Ġcompared Ġwith Ġthe Ġstance Ġdetection Ġresults Ġgiven Ġin Ġthe Ġliterature . </s> We Ġhave Ġalso Ġevaluated ĠS VM Ġclass ifiers Ġwhich Ġuse Ġonly Ġbig rams Ġas Ġfeatures , Ġas Ġn gram - based Ġclass ifiers Ġhave Ġbeen Ġreported Ġto Ġperform Ġbetter Ġfor Ġthe Ġstance Ġdetection Ġproblem ĠB IB REF 0 Ġ. ĠHowever , Ġwe Ġhave Ġobserved Ġthat Ġusing Ġbig rams Ġas Ġthe Ġsole Ġfeatures Ġof Ġthe ĠS VM Ġclass ifiers Ġleads Ġto Ġquite Ġpoor Ġresults . ĠThis Ġobservation Ġmay Ġbe Ġdue Ġto Ġthe Ġrelatively Ġlimited Ġsize Ġof Ġthe Ġtweet Ġdata Ġset Ġemployed . ĠStill , Ġwe Ġcan Ġconclude Ġthat Ġun ig ram - based Ġfeatures Ġlead Ġto Ġsuperior Ġresults Ġcompared Ġto Ġthe Ġresults Ġobtained Ġusing Ġbig rams Ġas Ġfeatures , Ġbased Ġon Ġour Ġexperiments Ġon Ġour Ġdata Ġset . ĠYet , Ġn gram - based Ġfeatures Ġmay Ġbe Ġemployed Ġon Ġthe Ġextended Ġversions Ġof Ġthe Ġdata Ġset Ġto Ġverify Ġthis Ġconclusion Ġwithin Ġthe Ġcourse Ġof Ġfuture Ġwork . </s> With Ġan Ġintention Ġto Ġexploit Ġthe Ġcontribution Ġof Ġhashtag Ġuse Ġto Ġstance Ġdetection , Ġwe Ġhave Ġalso Ġused Ġthe Ġexistence Ġof Ġhasht ags Ġin Ġtweets Ġas Ġan Ġadditional Ġfeature Ġto Ġun ig rams . ĠThe Ġcorresponding Ġevaluation Ġresults Ġof Ġthe ĠS VM Ġclass ifiers Ġusing Ġun ig rams Ġtogether Ġthe Ġexistence Ġof Ġhasht ags Ġas Ġfeatures Ġare Ġprovided Ġin ĠTable ĠT AB REF 2 Ġ. </s> When Ġthe Ġresults Ġgiven Ġin ĠTable ĠT AB REF 2 Ġare Ġcompared Ġwith Ġthe Ġresults Ġin ĠTable ĠT AB REF 1 Ġ, Ġa Ġslight Ġdecrease Ġin ĠF - Measure Ġ( 0 . 5 %) Ġfor ĠTarget - 1 Ġis Ġobserved , Ġwhile Ġthe Ġoverall ĠF - Measure Ġvalue Ġfor ĠTarget - 2 Ġhas Ġincreased Ġby Ġ1 . 8 %. ĠAlthough Ġwe Ġcould Ġnot Ġderive Ġsound Ġconclusions Ġmainly Ġdue Ġto Ġthe Ġrelatively Ġsmall Ġsize Ġof Ġour Ġdata Ġset , Ġthe Ġincrease Ġin Ġthe Ġperformance Ġof Ġthe ĠS VM Ġclass ifier ĠTarget - 2 Ġis Ġan Ġencouraging Ġevidence Ġfor Ġthe Ġexploitation Ġof Ġhasht ags Ġin Ġa Ġstance Ġdetection Ġsystem . ĠWe Ġleave Ġother Ġways Ġof Ġexploiting Ġhasht ags Ġfor Ġstance Ġdetection Ġas Ġa Ġfuture Ġwork . </s> To Ġsum Ġup , Ġour Ġevaluation Ġresults Ġare Ġsignificant Ġas Ġreference Ġresults Ġto Ġbe Ġused Ġfor Ġcomparison Ġpurposes Ġand Ġprovides Ġevidence Ġfor Ġthe Ġutility Ġof Ġun ig ram - based Ġand Ġhashtag - related Ġfeatures Ġin ĠS VM Ġclass ifiers Ġfor Ġthe Ġstance Ġdetection Ġproblem Ġin ĠTurkish Ġtweets . </s> Future ĠProspect s </s> Future Ġwork Ġbased Ġon Ġthe Ġcurrent Ġstudy Ġincludes Ġthe Ġfollowing : </s> Conclusion </s> St ance Ġdetection Ġis Ġa Ġconsiderably Ġnew Ġresearch Ġarea Ġin Ġnatural Ġlanguage Ġprocessing Ġand Ġis Ġconsidered Ġwithin Ġthe Ġscope Ġof Ġthe Ġwell - stud ied Ġtopic Ġof Ġsentiment Ġanalysis . ĠIt Ġis Ġthe Ġdetection Ġof Ġstance Ġwithin Ġtext Ġtowards Ġa Ġtarget Ġwhich Ġmay Ġbe Ġexplicitly Ġspecified Ġin Ġthe Ġtext Ġor Ġnot . ĠIn Ġthis Ġstudy , Ġwe Ġpresent Ġa Ġstance - annot ated Ġtweet Ġdata Ġset Ġin ĠTurkish Ġwhere Ġthe Ġtargets Ġof Ġthe Ġannot ated Ġstances Ġare Ġtwo Ġpopular Ġsports Ġclubs Ġin ĠTurkey . ĠThe Ġcorresponding Ġannotations Ġare Ġmade Ġpublicly - available Ġfor Ġresearch Ġpurposes . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġstance Ġdetection Ġdata Ġset Ġfor Ġthe ĠTurkish Ġlanguage Ġand Ġalso Ġthe Ġfirst Ġsports - related Ġstance - annot ated Ġdata Ġset . ĠAlso Ġpresented Ġin Ġthis Ġstudy Ġare ĠS VM Ġclass ifiers Ġ( one Ġfor Ġeach Ġtarget ) Ġutilizing Ġun ig ram Ġand Ġbig ram Ġfeatures Ġin Ġaddition Ġto Ġusing Ġthe Ġexistence Ġof Ġhasht ags Ġas Ġanother Ġfeature . Ġ10 - fold Ġcross Ġvalidation Ġresults Ġof Ġthese Ġclass ifiers Ġare Ġpresented Ġwhich Ġcan Ġbe Ġused Ġas Ġreference Ġresults Ġby Ġprospective Ġsystems . ĠBoth Ġthe Ġannot ated Ġdata Ġset Ġand Ġthe Ġclass ifiers Ġwith Ġevaluations Ġare Ġsignificant Ġsince Ġthey Ġare Ġthe Ġinitial Ġcontributions Ġto Ġstance Ġdetection Ġproblem Ġin ĠTurkish Ġtweets .