Gaze Sarcasm using Multi Instance Logistic Regression.
ABSTRACTIVE
the MILR classifier
EXTRACTIVE
FLOAT SELECTED: Table 3: Classification results for different feature combinations. P→ Precision, R→Recall, F→ F˙score, Kappa→ Kappa statistics show agreement with the gold labels. Subscripts 1 and -1 correspond to sarcasm and non-sarcasm classes respectively.
For all regular classifiers, the gaze features are averaged across participants and augmented with linguistic and sarcasm related features. For the MILR classifier, the gaze features derived from each participant are augmented with linguistic features and thus, a multi instance “bag” of features is formed for each sentence in the training data. This multi-instance dataset is given to an MILR classifier, which follows the standard multi instance assumption to derive class-labels for each bag. For all the classifiers, our feature combination outperforms the baselines (considering only unigram features) as well as BIBREF3 , with the MILR classifier getting an F-score improvement of 3.7% and Kappa difference of 0.08. We also achieve an improvement of 2% over the baseline, using SVM classifier, when we employ our feature set. We also observe that the gaze features alone, also capture the differences between sarcasm and non-sarcasm classes with a high-precision but a low recall.
1701.05574
Introduction </s> S arc asm Ġis Ġan Ġintensive , Ġindirect Ġand Ġcomplex Ġconstruct Ġthat Ġis Ġoften Ġintended Ġto Ġexpress Ġcontempt Ġor Ġridicule Ġ. ĠS arc asm , Ġin Ġspeech , Ġis Ġmulti - mod al , Ġinvolving Ġtone , Ġbody - language Ġand Ġgestures Ġalong Ġwith Ġlinguistic Ġartifacts Ġused Ġin Ġspeech . ĠS arc asm Ġin Ġtext , Ġon Ġthe Ġother Ġhand , Ġis Ġmore Ġrestrictive Ġwhen Ġit Ġcomes Ġto Ġsuch Ġnon - ling u istic Ġmod alities . ĠThis Ġmakes Ġrecognizing Ġtextual Ġsarc asm Ġmore Ġchallenging Ġfor Ġboth Ġhumans Ġand Ġmachines . </s> S arc asm Ġdetection Ġplays Ġan Ġindispensable Ġrole Ġin Ġapplications Ġlike Ġonline Ġreview Ġsummar izers , Ġdialog Ġsystems , Ġrecommendation Ġsystems Ġand Ġsentiment Ġanaly zers . ĠThis Ġmakes Ġautomatic Ġdetection Ġof Ġsarc asm Ġan Ġimportant Ġproblem . ĠHowever , Ġit Ġhas Ġbeen Ġquite Ġdifficult Ġto Ġsolve Ġsuch Ġa Ġproblem Ġwith Ġtraditional ĠN LP Ġtools Ġand Ġtechniques . ĠThis Ġis Ġapparent Ġfrom Ġthe Ġresults Ġreported Ġby Ġthe Ġsurvey Ġfrom ĠD BL P : j ournals / cor r / J oshi BC 16 . ĠThe Ġfollowing Ġdiscussion Ġbrings Ġmore Ġinsights Ġinto Ġthis . </s> Consider Ġa Ġscenario Ġwhere Ġan Ġonline Ġreviewer Ġgives Ġa Ġnegative Ġopinion Ġabout Ġa Ġmovie Ġthrough Ġsarc asm : ĠâĢ ľ This Ġis Ġthe Ġkind Ġof Ġmovie Ġyou Ġsee Ġbecause Ġthe Ġtheater Ġhas Ġair Ġconditioning âĢ Ŀ . ĠIt Ġis Ġdifficult Ġfor Ġan Ġautomatic Ġsentiment Ġanaly zer Ġto Ġassign Ġa Ġrating Ġto Ġthe Ġmovie Ġand , Ġin Ġthe Ġabsence Ġof Ġany Ġother Ġinformation , Ġsuch Ġa Ġsystem Ġmay Ġnot Ġbe Ġable Ġto Ġcomprehend Ġthat Ġpriorit izing Ġthe Ġair - condition ing Ġfacilities Ġof Ġthe Ġtheater Ġover Ġthe Ġmovie Ġexperience Ġindicates Ġa Ġnegative Ġsentiment Ġtowards Ġthe Ġmovie . ĠThis Ġgives Ġan Ġintuition Ġto Ġwhy , Ġfor Ġsarc asm Ġdetection , Ġit Ġis Ġnecessary Ġto Ġgo Ġbeyond Ġtextual Ġanalysis . </s> We Ġaim Ġto Ġaddress Ġthis Ġproblem Ġby Ġexploiting Ġthe Ġpsych ol ingu istic Ġside Ġof Ġsarc asm Ġdetection , Ġusing Ġcognitive Ġfeatures Ġextracted Ġwith Ġthe Ġhelp Ġof Ġeye - tracking . ĠA Ġmotivation Ġto Ġconsider Ġcognitive Ġfeatures Ġcomes Ġfrom Ġanalyzing Ġhuman Ġeye - move ment Ġtraject ories Ġthat Ġsupports Ġthe Ġconjecture : ĠReading Ġsarcastic Ġtexts Ġinduces Ġdistinctive Ġeye Ġmovement Ġpatterns , Ġcompared Ġto Ġliteral Ġtexts . ĠThe Ġcognitive Ġfeatures , Ġderived Ġfrom Ġhuman Ġeye Ġmovement Ġpatterns Ġobserved Ġduring Ġreading , Ġinclude Ġtwo Ġprimary Ġfeature Ġtypes : </s> The Ġcognitive Ġfeatures , Ġalong Ġwith Ġtextual Ġfeatures Ġused Ġin Ġbest Ġavailable Ġsarc asm Ġdetectors , Ġare Ġused Ġto Ġtrain Ġbinary Ġclass ifiers Ġagainst Ġgiven Ġsarc asm Ġlabels . ĠOur Ġexperiments Ġshow Ġsignificant Ġimprovement Ġin Ġclassification Ġaccuracy Ġover Ġthe Ġstate Ġof Ġthe Ġart , Ġby Ġperforming Ġsuch Ġaug mentation . </s> Related ĠWork </s> S arc asm , Ġin Ġgeneral , Ġhas Ġbeen Ġthe Ġfocus Ġof Ġresearch Ġfor Ġquite Ġsome Ġtime . ĠIn Ġone Ġof Ġthe Ġpioneering Ġworks Ġj org ensen 1984 test Ġexplained Ġhow Ġsarc asm Ġarises Ġwhen Ġa Ġfigur ative Ġmeaning Ġis Ġused Ġopposite Ġto Ġthe Ġliteral Ġmeaning Ġof Ġthe Ġutter ance . ĠIn Ġthe Ġword Ġof Ġcl ark 1984 pret ense , Ġsarc asm Ġprocessing Ġinvolves Ġcance ling Ġthe Ġindirectly Ġneg ated Ġmessage Ġand Ġreplacing Ġit Ġwith Ġthe Ġimplicated Ġone . Ġg ior a 1995 ir ony , Ġon Ġthe Ġother Ġhand , Ġdefine Ġsarc asm Ġas Ġa Ġmode Ġof Ġindirect Ġneg ation Ġthat Ġrequires Ġprocessing Ġof Ġboth Ġneg ated Ġand Ġimplicated Ġmessages . Ġiv ank o 2003 context Ġdefine Ġsarc asm Ġas Ġa Ġsix Ġtuple Ġentity Ġconsisting Ġof Ġa Ġspeaker , Ġa Ġlistener , ĠContext , ĠUt ter ance , ĠLit eral ĠProposition Ġand ĠInt ended ĠProposition Ġand Ġstudy Ġthe Ġcognitive Ġaspects Ġof Ġsarc asm Ġprocessing . </s> Com put ational Ġlingu ists Ġhave Ġpreviously Ġaddressed Ġthis Ġproblem Ġusing Ġrule Ġbased Ġand Ġstatistical Ġtechniques , Ġthat Ġmake Ġuse Ġof Ġ: Ġ( a ) ĠUn ig rams Ġand ĠPr ag matic Ġfeatures ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ( b ) ĠSt yl istic Ġpatterns ĠB IB REF 4 Ġand Ġpatterns Ġrelated Ġto Ġsituational Ġdisparity ĠB IB REF 5 Ġand Ġ( c ) ĠHast ag Ġinterpretations ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. </s> Most Ġof Ġthe Ġpreviously Ġdone Ġwork Ġon Ġsarc asm Ġdetection Ġuses Ġdistant Ġsupervision Ġbased Ġtechniques Ġ( ex : Ġleveraging Ġhasht ags ) Ġand Ġstyl istic / pr ag matic Ġfeatures Ġ( em otic ons , Ġlaughter Ġexpressions Ġsuch Ġas ĠâĢ ľ lol âĢ Ŀ Ġetc ). ĠBut , Ġdetecting Ġsarc asm Ġin Ġlingu istically Ġwell - formed Ġstructures , Ġin Ġabsence Ġof Ġexplicit Ġcues Ġor Ġinformation Ġ( like Ġemot icons ), Ġproves Ġto Ġbe Ġhard Ġusing Ġsuch Ġlinguistic / st yl istic Ġfeatures Ġalone . </s> With Ġthe Ġadvent Ġof Ġsophisticated Ġeye - track ers Ġand Ġelectro / mag net o - ence phal ographic Ġ( EE G / M EG ) Ġdevices , Ġit Ġhas Ġbeen Ġpossible Ġto Ġdelve Ġdeep Ġinto Ġthe Ġcognitive Ġunder p inn ings Ġof Ġsarc asm Ġunderstanding . ĠFil ik 2014 , Ġusing Ġa Ġseries Ġof Ġeye - tracking Ġand ĠEEG Ġexperiments Ġtry Ġto Ġshow Ġthat Ġfor Ġunfamiliar Ġiron ies , Ġthe Ġliteral Ġinterpretation Ġwould Ġbe Ġcomputed Ġfirst . ĠThey Ġalso Ġshow Ġthat Ġa Ġmismatch Ġwith Ġcontext Ġwould Ġlead Ġto Ġa Ġre - interpret ation Ġof Ġthe Ġstatement , Ġas Ġbeing Ġironic . ĠCam bl in 2007 103 Ġshow Ġthat Ġin Ġmulti - sent ence Ġpassages , Ġdiscourse Ġcon gru ence Ġhas Ġrobust Ġeffects Ġon Ġeye Ġmovements . ĠThis Ġalso Ġimplies Ġthat Ġdisrupted Ġprocessing Ġoccurs Ġfor Ġdiscourse Ġinc ong ru ent Ġwords , Ġeven Ġthough Ġthey Ġare Ġperfectly Ġcon gru ous Ġat Ġthe Ġsentence Ġlevel . ĠIn Ġour Ġprevious Ġwork ĠB IB REF 8 Ġ, Ġwe Ġaugment Ġcognitive Ġfeatures , Ġderived Ġfrom Ġeye - move ment Ġpatterns Ġof Ġreaders , Ġwith Ġtextual Ġfeatures Ġto Ġdetect Ġwhether Ġa Ġhuman Ġreader Ġhas Ġrealized Ġthe Ġpresence Ġof Ġsarc asm Ġin Ġtext Ġor Ġnot . </s> The Ġrecent Ġadvancements Ġin Ġthe Ġliterature Ġdiscussed Ġabove , Ġmotivate Ġus Ġto Ġexplore Ġgaze - based Ġcognition Ġfor Ġsarc asm Ġdetection . ĠAs Ġfar Ġas Ġwe Ġknow , Ġour Ġwork Ġis Ġthe Ġfirst Ġof Ġits Ġkind . </s> Eye - tracking ĠDatabase Ġfor ĠS arc asm ĠAnalysis </s> S arc asm Ġoften Ġeman ates Ġfrom Ġinc ong ru ity ĠB IB REF 9 Ġ, Ġwhich Ġen forces Ġthe Ġbrain Ġto Ġre analy ze Ġit ĠB IB REF 10 Ġ. ĠThis , Ġin Ġturn , Ġaffects Ġthe Ġway Ġeyes Ġmove Ġthrough Ġthe Ġtext . ĠHence , Ġdistinctive Ġeye - move ment Ġpatterns Ġmay Ġbe Ġobserved Ġin Ġthe Ġcase Ġof Ġsuccessful Ġprocessing Ġof Ġsarc asm Ġin Ġtext Ġin Ġcontrast Ġto Ġliteral Ġtexts . ĠThis Ġhypothesis Ġforms Ġthe Ġcru x Ġof Ġour Ġmethod Ġfor Ġsarc asm Ġdetection Ġand Ġwe Ġvalidate Ġthis Ġusing Ġour Ġpreviously Ġreleased Ġfreely Ġavailable Ġsarc asm Ġdataset ĠB IB REF 8 Ġenriched Ġwith Ġgaze Ġinformation . </s> Document ĠDescription </s> The Ġdatabase Ġconsists Ġof Ġ1 , 000 Ġshort Ġtexts , Ġeach Ġhaving Ġ10 - 40 Ġwords . ĠOut Ġof Ġthese , Ġ350 Ġare Ġsarcastic Ġand Ġare Ġcollected Ġas Ġfollows : Ġ( a ) Ġ103 Ġsentences Ġare Ġfrom Ġtwo Ġpopular Ġsarcastic Ġquote Ġwebsites , Ġ( b ) Ġ76 Ġsarcastic Ġshort Ġmovie Ġreviews Ġare Ġmanually Ġextracted Ġfrom Ġthe ĠAmazon ĠMovie ĠCorpus ĠB IB REF 11 Ġby Ġtwo Ġlingu ists . Ġ( c ) Ġ171 Ġtweets Ġare Ġdownloaded Ġusing Ġthe Ġhashtag Ġ# s arc asm Ġfrom ĠTwitter . ĠThe Ġ650 Ġnon - s ar cast ic Ġtexts Ġare Ġeither Ġdownloaded Ġfrom ĠTwitter Ġor Ġextracted Ġfrom Ġthe ĠAmazon ĠMovie ĠReview Ġcorpus . ĠThe Ġsentences Ġdo Ġnot Ġcontain Ġwords / ph r ases Ġthat Ġare Ġhighly Ġtopic Ġor Ġculture Ġspecific . ĠThe Ġtweets Ġwere Ġnormalized Ġto Ġmake Ġthem Ġlingu istically Ġwell Ġformed Ġto Ġavoid Ġdifficulty Ġin Ġinterpreting Ġsocial Ġmedia Ġling o . ĠEvery Ġsentence Ġin Ġour Ġdataset Ġcarries Ġpositive Ġor Ġnegative Ġopinion Ġabout Ġspecific ĠâĢ ľ as pects âĢ Ŀ . ĠFor Ġexample , Ġthe Ġsentence ĠâĢ ľ The Ġmovie Ġis Ġextremely Ġwell Ġcast âĢ Ŀ Ġhas Ġpositive Ġsentiment Ġabout Ġthe Ġaspect ĠâĢ ľ cast âĢ Ŀ . </s> The Ġannot ators Ġwere Ġseven Ġgraduate Ġstudents Ġwith Ġscience Ġand Ġengineering Ġbackground , Ġand Ġpossess Ġgood ĠEnglish Ġproficiency . ĠThey Ġwere Ġgiven Ġa Ġset Ġof Ġinstructions Ġbeforehand Ġand Ġare Ġadvised Ġto Ġseek Ġclar ifications Ġbefore Ġthey Ġproceed . ĠThe Ġinstructions Ġmention Ġthe Ġnature Ġof Ġthe Ġtask , Ġannotation Ġinput Ġmethod , Ġand Ġnecessity Ġof Ġhead Ġmovement Ġminim ization Ġduring Ġthe Ġexperiment . </s> Task ĠDescription </s> The Ġtask Ġassigned Ġto Ġannot ators Ġwas Ġto Ġread Ġsentences Ġone Ġat Ġa Ġtime Ġand Ġlabel Ġthem Ġwith Ġwith Ġbinary Ġlabels Ġindicating Ġthe Ġpol arity Ġ( i . e ., Ġpositive / negative ). ĠNote Ġthat , Ġthe Ġparticipants Ġwere Ġnot Ġinstructed Ġto Ġannot ate Ġwhether Ġa Ġsentence Ġis Ġsarcastic Ġor Ġnot ., Ġto Ġrule Ġout Ġthe ĠPrim ing ĠEffect Ġ( i . e ., Ġif Ġsarc asm Ġis Ġexpected Ġbeforehand , Ġprocessing Ġinc ong ru ity Ġbecomes Ġrelatively Ġeasier ĠB IB REF 12 Ġ). ĠThe Ġsetup Ġensures Ġits ĠâĢ ľ ec ological Ġvalidity âĢ Ŀ Ġin Ġtwo Ġways : Ġ( 1 ) ĠReaders Ġare Ġnot Ġgiven Ġany Ġclue Ġthat Ġthey Ġhave Ġto Ġtreat Ġsarc asm Ġwith Ġspecial Ġattention . ĠThis Ġis Ġdone Ġby Ġsetting Ġthe Ġtask Ġto Ġpol arity Ġannotation Ġ( instead Ġof Ġsarc asm Ġdetection ). Ġ( 2 ) ĠSar cast ic Ġsentences Ġare Ġmixed Ġwith Ġnon Ġsarcastic Ġtext , Ġwhich Ġdoes Ġnot Ġgive Ġprior Ġknowledge Ġabout Ġwhether Ġthe Ġforthcoming Ġtext Ġwill Ġbe Ġsarcastic Ġor Ġnot . </s> The Ġeye - tracking Ġexperiment Ġis Ġconducted Ġby Ġfollowing Ġthe Ġstandard Ġnorms Ġin Ġeye - move ment Ġresearch ĠB IB REF 13 Ġ. ĠAt Ġa Ġtime , Ġone Ġsentence Ġis Ġdisplayed Ġto Ġthe Ġreader Ġalong Ġwith Ġthe ĠâĢ ľ as pect âĢ Ŀ Ġwith Ġrespect Ġto Ġwhich Ġthe Ġannotation Ġhas Ġto Ġbe Ġprovided . ĠWhile Ġreading , Ġan ĠSR - Research ĠEy el ink - 1000 Ġeye - tr acker Ġ( mon ocular Ġremote Ġmode , Ġsampling Ġrate Ġ500 Hz ) Ġrecords Ġseveral Ġeye - move ment Ġparameters Ġlike Ġfix ations Ġ( a Ġlong Ġstay Ġof Ġgaze ) Ġand Ġs acc ade Ġ( quick Ġjumping Ġof Ġgaze Ġbetween Ġtwo Ġpositions Ġof Ġrest ) Ġand Ġpupil Ġsize . </s> The Ġaccuracy Ġof Ġpol arity Ġannotation Ġvaries Ġbetween Ġ72 %- 91 % Ġfor Ġsarcastic Ġtexts Ġand Ġ75 %- 91 % Ġfor Ġnon - s ar cast ic Ġtext , Ġshowing Ġthe Ġinherent Ġdifficulty Ġof Ġsentiment Ġannotation , Ġwhen Ġsarc asm Ġis Ġpresent Ġin Ġthe Ġtext Ġunder Ġconsideration . ĠAn notation Ġerrors Ġmay Ġbe Ġattributed Ġto : Ġ( a ) Ġlack Ġof Ġpatience / att ention Ġwhile Ġreading , Ġ( b ) Ġissues Ġrelated Ġto Ġtext Ġcomprehension , Ġand Ġ( c ) Ġconfusion / ind ec is iveness Ġcaused Ġdue Ġto Ġlack Ġof Ġcontext . </s> For Ġour Ġanalysis , Ġwe Ġdo Ġnot Ġdiscard Ġthe Ġincorrect Ġannotations Ġpresent Ġin Ġthe Ġdatabase . ĠSince Ġour Ġsystem Ġeventually Ġaims Ġto Ġinvolve Ġonline Ġreaders Ġfor Ġsarc asm Ġdetection , Ġit Ġwill Ġbe Ġhard Ġto Ġsegreg ate Ġreaders Ġwho Ġmisinterpret Ġthe Ġtext . ĠWe Ġmake Ġa Ġrational Ġassumption Ġthat , Ġfor Ġa Ġparticular Ġtext , Ġmost Ġof Ġthe Ġreaders , Ġfrom Ġa Ġfairly Ġlarge Ġpopulation , Ġwill Ġbe Ġable Ġto Ġidentify Ġsarc asm . ĠUnder Ġthis Ġassumption , Ġthe Ġeye - move ment Ġparameters , Ġaveraged Ġacross Ġall Ġreaders Ġin Ġour Ġsetting , Ġmay Ġnot Ġbe Ġsignificantly Ġdistorted Ġby Ġa Ġfew Ġreaders Ġwho Ġwould Ġhave Ġfailed Ġto Ġidentify Ġsarc asm . ĠThis Ġassumption Ġis Ġapplicable Ġfor Ġboth Ġregular Ġand Ġmulti - instance Ġbased Ġclass ifiers Ġexplained Ġin Ġsection ĠSEC REF 6 Ġ. </s> Analysis Ġof ĠEye - move ment ĠData </s> We Ġobserve Ġdistinct Ġbehavior Ġduring Ġsarc asm Ġreading , Ġby Ġanalyzing Ġthe ĠâĢ ľ fix ation Ġduration Ġon Ġthe Ġtext âĢ Ŀ Ġ( also Ġreferred Ġto Ġas ĠâĢ ľ d well Ġtime âĢ Ŀ Ġin Ġthe Ġliterature ) Ġand ĠâĢ ľ scan path s âĢ Ŀ Ġof Ġthe Ġreaders . </s> Vari ation Ġin Ġthe ĠAverage ĠFix ation ĠDuration Ġper ĠWord </s> Since Ġsarc asm Ġin Ġtext Ġcan Ġbe Ġexpected Ġto Ġinduce Ġcognitive Ġload , Ġit Ġis Ġreasonable Ġto Ġbelieve Ġthat Ġit Ġwould Ġrequire Ġmore Ġprocessing Ġtime ĠB IB REF 14 Ġ. ĠHence , Ġfixation Ġduration Ġnormalized Ġover Ġtotal Ġword Ġcount Ġshould Ġusually Ġbe Ġhigher Ġfor Ġa Ġsarcastic Ġtext Ġthan Ġfor Ġa Ġnon - s ar cast ic Ġone . ĠWe Ġobserve Ġthis Ġfor Ġall Ġparticipants Ġin Ġour Ġdataset , Ġwith Ġthe Ġaverage Ġfixation Ġduration Ġper Ġword Ġfor Ġsarcastic Ġtexts Ġbeing Ġat Ġleast Ġ1 . 5 Ġtimes Ġmore Ġthan Ġthat Ġof Ġnon - s ar cast ic Ġtexts . ĠTo Ġtest Ġthe Ġstatistical Ġsignificance , Ġwe Ġconduct Ġa Ġtwo - tailed Ġt - test Ġ( assuming Ġunequal Ġvariance ) Ġto Ġcompare Ġthe Ġaverage Ġfixation Ġduration Ġper Ġword Ġfor Ġsarcastic Ġand Ġnon - s ar cast ic Ġtexts . ĠThe Ġhypothesized Ġmean Ġdifference Ġis Ġset Ġto Ġ0 Ġand Ġthe Ġerror Ġtolerance Ġlimit Ġ( ĠIN LINE FORM 0 Ġ) Ġis Ġset Ġto Ġ0 . 05 . ĠThe Ġt - test Ġanalysis , Ġpresented Ġin ĠTable ĠT AB REF 11 Ġ, Ġshows Ġthat Ġfor Ġall Ġparticipants , Ġa Ġstatistically Ġsignificant Ġdifference Ġexists Ġbetween Ġthe Ġaverage Ġfixation Ġduration Ġper Ġword Ġfor Ġsarc asm Ġ( higher Ġaverage Ġfixation Ġduration ) Ġand Ġnon - s arc asm Ġ( lower Ġaverage Ġfixation Ġduration ). ĠThis Ġaff irms Ġthat Ġthe Ġpresence Ġof Ġsarc asm Ġaffects Ġthe Ġduration Ġof Ġfixation Ġon Ġwords . </s> It Ġis Ġimportant Ġto Ġnote Ġthat Ġlonger Ġfix ations Ġmay Ġalso Ġbe Ġcaused Ġby Ġother Ġlinguistic Ġsubt let ies Ġ( such Ġas Ġdifficult Ġwords , Ġambiguity Ġand Ġsynt act ically Ġcomplex Ġstructures ) Ġcausing Ġdelay Ġin Ġcomprehension , Ġor Ġocc ul om otor Ġcontrol Ġproblems Ġforcing Ġreaders Ġto Ġspend Ġtime Ġadjusting Ġeye - mus cles . ĠSo , Ġan Ġelevated Ġaverage Ġfixation Ġduration Ġper Ġword Ġmay Ġnot Ġsufficiently Ġindicate Ġthe Ġpresence Ġof Ġsarc asm . ĠBut Ġwe Ġwould Ġalso Ġlike Ġto Ġshare Ġthat , Ġfor Ġour Ġdataset , Ġwhen Ġwe Ġconsidered Ġread ability Ġ( F les ch Ġread ability Ġease - score ĠB IB REF 15 Ġ), Ġnumber Ġof Ġwords Ġin Ġa Ġsentence Ġand Ġaverage Ġcharacter Ġper Ġword Ġalong Ġwith Ġthe Ġsarc asm Ġlabel Ġas Ġthe Ġpredict ors Ġof Ġaverage Ġfixation Ġduration Ġfollowing Ġa Ġlinear Ġmixed Ġeffect Ġmodel ĠB IB REF 16 Ġ, Ġsarc asm Ġlabel Ġturned Ġout Ġto Ġbe Ġthe Ġmost Ġsignificant Ġpredictor Ġwith Ġa Ġmaximum Ġslope . ĠThis Ġindicates Ġthat Ġaverage Ġfixation Ġduration Ġper Ġword Ġhas Ġa Ġstrong Ġconnection Ġwith Ġthe Ġtext Ġbeing Ġsarcastic , Ġat Ġleast Ġin Ġour Ġdataset . </s> We Ġnow Ġanalyze Ġscan path s Ġto Ġgain Ġmore Ġinsights Ġinto Ġthe Ġsarc asm Ġcomprehension Ġprocess . </s> Analysis Ġof ĠScan path s </s> Scan path s Ġare Ġline - graph s Ġthat Ġcontain Ġfix ations Ġas Ġnodes Ġand Ġs acc ades Ġas Ġedges ; Ġthe Ġrad ii Ġof Ġthe Ġnodes Ġrepresent Ġthe Ġfixation Ġduration . ĠA Ġscan path Ġcorresponds Ġto Ġa Ġparticipant 's Ġeye - move ment Ġpattern Ġwhile Ġreading Ġa Ġparticular Ġsentence . ĠFigure ĠFIG REF 14 Ġpresents Ġscan path s Ġof Ġthree Ġparticipants Ġfor Ġthe Ġsarcastic Ġsentence ĠS 1 Ġand Ġthe Ġnon - s ar cast ic Ġsentence ĠS 2 . ĠThe Ġx - axis Ġof Ġthe Ġgraph Ġrepresents Ġthe Ġsequence Ġof Ġwords Ġa Ġreader Ġreads , Ġand Ġthe Ġy - axis Ġrepresents Ġa Ġtemporal Ġsequence Ġin Ġmilliseconds . </s> Consider Ġa Ġsarcastic Ġtext Ġcontaining Ġinc ong ru ous Ġphrases ĠA Ġand ĠB . ĠOur Ġqualitative Ġscan path - analysis Ġreveals Ġthat Ġscan path s Ġwith Ġrespect Ġto Ġsarc asm Ġprocessing Ġhave Ġtwo Ġtypical Ġcharacteristics . ĠOften , Ġa Ġlong Ġregression Ġ- Ġa Ġs acc ade Ġthat Ġgoes Ġto Ġa Ġpreviously Ġvisited Ġsegment Ġ- Ġis Ġobserved Ġwhen Ġa Ġreader Ġstarts Ġreading ĠB Ġafter Ġsk imming Ġthrough ĠA . ĠIn Ġa Ġfew Ġcases , Ġthe Ġfixation Ġduration Ġon ĠA Ġand ĠB Ġare Ġsignificantly Ġhigher Ġthan Ġthe Ġaverage Ġfixation Ġduration Ġper Ġword . ĠIn Ġsentence ĠS 1 , Ġwe Ġsee Ġlong Ġand Ġmultiple Ġregress ions Ġfrom Ġthe Ġtwo Ġinc ong ru ous Ġphrases ĠâĢ ľ mis con ception âĢ Ŀ Ġand ĠâĢ ľ cher ish âĢ Ŀ , Ġand Ġa Ġfew Ġinstances Ġwhere Ġphrases ĠâĢ ľ always Ġcherish âĢ Ŀ Ġand ĠâĢ ľ original Ġmisconception âĢ Ŀ Ġare Ġfix ated Ġlonger Ġthan Ġusual . ĠSuch Ġeye - move ment Ġbehaviors Ġare Ġnot Ġseen Ġfor ĠS 2 . </s> Though Ġsarc asm Ġinduces Ġdistinctive Ġscan path s Ġlike Ġthe Ġones Ġdepicted Ġin ĠFigure ĠFIG REF 14 Ġin Ġthe Ġobserved Ġexamples , Ġpresence Ġof Ġsuch Ġpatterns Ġis Ġnot Ġsufficient Ġto Ġguarantee Ġsarc asm ; Ġsuch Ġpatterns Ġmay Ġalso Ġpossibly Ġarise Ġfrom Ġliteral Ġtexts . ĠWe Ġbelieve Ġthat Ġa Ġcombination Ġof Ġlinguistic Ġfeatures , Ġread ability Ġof Ġtext Ġand Ġfeatures Ġderived Ġfrom Ġscan path s Ġwould Ġhelp Ġdiscrim in ative Ġmachine Ġlearning Ġmodels Ġlearn Ġsarc asm Ġbetter . </s> Features Ġfor ĠS arc asm ĠDetection </s> We Ġdescribe Ġthe Ġfeatures Ġused Ġfor Ġsarc asm Ġdetection Ġin ĠTable Ġ. ĠThe Ġfeatures Ġenlisted Ġunder Ġlex ical , impl icit Ġinc ong ru ity Ġand Ġexplicit Ġinc ong ru ity Ġare Ġborrowed Ġfrom Ġvarious Ġliterature Ġ( pred omin antly Ġfrom Ġj oshi 2015 har ness ing ). ĠThese Ġfeatures Ġare Ġessential Ġto Ġseparate Ġsarc asm Ġfrom Ġother Ġforms Ġsemantic Ġinc ong ru ity Ġin Ġtext Ġ( for Ġexample Ġambiguity Ġarising Ġfrom Ġsemantic Ġambiguity Ġor Ġfrom Ġmetaphors ). ĠTwo Ġadditional Ġtextual Ġfeatures Ġviz . Ġread ability Ġand Ġword Ġcount Ġof Ġthe Ġtext Ġare Ġalso Ġtaken Ġunder Ġconsideration . ĠThese Ġfeatures Ġare Ġused Ġto Ġreduce Ġthe Ġeffect Ġof Ġtext Ġhardness Ġand Ġtext Ġlength Ġon Ġthe Ġeye - move ment Ġpatterns . </s> Simple ĠG aze ĠBased ĠFeatures </s> Read ers ' Ġeye - move ment Ġbehavior , Ġcharacterized Ġby Ġfix ations , Ġforward Ġs acc ades , Ġsk ips Ġand Ġregress ions , Ġcan Ġbe Ġdirectly Ġquant ified Ġby Ġsimple Ġstatistical Ġaggregation Ġ( i . e ., Ġeither Ġcomputing Ġfeatures Ġfor Ġindividual Ġparticipants Ġand Ġthen Ġaveraging Ġor Ġperforming Ġa Ġmulti - instance Ġbased Ġlearning Ġas Ġexplained Ġin Ġsection ĠSEC REF 6 Ġ). ĠSince Ġthese Ġeye - move ment Ġattributes Ġrelate Ġto Ġthe Ġcognitive Ġprocess Ġin Ġreading ĠB IB REF 17 Ġ, Ġwe Ġconsider Ġthese Ġas Ġfeatures Ġin Ġour Ġmodel . ĠSome Ġof Ġthese Ġfeatures Ġhave Ġbeen Ġreported Ġby Ġsarc asm under stand ability Ġfor Ġmodeling Ġsarc asm Ġunderstand ability Ġof Ġreaders . ĠHowever , Ġas Ġfar Ġas Ġwe Ġknow , Ġthese Ġfeatures Ġare Ġbeing Ġintroduced Ġin ĠN LP Ġtasks Ġlike Ġtextual Ġsarc asm Ġdetection Ġfor Ġthe Ġfirst Ġtime . ĠThe Ġvalues Ġof Ġthese Ġfeatures Ġare Ġbelieved Ġto Ġincrease Ġwith Ġthe Ġincrease Ġin Ġthe Ġdegree Ġof Ġsurpr isal Ġcaused Ġby Ġinc ong ru ity Ġin Ġtext Ġ( except Ġskip Ġcount , Ġwhich Ġwill Ġdecrease ). </s> Com plex ĠG aze ĠBased ĠFeatures </s> For Ġthese Ġfeatures , Ġwe Ġrely Ġon Ġa Ġgraph Ġstructure , Ġnamely ĠâĢ ľ sal iency Ġgraphs ", Ġderived Ġfrom Ġeye - g aze Ġinformation Ġand Ġword Ġsequences Ġin Ġthe Ġtext . </s> For Ġeach Ġreader Ġand Ġeach Ġsentence , Ġwe Ġconstruct Ġa ĠâĢ ľ sal iency Ġgraph âĢ Ŀ , Ġrepresenting Ġthe Ġreader 's Ġattention Ġcharacteristics . ĠA Ġsal iency Ġgraph Ġfor Ġa Ġsentence ĠIN LINE FORM 0 Ġfor Ġa Ġreader ĠIN LINE FORM 1 Ġ, Ġrepresented Ġas ĠIN LINE FORM 2 Ġ, Ġis Ġa Ġgraph Ġwith Ġvert ices Ġ( ĠIN LINE FORM 3 Ġ) Ġand Ġedges Ġ( ĠIN LINE FORM 4 Ġ) Ġwhere Ġeach Ġvertex ĠIN LINE FORM 5 Ġcorresponds Ġto Ġa Ġword Ġin ĠIN LINE FORM 6 Ġ( may Ġnot Ġbe Ġunique ) Ġand Ġthere Ġexists Ġan Ġedge ĠIN LINE FORM 7 Ġbetween Ġvert ices ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġif ĠR Ġperforms Ġat Ġleast Ġone Ġs acc ade Ġbetween Ġthe Ġwords Ġcorresponding Ġto ĠIN LINE FORM 10 Ġand ĠIN LINE FORM 11 Ġ. </s> Figure ĠFIG REF 15 Ġshows Ġan Ġexample Ġof Ġa Ġsal iency Ġgraph . A Ġsal iency Ġgraph Ġmay Ġbe Ġweighted , Ġbut Ġnot Ġnecessarily Ġconnected , Ġfor Ġa Ġgiven Ġtext Ġ( as Ġthere Ġmay Ġbe Ġwords Ġin Ġthe Ġgiven Ġtext Ġwith Ġno Ġfixation Ġon Ġthem ). ĠThe ĠâĢ ľ complex âĢ Ŀ Ġgaze Ġfeatures Ġderived Ġfrom Ġsal iency Ġgraphs Ġare Ġalso Ġmotivated Ġby Ġthe Ġtheory Ġof Ġinc ong ru ity . ĠFor Ġinstance , ĠEdge ĠD ensity Ġof Ġa Ġsal iency Ġgraph Ġincreases Ġwith Ġthe Ġnumber Ġof Ġdistinct Ġs acc ades , Ġwhich Ġcould Ġarise Ġfrom Ġthe Ġcomplexity Ġcaused Ġby Ġpresence Ġof Ġsarc asm . ĠSimilarly , Ġthe Ġhighest Ġweighted Ġdegree Ġof Ġa Ġgraph Ġis Ġexpected Ġto Ġbe Ġhigher , Ġif Ġthe Ġnode Ġcorresponds Ġto Ġa Ġphrase , Ġinc ong ru ous Ġto Ġsome Ġother Ġphrase Ġin Ġthe Ġtext . </s> The ĠS arc asm ĠClass ifier </s> We Ġinterpret Ġsarc asm Ġdetection Ġas Ġa Ġbinary Ġclassification Ġproblem . ĠThe Ġtraining Ġdata Ġconstitutes Ġ9 94 Ġexamples Ġcreated Ġusing Ġour Ġeye - move ment Ġdatabase Ġfor Ġsarc asm Ġdetection . ĠTo Ġcheck Ġthe Ġeffectiveness Ġof Ġour Ġfeature Ġset , Ġwe Ġobserve Ġthe Ġperformance Ġof Ġmultiple Ġclassification Ġtechniques Ġon Ġour Ġdataset Ġthrough Ġa Ġstrat ified Ġ10 - fold Ġcross Ġvalidation . ĠWe Ġalso Ġcompare Ġthe Ġclassification Ġaccuracy Ġof Ġour Ġsystem Ġand Ġthe Ġbest Ġavailable Ġsystems Ġproposed Ġby Ġr il off 2013 s arc asm Ġand Ġj oshi 2015 har ness ing Ġon Ġour Ġdataset . ĠUsing ĠWe ka ĠB IB REF 18 Ġand ĠLib S VM ĠB IB REF 19 ĠAPIs , Ġwe Ġimplement Ġthe Ġfollowing Ġclass ifiers : </s> Results </s> Table ĠT AB REF 17 Ġshows Ġthe Ġclassification Ġresults Ġconsidering Ġvarious Ġfeature Ġcombinations Ġfor Ġdifferent Ġclass ifiers Ġand Ġother Ġsystems . ĠThese Ġare : </s> Un ig ram Ġ( with Ġprincipal Ġcomponents Ġof Ġun ig ram Ġfeature Ġvectors ), </s> S arc asm Ġ( the Ġfeature - set Ġreported Ġby Ġj oshi 2015 har ness ing Ġsubs uming Ġun ig ram Ġfeatures Ġand Ġfeatures Ġfrom Ġother Ġreported Ġsystems ) </s> G aze Ġ( the Ġsimple Ġand Ġcomplex Ġcognitive Ġfeatures Ġwe Ġintroduce , Ġalong Ġwith Ġread ability Ġand Ġword Ġcount Ġfeatures ), Ġand </s> G aze + S arc asm Ġ( the Ġcomplete Ġset Ġof Ġfeatures ). </s> For Ġall Ġregular Ġclass ifiers , Ġthe Ġgaze Ġfeatures Ġare Ġaveraged Ġacross Ġparticipants Ġand Ġaugmented Ġwith Ġlinguistic Ġand Ġsarc asm Ġrelated Ġfeatures . ĠFor Ġthe ĠMIL R Ġclass ifier , Ġthe Ġgaze Ġfeatures Ġderived Ġfrom Ġeach Ġparticipant Ġare Ġaugmented Ġwith Ġlinguistic Ġfeatures Ġand Ġthus , Ġa Ġmulti Ġinstance ĠâĢ ľ bag âĢ Ŀ Ġof Ġfeatures Ġis Ġformed Ġfor Ġeach Ġsentence Ġin Ġthe Ġtraining Ġdata . ĠThis Ġmulti - instance Ġdataset Ġis Ġgiven Ġto Ġan ĠMIL R Ġclass ifier , Ġwhich Ġfollows Ġthe Ġstandard Ġmulti Ġinstance Ġassumption Ġto Ġderive Ġclass - lab els Ġfor Ġeach Ġbag . </s> For Ġall Ġthe Ġclass ifiers , Ġour Ġfeature Ġcombination Ġoutper forms Ġthe Ġbas elines Ġ( consider ing Ġonly Ġun ig ram Ġfeatures ) Ġas Ġwell Ġas ĠB IB REF 3 Ġ, Ġwith Ġthe ĠMIL R Ġclass ifier Ġgetting Ġan ĠF - score Ġimprovement Ġof Ġ3 . 7 % Ġand ĠKappa Ġdifference Ġof Ġ0 . 08 . ĠWe Ġalso Ġachieve Ġan Ġimprovement Ġof Ġ2 % Ġover Ġthe Ġbaseline , Ġusing ĠS VM Ġclass ifier , Ġwhen Ġwe Ġemploy Ġour Ġfeature Ġset . ĠWe Ġalso Ġobserve Ġthat Ġthe Ġgaze Ġfeatures Ġalone , Ġalso Ġcapture Ġthe Ġdifferences Ġbetween Ġsarc asm Ġand Ġnon - s arc asm Ġclasses Ġwith Ġa Ġhigh - pre cision Ġbut Ġa Ġlow Ġrecall . </s> To Ġsee Ġif Ġthe Ġimprovement Ġobtained Ġis Ġstatistically Ġsignificant Ġover Ġthe Ġstate - of - the Ġart Ġsystem Ġwith Ġtextual Ġsarc asm Ġfeatures Ġalone , Ġwe Ġperform ĠMcN em ar Ġtest . ĠThe Ġoutput Ġof Ġthe ĠS VM Ġclass ifier Ġusing Ġonly Ġlinguistic Ġfeatures Ġused Ġfor Ġsarc asm Ġdetection Ġby Ġj oshi 2015 har ness ing Ġand Ġthe Ġoutput Ġof Ġthe ĠMIL R Ġclass ifier Ġwith Ġthe Ġcomplete Ġset Ġof Ġfeatures Ġare Ġcompared , Ġsetting Ġthreshold ĠIN LINE FORM 0 Ġ. ĠThere Ġwas Ġa Ġsignificant Ġdifference Ġin Ġthe Ġclass ifier 's Ġaccuracy Ġwith Ġp ( two - tailed ) Ġ= Ġ0 . 02 Ġwith Ġan Ġodds - rat io Ġof Ġ1 . 43 , Ġshowing Ġthat Ġthe Ġclassification Ġaccuracy Ġimprovement Ġis Ġunlikely Ġto Ġbe Ġobserved Ġby Ġchance Ġin Ġ95 % Ġconfidence Ġinterval . </s> Considering ĠReading ĠTime Ġas Ġa ĠCognitive ĠFeature Ġalong Ġwith ĠS arc asm ĠFeatures </s> One Ġmay Ġargue Ġthat , Ġconsidering Ġsimple Ġmeasures Ġof Ġreading Ġeffort Ġlike ĠâĢ ľ reading Ġtime âĢ Ŀ Ġas Ġcognitive Ġfeature Ġinstead Ġof Ġthe Ġexpensive Ġeye - tracking Ġfeatures Ġfor Ġsarc asm Ġdetection Ġmay Ġbe Ġa Ġcost - effective Ġsolution . ĠTo Ġexamine Ġthis , Ġwe Ġrepeated Ġour Ġexperiments Ġwith ĠâĢ ľ reading Ġtime âĢ Ŀ Ġconsidered Ġas Ġthe Ġonly Ġcognitive Ġfeature , Ġaugmented Ġwith Ġthe Ġtextual Ġfeatures . ĠThe ĠF - sc ores Ġof Ġall Ġthe Ġclass ifiers Ġturn Ġout Ġto Ġbe Ġclose Ġto Ġthat Ġof Ġthe Ġclass ifiers Ġconsidering Ġsarc asm Ġfeature Ġalone Ġand Ġthe Ġdifference Ġin Ġthe Ġimprovement Ġis Ġnot Ġstatistically Ġsignificant Ġ( ĠIN LINE FORM 0 Ġ). ĠOne Ġthe Ġother Ġhand , ĠF - sc ores Ġwith Ġgaze Ġfeatures Ġare Ġsuperior Ġto Ġthe ĠF - sc ores Ġwhen Ġreading Ġtime Ġis Ġconsidered Ġas Ġa Ġcognitive Ġfeature . </s> How ĠEffective Ġare Ġthe ĠCognitive ĠFeatures </s> We Ġexamine Ġthe Ġeffectiveness Ġof Ġcognitive Ġfeatures Ġon Ġthe Ġclassification Ġaccuracy Ġby Ġvarying Ġthe Ġinput Ġtraining Ġdata Ġsize . ĠTo Ġexamine Ġthis , Ġwe Ġcreate Ġa Ġstrat ified Ġ( keeping Ġthe Ġclass Ġratio Ġconstant ) Ġrandom Ġtrain - test Ġsplit Ġof Ġ80 %: 20 %. ĠWe Ġtrain Ġour Ġclass ifier Ġwith Ġ100 %, Ġ90 %, Ġ80 % Ġand Ġ70 % Ġof Ġthe Ġtraining Ġdata Ġwith Ġour Ġwhole Ġfeature Ġset , Ġand Ġthe Ġfeature Ġcombination Ġfrom Ġj oshi 2015 har ness ing . ĠThe Ġgoodness Ġof Ġour Ġsystem Ġis Ġdemonstrated Ġby Ġimprovements Ġin ĠF - score Ġand ĠKappa Ġstatistics , Ġshown Ġin ĠFigure ĠFIG REF 22 Ġ. </s> We Ġfurther Ġanalyze Ġthe Ġimportance Ġof Ġfeatures Ġby Ġranking Ġthe Ġfeatures Ġbased Ġon Ġ( a ) ĠChi Ġsquared Ġtest , Ġand Ġ( b ) ĠInformation ĠGain Ġtest , Ġusing ĠWe ka 's Ġattribute Ġselection Ġmodule . ĠFigure ĠFIG REF 23 Ġshows Ġthe Ġtop Ġ20 Ġranked Ġfeatures Ġproduced Ġby Ġboth Ġthe Ġtests . ĠFor Ġboth Ġthe Ġcases , Ġwe Ġobserve Ġ16 Ġout Ġof Ġtop Ġ20 Ġfeatures Ġto Ġbe Ġgaze Ġfeatures . ĠFurther , Ġin Ġeach Ġof Ġthe Ġcases , ĠAverage ĠFix ation ĠDuration Ġper ĠWord Ġand ĠL arg est ĠReg ression ĠPosition Ġare Ġseen Ġto Ġbe Ġthe Ġtwo Ġmost Ġsignificant Ġfeatures . </s> Example ĠCases </s> Table ĠT AB REF 21 Ġshows Ġa Ġfew Ġexample Ġcases Ġfrom Ġthe Ġexperiment Ġwith Ġstrat ified Ġ80 %- 20 % Ġtrain - test Ġsplit . </s> Example Ġsentence Ġ1 Ġis Ġsarcastic , Ġand Ġrequires Ġextra - ling u istic Ġknowledge Ġ( about Ġpoor Ġliving Ġconditions Ġat ĠManchester ). ĠHence , Ġthe Ġsarc asm Ġdetector Ġrelying Ġonly Ġon Ġtextual Ġfeatures Ġis Ġunable Ġto Ġdetect Ġthe Ġunderlying Ġinc ong ru ity . ĠHowever , Ġour Ġsystem Ġpredicts Ġthe Ġlabel Ġsuccessfully , Ġpossibly Ġhelped Ġby Ġthe Ġgaze Ġfeatures . </s> Similarly , Ġfor Ġsentence Ġ2 , Ġthe Ġfalse Ġsense Ġof Ġpresence Ġof Ġinc ong ru ity Ġ( due Ġto Ġphrases Ġlike ĠâĢ ľ Hel ped Ġme âĢ Ŀ Ġand ĠâĢ ľ Can 't Ġstop âĢ Ŀ ) Ġaffects Ġthe Ġsystem Ġwith Ġonly Ġlinguistic Ġfeatures . ĠOur Ġsystem , Ġthough , Ġperforms Ġwell Ġin Ġthis Ġcase Ġalso . </s> Sent ence Ġ3 Ġpresents Ġa Ġfalse - negative Ġcase Ġwhere Ġit Ġwas Ġhard Ġfor Ġeven Ġhumans Ġto Ġget Ġthe Ġsarc asm . ĠThis Ġis Ġwhy Ġour Ġgaze Ġfeatures Ġ( and Ġsubsequently Ġthe Ġcomplete Ġset Ġof Ġfeatures ) Ġaccount Ġfor Ġerroneous Ġprediction . </s> In Ġsentence Ġ4 , Ġgaze Ġfeatures Ġalone Ġfalse - ind icate Ġpresence Ġof Ġinc ong ru ity , Ġwhereas Ġthe Ġsystem Ġpredicts Ġcorrectly Ġwhen Ġgaze Ġand Ġlinguistic Ġfeatures Ġare Ġtaken Ġtogether . </s> From Ġthese Ġexamples , Ġit Ġcan Ġbe Ġinferred Ġthat , Ġonly Ġgaze Ġfeatures Ġwould Ġnot Ġhave Ġsu ffic ed Ġto Ġrule Ġout Ġthe Ġpossibility Ġof Ġdetecting Ġother Ġforms Ġof Ġinc ong ru ity Ġthat Ġdo Ġnot Ġresult Ġin Ġsarc asm . </s> Error ĠAnalysis </s> Er rors Ġcommitted Ġby Ġour Ġsystem Ġarise Ġfrom Ġmultiple Ġfactors , Ġstarting Ġfrom Ġlimitations Ġof Ġthe Ġeye - tr acker Ġhardware Ġto Ġerrors Ġcommitted Ġby Ġlinguistic Ġtools Ġand Ġresources . ĠAlso , Ġaggreg ating Ġvarious Ġeye - tracking Ġparameters Ġto Ġextract Ġthe Ġcognitive Ġfeatures Ġmay Ġhave Ġcaused Ġinformation Ġloss Ġin Ġthe Ġregular Ġclassification Ġsetting . </s> Conclusion </s> In Ġthe Ġcurrent Ġwork , Ġwe Ġcreated Ġa Ġnovel Ġframework Ġto Ġdetect Ġsarc asm , Ġthat Ġderives Ġinsights Ġfrom Ġhuman Ġcognition , Ġthat Ġmanifests Ġover Ġeye Ġmovement Ġpatterns . ĠWe Ġhypothesized Ġthat Ġdistinctive Ġeye - move ment Ġpatterns , Ġassociated Ġwith Ġreading Ġsarcastic Ġtext , Ġenables Ġimproved Ġdetection Ġof Ġsarc asm . ĠWe Ġaugmented Ġtraditional Ġlinguistic Ġfeatures Ġwith Ġcognitive Ġfeatures Ġobtained Ġfrom Ġreaders ' Ġeye - move ment Ġdata Ġin Ġthe Ġform Ġof Ġsimple Ġgaze - based Ġfeatures Ġand Ġcomplex Ġfeatures Ġderived Ġfrom Ġa Ġgraph Ġstructure . ĠThis Ġextended Ġfeature - set Ġimproved Ġthe Ġsuccess Ġrate Ġof Ġthe Ġsarc asm Ġdetector Ġby Ġ3 . 7 %, Ġover Ġthe Ġbest Ġavailable Ġsystem . ĠUsing Ġcognitive Ġfeatures Ġin Ġan ĠN LP ĠProcessing Ġsystem Ġlike Ġours Ġis Ġthe Ġfirst Ġproposal Ġof Ġits Ġkind . </s> Our Ġgeneral Ġapproach Ġmay Ġbe Ġuseful Ġin Ġother ĠN LP Ġsub - are as Ġlike Ġsentiment Ġand Ġemotion Ġanalysis , Ġtext Ġsummar ization Ġand Ġquestion Ġanswering , Ġwhere Ġconsidering Ġtextual Ġclues Ġalone Ġdoes Ġnot Ġprove Ġto Ġbe Ġsufficient . ĠWe Ġpropose Ġto Ġaugment Ġthis Ġwork Ġin Ġfuture Ġby Ġexploring Ġdeeper Ġgraph Ġand Ġgaze Ġfeatures . ĠWe Ġalso Ġpropose Ġto Ġdevelop Ġmodels Ġfor Ġthe Ġpurpose Ġof Ġlearning Ġcomplex Ġgaze Ġfeature Ġrepresentation , Ġthat Ġaccounts Ġfor Ġthe Ġpower Ġof Ġindividual Ġeye Ġmovement Ġpatterns Ġalong Ġwith Ġthe Ġaggreg ated Ġpatterns Ġof Ġeye Ġmovements . </s> Acknowled gments </s> We Ġthank Ġthe Ġmembers Ġof ĠCF IL T ĠLab , Ġespecially ĠJay a ĠJ ha Ġand ĠMe gh na ĠSingh , Ġand Ġthe Ġstudents Ġof ĠI IT ĠBombay Ġfor Ġtheir Ġhelp Ġand Ġsupport .
Introduction Sarcasm is an intensive, indirect and complex construct that is often intended to express contempt or ridicule . Sarcasm, in speech, is multi-modal, involving tone, body-language and gestures along with linguistic artifacts used in speech. Sarcasm in text, on the other hand, is more restrictive when it comes to such non-linguistic modalities. This makes recognizing textual sarcasm more challenging for both humans and machines. Sarcasm detection plays an indispensable role in applications like online review summarizers, dialog systems, recommendation systems and sentiment analyzers. This makes automatic detection of sarcasm an important problem. However, it has been quite difficult to solve such a problem with traditional NLP tools and techniques. This is apparent from the results reported by the survey from DBLP:journals/corr/JoshiBC16. The following discussion brings more insights into this. Consider a scenario where an online reviewer gives a negative opinion about a movie through sarcasm: “This is the kind of movie you see because the theater has air conditioning”. It is difficult for an automatic sentiment analyzer to assign a rating to the movie and, in the absence of any other information, such a system may not be able to comprehend that prioritizing the air-conditioning facilities of the theater over the movie experience indicates a negative sentiment towards the movie. This gives an intuition to why, for sarcasm detection, it is necessary to go beyond textual analysis. We aim to address this problem by exploiting the psycholinguistic side of sarcasm detection, using cognitive features extracted with the help of eye-tracking. A motivation to consider cognitive features comes from analyzing human eye-movement trajectories that supports the conjecture: Reading sarcastic texts induces distinctive eye movement patterns, compared to literal texts. The cognitive features, derived from human eye movement patterns observed during reading, include two primary feature types: The cognitive features, along with textual features used in best available sarcasm detectors, are used to train binary classifiers against given sarcasm labels. Our experiments show significant improvement in classification accuracy over the state of the art, by performing such augmentation. Related Work Sarcasm, in general, has been the focus of research for quite some time. In one of the pioneering works jorgensen1984test explained how sarcasm arises when a figurative meaning is used opposite to the literal meaning of the utterance. In the word of clark1984pretense, sarcasm processing involves canceling the indirectly negated message and replacing it with the implicated one. giora1995irony, on the other hand, define sarcasm as a mode of indirect negation that requires processing of both negated and implicated messages. ivanko2003context define sarcasm as a six tuple entity consisting of a speaker, a listener, Context, Utterance, Literal Proposition and Intended Proposition and study the cognitive aspects of sarcasm processing. Computational linguists have previously addressed this problem using rule based and statistical techniques, that make use of : (a) Unigrams and Pragmatic features BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 (b) Stylistic patterns BIBREF4 and patterns related to situational disparity BIBREF5 and (c) Hastag interpretations BIBREF6 , BIBREF7 . Most of the previously done work on sarcasm detection uses distant supervision based techniques (ex: leveraging hashtags) and stylistic/pragmatic features (emoticons, laughter expressions such as “lol” etc). But, detecting sarcasm in linguistically well-formed structures, in absence of explicit cues or information (like emoticons), proves to be hard using such linguistic/stylistic features alone. With the advent of sophisticated eye-trackers and electro/magneto-encephalographic (EEG/MEG) devices, it has been possible to delve deep into the cognitive underpinnings of sarcasm understanding. Filik2014, using a series of eye-tracking and EEG experiments try to show that for unfamiliar ironies, the literal interpretation would be computed first. They also show that a mismatch with context would lead to a re-interpretation of the statement, as being ironic. Camblin2007103 show that in multi-sentence passages, discourse congruence has robust effects on eye movements. This also implies that disrupted processing occurs for discourse incongruent words, even though they are perfectly congruous at the sentence level. In our previous work BIBREF8 , we augment cognitive features, derived from eye-movement patterns of readers, with textual features to detect whether a human reader has realized the presence of sarcasm in text or not. The recent advancements in the literature discussed above, motivate us to explore gaze-based cognition for sarcasm detection. As far as we know, our work is the first of its kind. Eye-tracking Database for Sarcasm Analysis Sarcasm often emanates from incongruity BIBREF9 , which enforces the brain to reanalyze it BIBREF10 . This, in turn, affects the way eyes move through the text. Hence, distinctive eye-movement patterns may be observed in the case of successful processing of sarcasm in text in contrast to literal texts. This hypothesis forms the crux of our method for sarcasm detection and we validate this using our previously released freely available sarcasm dataset BIBREF8 enriched with gaze information. Document Description The database consists of 1,000 short texts, each having 10-40 words. Out of these, 350 are sarcastic and are collected as follows: (a) 103 sentences are from two popular sarcastic quote websites, (b) 76 sarcastic short movie reviews are manually extracted from the Amazon Movie Corpus BIBREF11 by two linguists. (c) 171 tweets are downloaded using the hashtag #sarcasm from Twitter. The 650 non-sarcastic texts are either downloaded from Twitter or extracted from the Amazon Movie Review corpus. The sentences do not contain words/phrases that are highly topic or culture specific. The tweets were normalized to make them linguistically well formed to avoid difficulty in interpreting social media lingo. Every sentence in our dataset carries positive or negative opinion about specific “aspects”. For example, the sentence “The movie is extremely well cast” has positive sentiment about the aspect “cast”. The annotators were seven graduate students with science and engineering background, and possess good English proficiency. They were given a set of instructions beforehand and are advised to seek clarifications before they proceed. The instructions mention the nature of the task, annotation input method, and necessity of head movement minimization during the experiment. Task Description The task assigned to annotators was to read sentences one at a time and label them with with binary labels indicating the polarity (i.e., positive/negative). Note that, the participants were not instructed to annotate whether a sentence is sarcastic or not., to rule out the Priming Effect (i.e., if sarcasm is expected beforehand, processing incongruity becomes relatively easier BIBREF12 ). The setup ensures its “ecological validity” in two ways: (1) Readers are not given any clue that they have to treat sarcasm with special attention. This is done by setting the task to polarity annotation (instead of sarcasm detection). (2) Sarcastic sentences are mixed with non sarcastic text, which does not give prior knowledge about whether the forthcoming text will be sarcastic or not. The eye-tracking experiment is conducted by following the standard norms in eye-movement research BIBREF13 . At a time, one sentence is displayed to the reader along with the “aspect” with respect to which the annotation has to be provided. While reading, an SR-Research Eyelink-1000 eye-tracker (monocular remote mode, sampling rate 500Hz) records several eye-movement parameters like fixations (a long stay of gaze) and saccade (quick jumping of gaze between two positions of rest) and pupil size. The accuracy of polarity annotation varies between 72%-91% for sarcastic texts and 75%-91% for non-sarcastic text, showing the inherent difficulty of sentiment annotation, when sarcasm is present in the text under consideration. Annotation errors may be attributed to: (a) lack of patience/attention while reading, (b) issues related to text comprehension, and (c) confusion/indecisiveness caused due to lack of context. For our analysis, we do not discard the incorrect annotations present in the database. Since our system eventually aims to involve online readers for sarcasm detection, it will be hard to segregate readers who misinterpret the text. We make a rational assumption that, for a particular text, most of the readers, from a fairly large population, will be able to identify sarcasm. Under this assumption, the eye-movement parameters, averaged across all readers in our setting, may not be significantly distorted by a few readers who would have failed to identify sarcasm. This assumption is applicable for both regular and multi-instance based classifiers explained in section SECREF6 . Analysis of Eye-movement Data We observe distinct behavior during sarcasm reading, by analyzing the “fixation duration on the text” (also referred to as “dwell time” in the literature) and “scanpaths” of the readers. Variation in the Average Fixation Duration per Word Since sarcasm in text can be expected to induce cognitive load, it is reasonable to believe that it would require more processing time BIBREF14 . Hence, fixation duration normalized over total word count should usually be higher for a sarcastic text than for a non-sarcastic one. We observe this for all participants in our dataset, with the average fixation duration per word for sarcastic texts being at least 1.5 times more than that of non-sarcastic texts. To test the statistical significance, we conduct a two-tailed t-test (assuming unequal variance) to compare the average fixation duration per word for sarcastic and non-sarcastic texts. The hypothesized mean difference is set to 0 and the error tolerance limit ( INLINEFORM0 ) is set to 0.05. The t-test analysis, presented in Table TABREF11 , shows that for all participants, a statistically significant difference exists between the average fixation duration per word for sarcasm (higher average fixation duration) and non-sarcasm (lower average fixation duration). This affirms that the presence of sarcasm affects the duration of fixation on words. It is important to note that longer fixations may also be caused by other linguistic subtleties (such as difficult words, ambiguity and syntactically complex structures) causing delay in comprehension, or occulomotor control problems forcing readers to spend time adjusting eye-muscles. So, an elevated average fixation duration per word may not sufficiently indicate the presence of sarcasm. But we would also like to share that, for our dataset, when we considered readability (Flesch readability ease-score BIBREF15 ), number of words in a sentence and average character per word along with the sarcasm label as the predictors of average fixation duration following a linear mixed effect model BIBREF16 , sarcasm label turned out to be the most significant predictor with a maximum slope. This indicates that average fixation duration per word has a strong connection with the text being sarcastic, at least in our dataset. We now analyze scanpaths to gain more insights into the sarcasm comprehension process. Analysis of Scanpaths Scanpaths are line-graphs that contain fixations as nodes and saccades as edges; the radii of the nodes represent the fixation duration. A scanpath corresponds to a participant's eye-movement pattern while reading a particular sentence. Figure FIGREF14 presents scanpaths of three participants for the sarcastic sentence S1 and the non-sarcastic sentence S2. The x-axis of the graph represents the sequence of words a reader reads, and the y-axis represents a temporal sequence in milliseconds. Consider a sarcastic text containing incongruous phrases A and B. Our qualitative scanpath-analysis reveals that scanpaths with respect to sarcasm processing have two typical characteristics. Often, a long regression - a saccade that goes to a previously visited segment - is observed when a reader starts reading B after skimming through A. In a few cases, the fixation duration on A and B are significantly higher than the average fixation duration per word. In sentence S1, we see long and multiple regressions from the two incongruous phrases “misconception” and “cherish”, and a few instances where phrases “always cherish” and “original misconception” are fixated longer than usual. Such eye-movement behaviors are not seen for S2. Though sarcasm induces distinctive scanpaths like the ones depicted in Figure FIGREF14 in the observed examples, presence of such patterns is not sufficient to guarantee sarcasm; such patterns may also possibly arise from literal texts. We believe that a combination of linguistic features, readability of text and features derived from scanpaths would help discriminative machine learning models learn sarcasm better. Features for Sarcasm Detection We describe the features used for sarcasm detection in Table . The features enlisted under lexical,implicit incongruity and explicit incongruity are borrowed from various literature (predominantly from joshi2015harnessing). These features are essential to separate sarcasm from other forms semantic incongruity in text (for example ambiguity arising from semantic ambiguity or from metaphors). Two additional textual features viz. readability and word count of the text are also taken under consideration. These features are used to reduce the effect of text hardness and text length on the eye-movement patterns. Simple Gaze Based Features Readers' eye-movement behavior, characterized by fixations, forward saccades, skips and regressions, can be directly quantified by simple statistical aggregation (i.e., either computing features for individual participants and then averaging or performing a multi-instance based learning as explained in section SECREF6 ). Since these eye-movement attributes relate to the cognitive process in reading BIBREF17 , we consider these as features in our model. Some of these features have been reported by sarcasmunderstandability for modeling sarcasm understandability of readers. However, as far as we know, these features are being introduced in NLP tasks like textual sarcasm detection for the first time. The values of these features are believed to increase with the increase in the degree of surprisal caused by incongruity in text (except skip count, which will decrease). Complex Gaze Based Features For these features, we rely on a graph structure, namely “saliency graphs", derived from eye-gaze information and word sequences in the text. For each reader and each sentence, we construct a “saliency graph”, representing the reader's attention characteristics. A saliency graph for a sentence INLINEFORM0 for a reader INLINEFORM1 , represented as INLINEFORM2 , is a graph with vertices ( INLINEFORM3 ) and edges ( INLINEFORM4 ) where each vertex INLINEFORM5 corresponds to a word in INLINEFORM6 (may not be unique) and there exists an edge INLINEFORM7 between vertices INLINEFORM8 and INLINEFORM9 if R performs at least one saccade between the words corresponding to INLINEFORM10 and INLINEFORM11 . Figure FIGREF15 shows an example of a saliency graph.A saliency graph may be weighted, but not necessarily connected, for a given text (as there may be words in the given text with no fixation on them). The “complex” gaze features derived from saliency graphs are also motivated by the theory of incongruity. For instance, Edge Density of a saliency graph increases with the number of distinct saccades, which could arise from the complexity caused by presence of sarcasm. Similarly, the highest weighted degree of a graph is expected to be higher, if the node corresponds to a phrase, incongruous to some other phrase in the text. The Sarcasm Classifier We interpret sarcasm detection as a binary classification problem. The training data constitutes 994 examples created using our eye-movement database for sarcasm detection. To check the effectiveness of our feature set, we observe the performance of multiple classification techniques on our dataset through a stratified 10-fold cross validation. We also compare the classification accuracy of our system and the best available systems proposed by riloff2013sarcasm and joshi2015harnessing on our dataset. Using Weka BIBREF18 and LibSVM BIBREF19 APIs, we implement the following classifiers: Results Table TABREF17 shows the classification results considering various feature combinations for different classifiers and other systems. These are: Unigram (with principal components of unigram feature vectors), Sarcasm (the feature-set reported by joshi2015harnessing subsuming unigram features and features from other reported systems) Gaze (the simple and complex cognitive features we introduce, along with readability and word count features), and Gaze+Sarcasm (the complete set of features). For all regular classifiers, the gaze features are averaged across participants and augmented with linguistic and sarcasm related features. For the MILR classifier, the gaze features derived from each participant are augmented with linguistic features and thus, a multi instance “bag” of features is formed for each sentence in the training data. This multi-instance dataset is given to an MILR classifier, which follows the standard multi instance assumption to derive class-labels for each bag. For all the classifiers, our feature combination outperforms the baselines (considering only unigram features) as well as BIBREF3 , with the MILR classifier getting an F-score improvement of 3.7% and Kappa difference of 0.08. We also achieve an improvement of 2% over the baseline, using SVM classifier, when we employ our feature set. We also observe that the gaze features alone, also capture the differences between sarcasm and non-sarcasm classes with a high-precision but a low recall. To see if the improvement obtained is statistically significant over the state-of-the art system with textual sarcasm features alone, we perform McNemar test. The output of the SVM classifier using only linguistic features used for sarcasm detection by joshi2015harnessing and the output of the MILR classifier with the complete set of features are compared, setting threshold INLINEFORM0 . There was a significant difference in the classifier's accuracy with p(two-tailed) = 0.02 with an odds-ratio of 1.43, showing that the classification accuracy improvement is unlikely to be observed by chance in 95% confidence interval. Considering Reading Time as a Cognitive Feature along with Sarcasm Features One may argue that, considering simple measures of reading effort like “reading time” as cognitive feature instead of the expensive eye-tracking features for sarcasm detection may be a cost-effective solution. To examine this, we repeated our experiments with “reading time” considered as the only cognitive feature, augmented with the textual features. The F-scores of all the classifiers turn out to be close to that of the classifiers considering sarcasm feature alone and the difference in the improvement is not statistically significant ( INLINEFORM0 ). One the other hand, F-scores with gaze features are superior to the F-scores when reading time is considered as a cognitive feature. How Effective are the Cognitive Features We examine the effectiveness of cognitive features on the classification accuracy by varying the input training data size. To examine this, we create a stratified (keeping the class ratio constant) random train-test split of 80%:20%. We train our classifier with 100%, 90%, 80% and 70% of the training data with our whole feature set, and the feature combination from joshi2015harnessing. The goodness of our system is demonstrated by improvements in F-score and Kappa statistics, shown in Figure FIGREF22 . We further analyze the importance of features by ranking the features based on (a) Chi squared test, and (b) Information Gain test, using Weka's attribute selection module. Figure FIGREF23 shows the top 20 ranked features produced by both the tests. For both the cases, we observe 16 out of top 20 features to be gaze features. Further, in each of the cases, Average Fixation Duration per Word and Largest Regression Position are seen to be the two most significant features. Example Cases Table TABREF21 shows a few example cases from the experiment with stratified 80%-20% train-test split. Example sentence 1 is sarcastic, and requires extra-linguistic knowledge (about poor living conditions at Manchester). Hence, the sarcasm detector relying only on textual features is unable to detect the underlying incongruity. However, our system predicts the label successfully, possibly helped by the gaze features. Similarly, for sentence 2, the false sense of presence of incongruity (due to phrases like “Helped me” and “Can't stop”) affects the system with only linguistic features. Our system, though, performs well in this case also. Sentence 3 presents a false-negative case where it was hard for even humans to get the sarcasm. This is why our gaze features (and subsequently the complete set of features) account for erroneous prediction. In sentence 4, gaze features alone false-indicate presence of incongruity, whereas the system predicts correctly when gaze and linguistic features are taken together. From these examples, it can be inferred that, only gaze features would not have sufficed to rule out the possibility of detecting other forms of incongruity that do not result in sarcasm. Error Analysis Errors committed by our system arise from multiple factors, starting from limitations of the eye-tracker hardware to errors committed by linguistic tools and resources. Also, aggregating various eye-tracking parameters to extract the cognitive features may have caused information loss in the regular classification setting. Conclusion In the current work, we created a novel framework to detect sarcasm, that derives insights from human cognition, that manifests over eye movement patterns. We hypothesized that distinctive eye-movement patterns, associated with reading sarcastic text, enables improved detection of sarcasm. We augmented traditional linguistic features with cognitive features obtained from readers' eye-movement data in the form of simple gaze-based features and complex features derived from a graph structure. This extended feature-set improved the success rate of the sarcasm detector by 3.7%, over the best available system. Using cognitive features in an NLP Processing system like ours is the first proposal of its kind. Our general approach may be useful in other NLP sub-areas like sentiment and emotion analysis, text summarization and question answering, where considering textual clues alone does not prove to be sufficient. We propose to augment this work in future by exploring deeper graph and gaze features. We also propose to develop models for the purpose of learning complex gaze feature representation, that accounts for the power of individual eye movement patterns along with the aggregated patterns of eye movements. Acknowledgments We thank the members of CFILT Lab, especially Jaya Jha and Meghna Singh, and the students of IIT Bombay for their help and support.
What is the best reported system?
bbb77f2d6685c9257763ca38afaaef29044b4018
What Ġis Ġthe Ġbest Ġreported Ġsystem ?
G aze ĠS arc asm Ġusing ĠMulti ĠInst ance ĠLog istic ĠReg ression .
<s> What Ġis Ġthe Ġbest Ġreported Ġsystem ? </s> Introduction </s> S arc asm Ġis Ġan Ġintensive , Ġindirect Ġand Ġcomplex Ġconstruct Ġthat Ġis Ġoften Ġintended Ġto Ġexpress Ġcontempt Ġor Ġridicule Ġ. ĠS arc asm , Ġin Ġspeech , Ġis Ġmulti - mod al , Ġinvolving Ġtone , Ġbody - language Ġand Ġgestures Ġalong Ġwith Ġlinguistic Ġartifacts Ġused Ġin Ġspeech . ĠS arc asm Ġin Ġtext , Ġon Ġthe Ġother Ġhand , Ġis Ġmore Ġrestrictive Ġwhen Ġit Ġcomes Ġto Ġsuch Ġnon - ling u istic Ġmod alities . ĠThis Ġmakes Ġrecognizing Ġtextual Ġsarc asm Ġmore Ġchallenging Ġfor Ġboth Ġhumans Ġand Ġmachines . </s> S arc asm Ġdetection Ġplays Ġan Ġindispensable Ġrole Ġin Ġapplications Ġlike Ġonline Ġreview Ġsummar izers , Ġdialog Ġsystems , Ġrecommendation Ġsystems Ġand Ġsentiment Ġanaly zers . ĠThis Ġmakes Ġautomatic Ġdetection Ġof Ġsarc asm Ġan Ġimportant Ġproblem . ĠHowever , Ġit Ġhas Ġbeen Ġquite Ġdifficult Ġto Ġsolve Ġsuch Ġa Ġproblem Ġwith Ġtraditional ĠN LP Ġtools Ġand Ġtechniques . ĠThis Ġis Ġapparent Ġfrom Ġthe Ġresults Ġreported Ġby Ġthe Ġsurvey Ġfrom ĠD BL P : j ournals / cor r / J oshi BC 16 . ĠThe Ġfollowing Ġdiscussion Ġbrings Ġmore Ġinsights Ġinto Ġthis . </s> Consider Ġa Ġscenario Ġwhere Ġan Ġonline Ġreviewer Ġgives Ġa Ġnegative Ġopinion Ġabout Ġa Ġmovie Ġthrough Ġsarc asm : ĠâĢ ľ This Ġis Ġthe Ġkind Ġof Ġmovie Ġyou Ġsee Ġbecause Ġthe Ġtheater Ġhas Ġair Ġconditioning âĢ Ŀ . ĠIt Ġis Ġdifficult Ġfor Ġan Ġautomatic Ġsentiment Ġanaly zer Ġto Ġassign Ġa Ġrating Ġto Ġthe Ġmovie Ġand , Ġin Ġthe Ġabsence Ġof Ġany Ġother Ġinformation , Ġsuch Ġa Ġsystem Ġmay Ġnot Ġbe Ġable Ġto Ġcomprehend Ġthat Ġpriorit izing Ġthe Ġair - condition ing Ġfacilities Ġof Ġthe Ġtheater Ġover Ġthe Ġmovie Ġexperience Ġindicates Ġa Ġnegative Ġsentiment Ġtowards Ġthe Ġmovie . ĠThis Ġgives Ġan Ġintuition Ġto Ġwhy , Ġfor Ġsarc asm Ġdetection , Ġit Ġis Ġnecessary Ġto Ġgo Ġbeyond Ġtextual Ġanalysis . </s> We Ġaim Ġto Ġaddress Ġthis Ġproblem Ġby Ġexploiting Ġthe Ġpsych ol ingu istic Ġside Ġof Ġsarc asm Ġdetection , Ġusing Ġcognitive Ġfeatures Ġextracted Ġwith Ġthe Ġhelp Ġof Ġeye - tracking . ĠA Ġmotivation Ġto Ġconsider Ġcognitive Ġfeatures Ġcomes Ġfrom Ġanalyzing Ġhuman Ġeye - move ment Ġtraject ories Ġthat Ġsupports Ġthe Ġconjecture : ĠReading Ġsarcastic Ġtexts Ġinduces Ġdistinctive Ġeye Ġmovement Ġpatterns , Ġcompared Ġto Ġliteral Ġtexts . ĠThe Ġcognitive Ġfeatures , Ġderived Ġfrom Ġhuman Ġeye Ġmovement Ġpatterns Ġobserved Ġduring Ġreading , Ġinclude Ġtwo Ġprimary Ġfeature Ġtypes : </s> The Ġcognitive Ġfeatures , Ġalong Ġwith Ġtextual Ġfeatures Ġused Ġin Ġbest Ġavailable Ġsarc asm Ġdetectors , Ġare Ġused Ġto Ġtrain Ġbinary Ġclass ifiers Ġagainst Ġgiven Ġsarc asm Ġlabels . ĠOur Ġexperiments Ġshow Ġsignificant Ġimprovement Ġin Ġclassification Ġaccuracy Ġover Ġthe Ġstate Ġof Ġthe Ġart , Ġby Ġperforming Ġsuch Ġaug mentation . </s> Related ĠWork </s> S arc asm , Ġin Ġgeneral , Ġhas Ġbeen Ġthe Ġfocus Ġof Ġresearch Ġfor Ġquite Ġsome Ġtime . ĠIn Ġone Ġof Ġthe Ġpioneering Ġworks Ġj org ensen 1984 test Ġexplained Ġhow Ġsarc asm Ġarises Ġwhen Ġa Ġfigur ative Ġmeaning Ġis Ġused Ġopposite Ġto Ġthe Ġliteral Ġmeaning Ġof Ġthe Ġutter ance . ĠIn Ġthe Ġword Ġof Ġcl ark 1984 pret ense , Ġsarc asm Ġprocessing Ġinvolves Ġcance ling Ġthe Ġindirectly Ġneg ated Ġmessage Ġand Ġreplacing Ġit Ġwith Ġthe Ġimplicated Ġone . Ġg ior a 1995 ir ony , Ġon Ġthe Ġother Ġhand , Ġdefine Ġsarc asm Ġas Ġa Ġmode Ġof Ġindirect Ġneg ation Ġthat Ġrequires Ġprocessing Ġof Ġboth Ġneg ated Ġand Ġimplicated Ġmessages . Ġiv ank o 2003 context Ġdefine Ġsarc asm Ġas Ġa Ġsix Ġtuple Ġentity Ġconsisting Ġof Ġa Ġspeaker , Ġa Ġlistener , ĠContext , ĠUt ter ance , ĠLit eral ĠProposition Ġand ĠInt ended ĠProposition Ġand Ġstudy Ġthe Ġcognitive Ġaspects Ġof Ġsarc asm Ġprocessing . </s> Com put ational Ġlingu ists Ġhave Ġpreviously Ġaddressed Ġthis Ġproblem Ġusing Ġrule Ġbased Ġand Ġstatistical Ġtechniques , Ġthat Ġmake Ġuse Ġof Ġ: Ġ( a ) ĠUn ig rams Ġand ĠPr ag matic Ġfeatures ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ( b ) ĠSt yl istic Ġpatterns ĠB IB REF 4 Ġand Ġpatterns Ġrelated Ġto Ġsituational Ġdisparity ĠB IB REF 5 Ġand Ġ( c ) ĠHast ag Ġinterpretations ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. </s> Most Ġof Ġthe Ġpreviously Ġdone Ġwork Ġon Ġsarc asm Ġdetection Ġuses Ġdistant Ġsupervision Ġbased Ġtechniques Ġ( ex : Ġleveraging Ġhasht ags ) Ġand Ġstyl istic / pr ag matic Ġfeatures Ġ( em otic ons , Ġlaughter Ġexpressions Ġsuch Ġas ĠâĢ ľ lol âĢ Ŀ Ġetc ). ĠBut , Ġdetecting Ġsarc asm Ġin Ġlingu istically Ġwell - formed Ġstructures , Ġin Ġabsence Ġof Ġexplicit Ġcues Ġor Ġinformation Ġ( like Ġemot icons ), Ġproves Ġto Ġbe Ġhard Ġusing Ġsuch Ġlinguistic / st yl istic Ġfeatures Ġalone . </s> With Ġthe Ġadvent Ġof Ġsophisticated Ġeye - track ers Ġand Ġelectro / mag net o - ence phal ographic Ġ( EE G / M EG ) Ġdevices , Ġit Ġhas Ġbeen Ġpossible Ġto Ġdelve Ġdeep Ġinto Ġthe Ġcognitive Ġunder p inn ings Ġof Ġsarc asm Ġunderstanding . ĠFil ik 2014 , Ġusing Ġa Ġseries Ġof Ġeye - tracking Ġand ĠEEG Ġexperiments Ġtry Ġto Ġshow Ġthat Ġfor Ġunfamiliar Ġiron ies , Ġthe Ġliteral Ġinterpretation Ġwould Ġbe Ġcomputed Ġfirst . ĠThey Ġalso Ġshow Ġthat Ġa Ġmismatch Ġwith Ġcontext Ġwould Ġlead Ġto Ġa Ġre - interpret ation Ġof Ġthe Ġstatement , Ġas Ġbeing Ġironic . ĠCam bl in 2007 103 Ġshow Ġthat Ġin Ġmulti - sent ence Ġpassages , Ġdiscourse Ġcon gru ence Ġhas Ġrobust Ġeffects Ġon Ġeye Ġmovements . ĠThis Ġalso Ġimplies Ġthat Ġdisrupted Ġprocessing Ġoccurs Ġfor Ġdiscourse Ġinc ong ru ent Ġwords , Ġeven Ġthough Ġthey Ġare Ġperfectly Ġcon gru ous Ġat Ġthe Ġsentence Ġlevel . ĠIn Ġour Ġprevious Ġwork ĠB IB REF 8 Ġ, Ġwe Ġaugment Ġcognitive Ġfeatures , Ġderived Ġfrom Ġeye - move ment Ġpatterns Ġof Ġreaders , Ġwith Ġtextual Ġfeatures Ġto Ġdetect Ġwhether Ġa Ġhuman Ġreader Ġhas Ġrealized Ġthe Ġpresence Ġof Ġsarc asm Ġin Ġtext Ġor Ġnot . </s> The Ġrecent Ġadvancements Ġin Ġthe Ġliterature Ġdiscussed Ġabove , Ġmotivate Ġus Ġto Ġexplore Ġgaze - based Ġcognition Ġfor Ġsarc asm Ġdetection . ĠAs Ġfar Ġas Ġwe Ġknow , Ġour Ġwork Ġis Ġthe Ġfirst Ġof Ġits Ġkind . </s> Eye - tracking ĠDatabase Ġfor ĠS arc asm ĠAnalysis </s> S arc asm Ġoften Ġeman ates Ġfrom Ġinc ong ru ity ĠB IB REF 9 Ġ, Ġwhich Ġen forces Ġthe Ġbrain Ġto Ġre analy ze Ġit ĠB IB REF 10 Ġ. ĠThis , Ġin Ġturn , Ġaffects Ġthe Ġway Ġeyes Ġmove Ġthrough Ġthe Ġtext . ĠHence , Ġdistinctive Ġeye - move ment Ġpatterns Ġmay Ġbe Ġobserved Ġin Ġthe Ġcase Ġof Ġsuccessful Ġprocessing Ġof Ġsarc asm Ġin Ġtext Ġin Ġcontrast Ġto Ġliteral Ġtexts . ĠThis Ġhypothesis Ġforms Ġthe Ġcru x Ġof Ġour Ġmethod Ġfor Ġsarc asm Ġdetection Ġand Ġwe Ġvalidate Ġthis Ġusing Ġour Ġpreviously Ġreleased Ġfreely Ġavailable Ġsarc asm Ġdataset ĠB IB REF 8 Ġenriched Ġwith Ġgaze Ġinformation . </s> Document ĠDescription </s> The Ġdatabase Ġconsists Ġof Ġ1 , 000 Ġshort Ġtexts , Ġeach Ġhaving Ġ10 - 40 Ġwords . ĠOut Ġof Ġthese , Ġ350 Ġare Ġsarcastic Ġand Ġare Ġcollected Ġas Ġfollows : Ġ( a ) Ġ103 Ġsentences Ġare Ġfrom Ġtwo Ġpopular Ġsarcastic Ġquote Ġwebsites , Ġ( b ) Ġ76 Ġsarcastic Ġshort Ġmovie Ġreviews Ġare Ġmanually Ġextracted Ġfrom Ġthe ĠAmazon ĠMovie ĠCorpus ĠB IB REF 11 Ġby Ġtwo Ġlingu ists . Ġ( c ) Ġ171 Ġtweets Ġare Ġdownloaded Ġusing Ġthe Ġhashtag Ġ# s arc asm Ġfrom ĠTwitter . ĠThe Ġ650 Ġnon - s ar cast ic Ġtexts Ġare Ġeither Ġdownloaded Ġfrom ĠTwitter Ġor Ġextracted Ġfrom Ġthe ĠAmazon ĠMovie ĠReview Ġcorpus . ĠThe Ġsentences Ġdo Ġnot Ġcontain Ġwords / ph r ases Ġthat Ġare Ġhighly Ġtopic Ġor Ġculture Ġspecific . ĠThe Ġtweets Ġwere Ġnormalized Ġto Ġmake Ġthem Ġlingu istically Ġwell Ġformed Ġto Ġavoid Ġdifficulty Ġin Ġinterpreting Ġsocial Ġmedia Ġling o . ĠEvery Ġsentence Ġin Ġour Ġdataset Ġcarries Ġpositive Ġor Ġnegative Ġopinion Ġabout Ġspecific ĠâĢ ľ as pects âĢ Ŀ . ĠFor Ġexample , Ġthe Ġsentence ĠâĢ ľ The Ġmovie Ġis Ġextremely Ġwell Ġcast âĢ Ŀ Ġhas Ġpositive Ġsentiment Ġabout Ġthe Ġaspect ĠâĢ ľ cast âĢ Ŀ . </s> The Ġannot ators Ġwere Ġseven Ġgraduate Ġstudents Ġwith Ġscience Ġand Ġengineering Ġbackground , Ġand Ġpossess Ġgood ĠEnglish Ġproficiency . ĠThey Ġwere Ġgiven Ġa Ġset Ġof Ġinstructions Ġbeforehand Ġand Ġare Ġadvised Ġto Ġseek Ġclar ifications Ġbefore Ġthey Ġproceed . ĠThe Ġinstructions Ġmention Ġthe Ġnature Ġof Ġthe Ġtask , Ġannotation Ġinput Ġmethod , Ġand Ġnecessity Ġof Ġhead Ġmovement Ġminim ization Ġduring Ġthe Ġexperiment . </s> Task ĠDescription </s> The Ġtask Ġassigned Ġto Ġannot ators Ġwas Ġto Ġread Ġsentences Ġone Ġat Ġa Ġtime Ġand Ġlabel Ġthem Ġwith Ġwith Ġbinary Ġlabels Ġindicating Ġthe Ġpol arity Ġ( i . e ., Ġpositive / negative ). ĠNote Ġthat , Ġthe Ġparticipants Ġwere Ġnot Ġinstructed Ġto Ġannot ate Ġwhether Ġa Ġsentence Ġis Ġsarcastic Ġor Ġnot ., Ġto Ġrule Ġout Ġthe ĠPrim ing ĠEffect Ġ( i . e ., Ġif Ġsarc asm Ġis Ġexpected Ġbeforehand , Ġprocessing Ġinc ong ru ity Ġbecomes Ġrelatively Ġeasier ĠB IB REF 12 Ġ). ĠThe Ġsetup Ġensures Ġits ĠâĢ ľ ec ological Ġvalidity âĢ Ŀ Ġin Ġtwo Ġways : Ġ( 1 ) ĠReaders Ġare Ġnot Ġgiven Ġany Ġclue Ġthat Ġthey Ġhave Ġto Ġtreat Ġsarc asm Ġwith Ġspecial Ġattention . ĠThis Ġis Ġdone Ġby Ġsetting Ġthe Ġtask Ġto Ġpol arity Ġannotation Ġ( instead Ġof Ġsarc asm Ġdetection ). Ġ( 2 ) ĠSar cast ic Ġsentences Ġare Ġmixed Ġwith Ġnon Ġsarcastic Ġtext , Ġwhich Ġdoes Ġnot Ġgive Ġprior Ġknowledge Ġabout Ġwhether Ġthe Ġforthcoming Ġtext Ġwill Ġbe Ġsarcastic Ġor Ġnot . </s> The Ġeye - tracking Ġexperiment Ġis Ġconducted Ġby Ġfollowing Ġthe Ġstandard Ġnorms Ġin Ġeye - move ment Ġresearch ĠB IB REF 13 Ġ. ĠAt Ġa Ġtime , Ġone Ġsentence Ġis Ġdisplayed Ġto Ġthe Ġreader Ġalong Ġwith Ġthe ĠâĢ ľ as pect âĢ Ŀ Ġwith Ġrespect Ġto Ġwhich Ġthe Ġannotation Ġhas Ġto Ġbe Ġprovided . ĠWhile Ġreading , Ġan ĠSR - Research ĠEy el ink - 1000 Ġeye - tr acker Ġ( mon ocular Ġremote Ġmode , Ġsampling Ġrate Ġ500 Hz ) Ġrecords Ġseveral Ġeye - move ment Ġparameters Ġlike Ġfix ations Ġ( a Ġlong Ġstay Ġof Ġgaze ) Ġand Ġs acc ade Ġ( quick Ġjumping Ġof Ġgaze Ġbetween Ġtwo Ġpositions Ġof Ġrest ) Ġand Ġpupil Ġsize . </s> The Ġaccuracy Ġof Ġpol arity Ġannotation Ġvaries Ġbetween Ġ72 %- 91 % Ġfor Ġsarcastic Ġtexts Ġand Ġ75 %- 91 % Ġfor Ġnon - s ar cast ic Ġtext , Ġshowing Ġthe Ġinherent Ġdifficulty Ġof Ġsentiment Ġannotation , Ġwhen Ġsarc asm Ġis Ġpresent Ġin Ġthe Ġtext Ġunder Ġconsideration . ĠAn notation Ġerrors Ġmay Ġbe Ġattributed Ġto : Ġ( a ) Ġlack Ġof Ġpatience / att ention Ġwhile Ġreading , Ġ( b ) Ġissues Ġrelated Ġto Ġtext Ġcomprehension , Ġand Ġ( c ) Ġconfusion / ind ec is iveness Ġcaused Ġdue Ġto Ġlack Ġof Ġcontext . </s> For Ġour Ġanalysis , Ġwe Ġdo Ġnot Ġdiscard Ġthe Ġincorrect Ġannotations Ġpresent Ġin Ġthe Ġdatabase . ĠSince Ġour Ġsystem Ġeventually Ġaims Ġto Ġinvolve Ġonline Ġreaders Ġfor Ġsarc asm Ġdetection , Ġit Ġwill Ġbe Ġhard Ġto Ġsegreg ate Ġreaders Ġwho Ġmisinterpret Ġthe Ġtext . ĠWe Ġmake Ġa Ġrational Ġassumption Ġthat , Ġfor Ġa Ġparticular Ġtext , Ġmost Ġof Ġthe Ġreaders , Ġfrom Ġa Ġfairly Ġlarge Ġpopulation , Ġwill Ġbe Ġable Ġto Ġidentify Ġsarc asm . ĠUnder Ġthis Ġassumption , Ġthe Ġeye - move ment Ġparameters , Ġaveraged Ġacross Ġall Ġreaders Ġin Ġour Ġsetting , Ġmay Ġnot Ġbe Ġsignificantly Ġdistorted Ġby Ġa Ġfew Ġreaders Ġwho Ġwould Ġhave Ġfailed Ġto Ġidentify Ġsarc asm . ĠThis Ġassumption Ġis Ġapplicable Ġfor Ġboth Ġregular Ġand Ġmulti - instance Ġbased Ġclass ifiers Ġexplained Ġin Ġsection ĠSEC REF 6 Ġ. </s> Analysis Ġof ĠEye - move ment ĠData </s> We Ġobserve Ġdistinct Ġbehavior Ġduring Ġsarc asm Ġreading , Ġby Ġanalyzing Ġthe ĠâĢ ľ fix ation Ġduration Ġon Ġthe Ġtext âĢ Ŀ Ġ( also Ġreferred Ġto Ġas ĠâĢ ľ d well Ġtime âĢ Ŀ Ġin Ġthe Ġliterature ) Ġand ĠâĢ ľ scan path s âĢ Ŀ Ġof Ġthe Ġreaders . </s> Vari ation Ġin Ġthe ĠAverage ĠFix ation ĠDuration Ġper ĠWord </s> Since Ġsarc asm Ġin Ġtext Ġcan Ġbe Ġexpected Ġto Ġinduce Ġcognitive Ġload , Ġit Ġis Ġreasonable Ġto Ġbelieve Ġthat Ġit Ġwould Ġrequire Ġmore Ġprocessing Ġtime ĠB IB REF 14 Ġ. ĠHence , Ġfixation Ġduration Ġnormalized Ġover Ġtotal Ġword Ġcount Ġshould Ġusually Ġbe Ġhigher Ġfor Ġa Ġsarcastic Ġtext Ġthan Ġfor Ġa Ġnon - s ar cast ic Ġone . ĠWe Ġobserve Ġthis Ġfor Ġall Ġparticipants Ġin Ġour Ġdataset , Ġwith Ġthe Ġaverage Ġfixation Ġduration Ġper Ġword Ġfor Ġsarcastic Ġtexts Ġbeing Ġat Ġleast Ġ1 . 5 Ġtimes Ġmore Ġthan Ġthat Ġof Ġnon - s ar cast ic Ġtexts . ĠTo Ġtest Ġthe Ġstatistical Ġsignificance , Ġwe Ġconduct Ġa Ġtwo - tailed Ġt - test Ġ( assuming Ġunequal Ġvariance ) Ġto Ġcompare Ġthe Ġaverage Ġfixation Ġduration Ġper Ġword Ġfor Ġsarcastic Ġand Ġnon - s ar cast ic Ġtexts . ĠThe Ġhypothesized Ġmean Ġdifference Ġis Ġset Ġto Ġ0 Ġand Ġthe Ġerror Ġtolerance Ġlimit Ġ( ĠIN LINE FORM 0 Ġ) Ġis Ġset Ġto Ġ0 . 05 . ĠThe Ġt - test Ġanalysis , Ġpresented Ġin ĠTable ĠT AB REF 11 Ġ, Ġshows Ġthat Ġfor Ġall Ġparticipants , Ġa Ġstatistically Ġsignificant Ġdifference Ġexists Ġbetween Ġthe Ġaverage Ġfixation Ġduration Ġper Ġword Ġfor Ġsarc asm Ġ( higher Ġaverage Ġfixation Ġduration ) Ġand Ġnon - s arc asm Ġ( lower Ġaverage Ġfixation Ġduration ). ĠThis Ġaff irms Ġthat Ġthe Ġpresence Ġof Ġsarc asm Ġaffects Ġthe Ġduration Ġof Ġfixation Ġon Ġwords . </s> It Ġis Ġimportant Ġto Ġnote Ġthat Ġlonger Ġfix ations Ġmay Ġalso Ġbe Ġcaused Ġby Ġother Ġlinguistic Ġsubt let ies Ġ( such Ġas Ġdifficult Ġwords , Ġambiguity Ġand Ġsynt act ically Ġcomplex Ġstructures ) Ġcausing Ġdelay Ġin Ġcomprehension , Ġor Ġocc ul om otor Ġcontrol Ġproblems Ġforcing Ġreaders Ġto Ġspend Ġtime Ġadjusting Ġeye - mus cles . ĠSo , Ġan Ġelevated Ġaverage Ġfixation Ġduration Ġper Ġword Ġmay Ġnot Ġsufficiently Ġindicate Ġthe Ġpresence Ġof Ġsarc asm . ĠBut Ġwe Ġwould Ġalso Ġlike Ġto Ġshare Ġthat , Ġfor Ġour Ġdataset , Ġwhen Ġwe Ġconsidered Ġread ability Ġ( F les ch Ġread ability Ġease - score ĠB IB REF 15 Ġ), Ġnumber Ġof Ġwords Ġin Ġa Ġsentence Ġand Ġaverage Ġcharacter Ġper Ġword Ġalong Ġwith Ġthe Ġsarc asm Ġlabel Ġas Ġthe Ġpredict ors Ġof Ġaverage Ġfixation Ġduration Ġfollowing Ġa Ġlinear Ġmixed Ġeffect Ġmodel ĠB IB REF 16 Ġ, Ġsarc asm Ġlabel Ġturned Ġout Ġto Ġbe Ġthe Ġmost Ġsignificant Ġpredictor Ġwith Ġa Ġmaximum Ġslope . ĠThis Ġindicates Ġthat Ġaverage Ġfixation Ġduration Ġper Ġword Ġhas Ġa Ġstrong Ġconnection Ġwith Ġthe Ġtext Ġbeing Ġsarcastic , Ġat Ġleast Ġin Ġour Ġdataset . </s> We Ġnow Ġanalyze Ġscan path s Ġto Ġgain Ġmore Ġinsights Ġinto Ġthe Ġsarc asm Ġcomprehension Ġprocess . </s> Analysis Ġof ĠScan path s </s> Scan path s Ġare Ġline - graph s Ġthat Ġcontain Ġfix ations Ġas Ġnodes Ġand Ġs acc ades Ġas Ġedges ; Ġthe Ġrad ii Ġof Ġthe Ġnodes Ġrepresent Ġthe Ġfixation Ġduration . ĠA Ġscan path Ġcorresponds Ġto Ġa Ġparticipant 's Ġeye - move ment Ġpattern Ġwhile Ġreading Ġa Ġparticular Ġsentence . ĠFigure ĠFIG REF 14 Ġpresents Ġscan path s Ġof Ġthree Ġparticipants Ġfor Ġthe Ġsarcastic Ġsentence ĠS 1 Ġand Ġthe Ġnon - s ar cast ic Ġsentence ĠS 2 . ĠThe Ġx - axis Ġof Ġthe Ġgraph Ġrepresents Ġthe Ġsequence Ġof Ġwords Ġa Ġreader Ġreads , Ġand Ġthe Ġy - axis Ġrepresents Ġa Ġtemporal Ġsequence Ġin Ġmilliseconds . </s> Consider Ġa Ġsarcastic Ġtext Ġcontaining Ġinc ong ru ous Ġphrases ĠA Ġand ĠB . ĠOur Ġqualitative Ġscan path - analysis Ġreveals Ġthat Ġscan path s Ġwith Ġrespect Ġto Ġsarc asm Ġprocessing Ġhave Ġtwo Ġtypical Ġcharacteristics . ĠOften , Ġa Ġlong Ġregression Ġ- Ġa Ġs acc ade Ġthat Ġgoes Ġto Ġa Ġpreviously Ġvisited Ġsegment Ġ- Ġis Ġobserved Ġwhen Ġa Ġreader Ġstarts Ġreading ĠB Ġafter Ġsk imming Ġthrough ĠA . ĠIn Ġa Ġfew Ġcases , Ġthe Ġfixation Ġduration Ġon ĠA Ġand ĠB Ġare Ġsignificantly Ġhigher Ġthan Ġthe Ġaverage Ġfixation Ġduration Ġper Ġword . ĠIn Ġsentence ĠS 1 , Ġwe Ġsee Ġlong Ġand Ġmultiple Ġregress ions Ġfrom Ġthe Ġtwo Ġinc ong ru ous Ġphrases ĠâĢ ľ mis con ception âĢ Ŀ Ġand ĠâĢ ľ cher ish âĢ Ŀ , Ġand Ġa Ġfew Ġinstances Ġwhere Ġphrases ĠâĢ ľ always Ġcherish âĢ Ŀ Ġand ĠâĢ ľ original Ġmisconception âĢ Ŀ Ġare Ġfix ated Ġlonger Ġthan Ġusual . ĠSuch Ġeye - move ment Ġbehaviors Ġare Ġnot Ġseen Ġfor ĠS 2 . </s> Though Ġsarc asm Ġinduces Ġdistinctive Ġscan path s Ġlike Ġthe Ġones Ġdepicted Ġin ĠFigure ĠFIG REF 14 Ġin Ġthe Ġobserved Ġexamples , Ġpresence Ġof Ġsuch Ġpatterns Ġis Ġnot Ġsufficient Ġto Ġguarantee Ġsarc asm ; Ġsuch Ġpatterns Ġmay Ġalso Ġpossibly Ġarise Ġfrom Ġliteral Ġtexts . ĠWe Ġbelieve Ġthat Ġa Ġcombination Ġof Ġlinguistic Ġfeatures , Ġread ability Ġof Ġtext Ġand Ġfeatures Ġderived Ġfrom Ġscan path s Ġwould Ġhelp Ġdiscrim in ative Ġmachine Ġlearning Ġmodels Ġlearn Ġsarc asm Ġbetter . </s> Features Ġfor ĠS arc asm ĠDetection </s> We Ġdescribe Ġthe Ġfeatures Ġused Ġfor Ġsarc asm Ġdetection Ġin ĠTable Ġ. ĠThe Ġfeatures Ġenlisted Ġunder Ġlex ical , impl icit Ġinc ong ru ity Ġand Ġexplicit Ġinc ong ru ity Ġare Ġborrowed Ġfrom Ġvarious Ġliterature Ġ( pred omin antly Ġfrom Ġj oshi 2015 har ness ing ). ĠThese Ġfeatures Ġare Ġessential Ġto Ġseparate Ġsarc asm Ġfrom Ġother Ġforms Ġsemantic Ġinc ong ru ity Ġin Ġtext Ġ( for Ġexample Ġambiguity Ġarising Ġfrom Ġsemantic Ġambiguity Ġor Ġfrom Ġmetaphors ). ĠTwo Ġadditional Ġtextual Ġfeatures Ġviz . Ġread ability Ġand Ġword Ġcount Ġof Ġthe Ġtext Ġare Ġalso Ġtaken Ġunder Ġconsideration . ĠThese Ġfeatures Ġare Ġused Ġto Ġreduce Ġthe Ġeffect Ġof Ġtext Ġhardness Ġand Ġtext Ġlength Ġon Ġthe Ġeye - move ment Ġpatterns . </s> Simple ĠG aze ĠBased ĠFeatures </s> Read ers ' Ġeye - move ment Ġbehavior , Ġcharacterized Ġby Ġfix ations , Ġforward Ġs acc ades , Ġsk ips Ġand Ġregress ions , Ġcan Ġbe Ġdirectly Ġquant ified Ġby Ġsimple Ġstatistical Ġaggregation Ġ( i . e ., Ġeither Ġcomputing Ġfeatures Ġfor Ġindividual Ġparticipants Ġand Ġthen Ġaveraging Ġor Ġperforming Ġa Ġmulti - instance Ġbased Ġlearning Ġas Ġexplained Ġin Ġsection ĠSEC REF 6 Ġ). ĠSince Ġthese Ġeye - move ment Ġattributes Ġrelate Ġto Ġthe Ġcognitive Ġprocess Ġin Ġreading ĠB IB REF 17 Ġ, Ġwe Ġconsider Ġthese Ġas Ġfeatures Ġin Ġour Ġmodel . ĠSome Ġof Ġthese Ġfeatures Ġhave Ġbeen Ġreported Ġby Ġsarc asm under stand ability Ġfor Ġmodeling Ġsarc asm Ġunderstand ability Ġof Ġreaders . ĠHowever , Ġas Ġfar Ġas Ġwe Ġknow , Ġthese Ġfeatures Ġare Ġbeing Ġintroduced Ġin ĠN LP Ġtasks Ġlike Ġtextual Ġsarc asm Ġdetection Ġfor Ġthe Ġfirst Ġtime . ĠThe Ġvalues Ġof Ġthese Ġfeatures Ġare Ġbelieved Ġto Ġincrease Ġwith Ġthe Ġincrease Ġin Ġthe Ġdegree Ġof Ġsurpr isal Ġcaused Ġby Ġinc ong ru ity Ġin Ġtext Ġ( except Ġskip Ġcount , Ġwhich Ġwill Ġdecrease ). </s> Com plex ĠG aze ĠBased ĠFeatures </s> For Ġthese Ġfeatures , Ġwe Ġrely Ġon Ġa Ġgraph Ġstructure , Ġnamely ĠâĢ ľ sal iency Ġgraphs ", Ġderived Ġfrom Ġeye - g aze Ġinformation Ġand Ġword Ġsequences Ġin Ġthe Ġtext . </s> For Ġeach Ġreader Ġand Ġeach Ġsentence , Ġwe Ġconstruct Ġa ĠâĢ ľ sal iency Ġgraph âĢ Ŀ , Ġrepresenting Ġthe Ġreader 's Ġattention Ġcharacteristics . ĠA Ġsal iency Ġgraph Ġfor Ġa Ġsentence ĠIN LINE FORM 0 Ġfor Ġa Ġreader ĠIN LINE FORM 1 Ġ, Ġrepresented Ġas ĠIN LINE FORM 2 Ġ, Ġis Ġa Ġgraph Ġwith Ġvert ices Ġ( ĠIN LINE FORM 3 Ġ) Ġand Ġedges Ġ( ĠIN LINE FORM 4 Ġ) Ġwhere Ġeach Ġvertex ĠIN LINE FORM 5 Ġcorresponds Ġto Ġa Ġword Ġin ĠIN LINE FORM 6 Ġ( may Ġnot Ġbe Ġunique ) Ġand Ġthere Ġexists Ġan Ġedge ĠIN LINE FORM 7 Ġbetween Ġvert ices ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġif ĠR Ġperforms Ġat Ġleast Ġone Ġs acc ade Ġbetween Ġthe Ġwords Ġcorresponding Ġto ĠIN LINE FORM 10 Ġand ĠIN LINE FORM 11 Ġ. </s> Figure ĠFIG REF 15 Ġshows Ġan Ġexample Ġof Ġa Ġsal iency Ġgraph . A Ġsal iency Ġgraph Ġmay Ġbe Ġweighted , Ġbut Ġnot Ġnecessarily Ġconnected , Ġfor Ġa Ġgiven Ġtext Ġ( as Ġthere Ġmay Ġbe Ġwords Ġin Ġthe Ġgiven Ġtext Ġwith Ġno Ġfixation Ġon Ġthem ). ĠThe ĠâĢ ľ complex âĢ Ŀ Ġgaze Ġfeatures Ġderived Ġfrom Ġsal iency Ġgraphs Ġare Ġalso Ġmotivated Ġby Ġthe Ġtheory Ġof Ġinc ong ru ity . ĠFor Ġinstance , ĠEdge ĠD ensity Ġof Ġa Ġsal iency Ġgraph Ġincreases Ġwith Ġthe Ġnumber Ġof Ġdistinct Ġs acc ades , Ġwhich Ġcould Ġarise Ġfrom Ġthe Ġcomplexity Ġcaused Ġby Ġpresence Ġof Ġsarc asm . ĠSimilarly , Ġthe Ġhighest Ġweighted Ġdegree Ġof Ġa Ġgraph Ġis Ġexpected Ġto Ġbe Ġhigher , Ġif Ġthe Ġnode Ġcorresponds Ġto Ġa Ġphrase , Ġinc ong ru ous Ġto Ġsome Ġother Ġphrase Ġin Ġthe Ġtext . </s> The ĠS arc asm ĠClass ifier </s> We Ġinterpret Ġsarc asm Ġdetection Ġas Ġa Ġbinary Ġclassification Ġproblem . ĠThe Ġtraining Ġdata Ġconstitutes Ġ9 94 Ġexamples Ġcreated Ġusing Ġour Ġeye - move ment Ġdatabase Ġfor Ġsarc asm Ġdetection . ĠTo Ġcheck Ġthe Ġeffectiveness Ġof Ġour Ġfeature Ġset , Ġwe Ġobserve Ġthe Ġperformance Ġof Ġmultiple Ġclassification Ġtechniques Ġon Ġour Ġdataset Ġthrough Ġa Ġstrat ified Ġ10 - fold Ġcross Ġvalidation . ĠWe Ġalso Ġcompare Ġthe Ġclassification Ġaccuracy Ġof Ġour Ġsystem Ġand Ġthe Ġbest Ġavailable Ġsystems Ġproposed Ġby Ġr il off 2013 s arc asm Ġand Ġj oshi 2015 har ness ing Ġon Ġour Ġdataset . ĠUsing ĠWe ka ĠB IB REF 18 Ġand ĠLib S VM ĠB IB REF 19 ĠAPIs , Ġwe Ġimplement Ġthe Ġfollowing Ġclass ifiers : </s> Results </s> Table ĠT AB REF 17 Ġshows Ġthe Ġclassification Ġresults Ġconsidering Ġvarious Ġfeature Ġcombinations Ġfor Ġdifferent Ġclass ifiers Ġand Ġother Ġsystems . ĠThese Ġare : </s> Un ig ram Ġ( with Ġprincipal Ġcomponents Ġof Ġun ig ram Ġfeature Ġvectors ), </s> S arc asm Ġ( the Ġfeature - set Ġreported Ġby Ġj oshi 2015 har ness ing Ġsubs uming Ġun ig ram Ġfeatures Ġand Ġfeatures Ġfrom Ġother Ġreported Ġsystems ) </s> G aze Ġ( the Ġsimple Ġand Ġcomplex Ġcognitive Ġfeatures Ġwe Ġintroduce , Ġalong Ġwith Ġread ability Ġand Ġword Ġcount Ġfeatures ), Ġand </s> G aze + S arc asm Ġ( the Ġcomplete Ġset Ġof Ġfeatures ). </s> For Ġall Ġregular Ġclass ifiers , Ġthe Ġgaze Ġfeatures Ġare Ġaveraged Ġacross Ġparticipants Ġand Ġaugmented Ġwith Ġlinguistic Ġand Ġsarc asm Ġrelated Ġfeatures . ĠFor Ġthe ĠMIL R Ġclass ifier , Ġthe Ġgaze Ġfeatures Ġderived Ġfrom Ġeach Ġparticipant Ġare Ġaugmented Ġwith Ġlinguistic Ġfeatures Ġand Ġthus , Ġa Ġmulti Ġinstance ĠâĢ ľ bag âĢ Ŀ Ġof Ġfeatures Ġis Ġformed Ġfor Ġeach Ġsentence Ġin Ġthe Ġtraining Ġdata . ĠThis Ġmulti - instance Ġdataset Ġis Ġgiven Ġto Ġan ĠMIL R Ġclass ifier , Ġwhich Ġfollows Ġthe Ġstandard Ġmulti Ġinstance Ġassumption Ġto Ġderive Ġclass - lab els Ġfor Ġeach Ġbag . </s> For Ġall Ġthe Ġclass ifiers , Ġour Ġfeature Ġcombination Ġoutper forms Ġthe Ġbas elines Ġ( consider ing Ġonly Ġun ig ram Ġfeatures ) Ġas Ġwell Ġas ĠB IB REF 3 Ġ, Ġwith Ġthe ĠMIL R Ġclass ifier Ġgetting Ġan ĠF - score Ġimprovement Ġof Ġ3 . 7 % Ġand ĠKappa Ġdifference Ġof Ġ0 . 08 . ĠWe Ġalso Ġachieve Ġan Ġimprovement Ġof Ġ2 % Ġover Ġthe Ġbaseline , Ġusing ĠS VM Ġclass ifier , Ġwhen Ġwe Ġemploy Ġour Ġfeature Ġset . ĠWe Ġalso Ġobserve Ġthat Ġthe Ġgaze Ġfeatures Ġalone , Ġalso Ġcapture Ġthe Ġdifferences Ġbetween Ġsarc asm Ġand Ġnon - s arc asm Ġclasses Ġwith Ġa Ġhigh - pre cision Ġbut Ġa Ġlow Ġrecall . </s> To Ġsee Ġif Ġthe Ġimprovement Ġobtained Ġis Ġstatistically Ġsignificant Ġover Ġthe Ġstate - of - the Ġart Ġsystem Ġwith Ġtextual Ġsarc asm Ġfeatures Ġalone , Ġwe Ġperform ĠMcN em ar Ġtest . ĠThe Ġoutput Ġof Ġthe ĠS VM Ġclass ifier Ġusing Ġonly Ġlinguistic Ġfeatures Ġused Ġfor Ġsarc asm Ġdetection Ġby Ġj oshi 2015 har ness ing Ġand Ġthe Ġoutput Ġof Ġthe ĠMIL R Ġclass ifier Ġwith Ġthe Ġcomplete Ġset Ġof Ġfeatures Ġare Ġcompared , Ġsetting Ġthreshold ĠIN LINE FORM 0 Ġ. ĠThere Ġwas Ġa Ġsignificant Ġdifference Ġin Ġthe Ġclass ifier 's Ġaccuracy Ġwith Ġp ( two - tailed ) Ġ= Ġ0 . 02 Ġwith Ġan Ġodds - rat io Ġof Ġ1 . 43 , Ġshowing Ġthat Ġthe Ġclassification Ġaccuracy Ġimprovement Ġis Ġunlikely Ġto Ġbe Ġobserved Ġby Ġchance Ġin Ġ95 % Ġconfidence Ġinterval . </s> Considering ĠReading ĠTime Ġas Ġa ĠCognitive ĠFeature Ġalong Ġwith ĠS arc asm ĠFeatures </s> One Ġmay Ġargue Ġthat , Ġconsidering Ġsimple Ġmeasures Ġof Ġreading Ġeffort Ġlike ĠâĢ ľ reading Ġtime âĢ Ŀ Ġas Ġcognitive Ġfeature Ġinstead Ġof Ġthe Ġexpensive Ġeye - tracking Ġfeatures Ġfor Ġsarc asm Ġdetection Ġmay Ġbe Ġa Ġcost - effective Ġsolution . ĠTo Ġexamine Ġthis , Ġwe Ġrepeated Ġour Ġexperiments Ġwith ĠâĢ ľ reading Ġtime âĢ Ŀ Ġconsidered Ġas Ġthe Ġonly Ġcognitive Ġfeature , Ġaugmented Ġwith Ġthe Ġtextual Ġfeatures . ĠThe ĠF - sc ores Ġof Ġall Ġthe Ġclass ifiers Ġturn Ġout Ġto Ġbe Ġclose Ġto Ġthat Ġof Ġthe Ġclass ifiers Ġconsidering Ġsarc asm Ġfeature Ġalone Ġand Ġthe Ġdifference Ġin Ġthe Ġimprovement Ġis Ġnot Ġstatistically Ġsignificant Ġ( ĠIN LINE FORM 0 Ġ). ĠOne Ġthe Ġother Ġhand , ĠF - sc ores Ġwith Ġgaze Ġfeatures Ġare Ġsuperior Ġto Ġthe ĠF - sc ores Ġwhen Ġreading Ġtime Ġis Ġconsidered Ġas Ġa Ġcognitive Ġfeature . </s> How ĠEffective Ġare Ġthe ĠCognitive ĠFeatures </s> We Ġexamine Ġthe Ġeffectiveness Ġof Ġcognitive Ġfeatures Ġon Ġthe Ġclassification Ġaccuracy Ġby Ġvarying Ġthe Ġinput Ġtraining Ġdata Ġsize . ĠTo Ġexamine Ġthis , Ġwe Ġcreate Ġa Ġstrat ified Ġ( keeping Ġthe Ġclass Ġratio Ġconstant ) Ġrandom Ġtrain - test Ġsplit Ġof Ġ80 %: 20 %. ĠWe Ġtrain Ġour Ġclass ifier Ġwith Ġ100 %, Ġ90 %, Ġ80 % Ġand Ġ70 % Ġof Ġthe Ġtraining Ġdata Ġwith Ġour Ġwhole Ġfeature Ġset , Ġand Ġthe Ġfeature Ġcombination Ġfrom Ġj oshi 2015 har ness ing . ĠThe Ġgoodness Ġof Ġour Ġsystem Ġis Ġdemonstrated Ġby Ġimprovements Ġin ĠF - score Ġand ĠKappa Ġstatistics , Ġshown Ġin ĠFigure ĠFIG REF 22 Ġ. </s> We Ġfurther Ġanalyze Ġthe Ġimportance Ġof Ġfeatures Ġby Ġranking Ġthe Ġfeatures Ġbased Ġon Ġ( a ) ĠChi Ġsquared Ġtest , Ġand Ġ( b ) ĠInformation ĠGain Ġtest , Ġusing ĠWe ka 's Ġattribute Ġselection Ġmodule . ĠFigure ĠFIG REF 23 Ġshows Ġthe Ġtop Ġ20 Ġranked Ġfeatures Ġproduced Ġby Ġboth Ġthe Ġtests . ĠFor Ġboth Ġthe Ġcases , Ġwe Ġobserve Ġ16 Ġout Ġof Ġtop Ġ20 Ġfeatures Ġto Ġbe Ġgaze Ġfeatures . ĠFurther , Ġin Ġeach Ġof Ġthe Ġcases , ĠAverage ĠFix ation ĠDuration Ġper ĠWord Ġand ĠL arg est ĠReg ression ĠPosition Ġare Ġseen Ġto Ġbe Ġthe Ġtwo Ġmost Ġsignificant Ġfeatures . </s> Example ĠCases </s> Table ĠT AB REF 21 Ġshows Ġa Ġfew Ġexample Ġcases Ġfrom Ġthe Ġexperiment Ġwith Ġstrat ified Ġ80 %- 20 % Ġtrain - test Ġsplit . </s> Example Ġsentence Ġ1 Ġis Ġsarcastic , Ġand Ġrequires Ġextra - ling u istic Ġknowledge Ġ( about Ġpoor Ġliving Ġconditions Ġat ĠManchester ). ĠHence , Ġthe Ġsarc asm Ġdetector Ġrelying Ġonly Ġon Ġtextual Ġfeatures Ġis Ġunable Ġto Ġdetect Ġthe Ġunderlying Ġinc ong ru ity . ĠHowever , Ġour Ġsystem Ġpredicts Ġthe Ġlabel Ġsuccessfully , Ġpossibly Ġhelped Ġby Ġthe Ġgaze Ġfeatures . </s> Similarly , Ġfor Ġsentence Ġ2 , Ġthe Ġfalse Ġsense Ġof Ġpresence Ġof Ġinc ong ru ity Ġ( due Ġto Ġphrases Ġlike ĠâĢ ľ Hel ped Ġme âĢ Ŀ Ġand ĠâĢ ľ Can 't Ġstop âĢ Ŀ ) Ġaffects Ġthe Ġsystem Ġwith Ġonly Ġlinguistic Ġfeatures . ĠOur Ġsystem , Ġthough , Ġperforms Ġwell Ġin Ġthis Ġcase Ġalso . </s> Sent ence Ġ3 Ġpresents Ġa Ġfalse - negative Ġcase Ġwhere Ġit Ġwas Ġhard Ġfor Ġeven Ġhumans Ġto Ġget Ġthe Ġsarc asm . ĠThis Ġis Ġwhy Ġour Ġgaze Ġfeatures Ġ( and Ġsubsequently Ġthe Ġcomplete Ġset Ġof Ġfeatures ) Ġaccount Ġfor Ġerroneous Ġprediction . </s> In Ġsentence Ġ4 , Ġgaze Ġfeatures Ġalone Ġfalse - ind icate Ġpresence Ġof Ġinc ong ru ity , Ġwhereas Ġthe Ġsystem Ġpredicts Ġcorrectly Ġwhen Ġgaze Ġand Ġlinguistic Ġfeatures Ġare Ġtaken Ġtogether . </s> From Ġthese Ġexamples , Ġit Ġcan Ġbe Ġinferred Ġthat , Ġonly Ġgaze Ġfeatures Ġwould Ġnot Ġhave Ġsu ffic ed Ġto Ġrule Ġout Ġthe Ġpossibility Ġof Ġdetecting Ġother Ġforms Ġof Ġinc ong ru ity Ġthat Ġdo Ġnot Ġresult Ġin Ġsarc asm . </s> Error ĠAnalysis </s> Er rors Ġcommitted Ġby Ġour Ġsystem Ġarise Ġfrom Ġmultiple Ġfactors , Ġstarting Ġfrom Ġlimitations Ġof Ġthe Ġeye - tr acker Ġhardware Ġto Ġerrors Ġcommitted Ġby Ġlinguistic Ġtools Ġand Ġresources . ĠAlso , Ġaggreg ating Ġvarious Ġeye - tracking Ġparameters Ġto Ġextract Ġthe Ġcognitive Ġfeatures Ġmay Ġhave Ġcaused Ġinformation Ġloss Ġin Ġthe Ġregular Ġclassification Ġsetting . </s> Conclusion </s> In Ġthe Ġcurrent Ġwork , Ġwe Ġcreated Ġa Ġnovel Ġframework Ġto Ġdetect Ġsarc asm , Ġthat Ġderives Ġinsights Ġfrom Ġhuman Ġcognition , Ġthat Ġmanifests Ġover Ġeye Ġmovement Ġpatterns . ĠWe Ġhypothesized Ġthat Ġdistinctive Ġeye - move ment Ġpatterns , Ġassociated Ġwith Ġreading Ġsarcastic Ġtext , Ġenables Ġimproved Ġdetection Ġof Ġsarc asm . ĠWe Ġaugmented Ġtraditional Ġlinguistic Ġfeatures Ġwith Ġcognitive Ġfeatures Ġobtained Ġfrom Ġreaders ' Ġeye - move ment Ġdata Ġin Ġthe Ġform Ġof Ġsimple Ġgaze - based Ġfeatures Ġand Ġcomplex Ġfeatures Ġderived Ġfrom Ġa Ġgraph Ġstructure . ĠThis Ġextended Ġfeature - set Ġimproved Ġthe Ġsuccess Ġrate Ġof Ġthe Ġsarc asm Ġdetector Ġby Ġ3 . 7 %, Ġover Ġthe Ġbest Ġavailable Ġsystem . ĠUsing Ġcognitive Ġfeatures Ġin Ġan ĠN LP ĠProcessing Ġsystem Ġlike Ġours Ġis Ġthe Ġfirst Ġproposal Ġof Ġits Ġkind . </s> Our Ġgeneral Ġapproach Ġmay Ġbe Ġuseful Ġin Ġother ĠN LP Ġsub - are as Ġlike Ġsentiment Ġand Ġemotion Ġanalysis , Ġtext Ġsummar ization Ġand Ġquestion Ġanswering , Ġwhere Ġconsidering Ġtextual Ġclues Ġalone Ġdoes Ġnot Ġprove Ġto Ġbe Ġsufficient . ĠWe Ġpropose Ġto Ġaugment Ġthis Ġwork Ġin Ġfuture Ġby Ġexploring Ġdeeper Ġgraph Ġand Ġgaze Ġfeatures . ĠWe Ġalso Ġpropose Ġto Ġdevelop Ġmodels Ġfor Ġthe Ġpurpose Ġof Ġlearning Ġcomplex Ġgaze Ġfeature Ġrepresentation , Ġthat Ġaccounts Ġfor Ġthe Ġpower Ġof Ġindividual Ġeye Ġmovement Ġpatterns Ġalong Ġwith Ġthe Ġaggreg ated Ġpatterns Ġof Ġeye Ġmovements . </s> Acknowled gments </s> We Ġthank Ġthe Ġmembers Ġof ĠCF IL T ĠLab , Ġespecially ĠJay a ĠJ ha Ġand ĠMe gh na ĠSingh , Ġand Ġthe Ġstudents Ġof ĠI IT ĠBombay Ġfor Ġtheir Ġhelp Ġand Ġsupport .