BLEU-1, Meteor , Rouge-L 
EXTRACTIVE
BLEU-1, Meteor ,  Rouge-L 
EXTRACTIVE
As described in the question-answer writing process, the answers in our dataset are different from those in some existing extractive datasets. Thus we consider the task of answer generation for TweetQA and we use several standard metrics for natural language generation to evaluate QA systems on our dataset, namely we consider BLEU-1 BIBREF16 , Meteor BIBREF17 and Rouge-L BIBREF18 in this paper.
As described in the question-answer writing process, the answers in our dataset are different from those in some existing extractive datasets. Thus we consider the task of answer generation for TweetQA and we use several standard metrics for natural language generation to evaluate QA systems on our dataset, namely we consider BLEU-1 BIBREF16 , Meteor BIBREF17 and Rouge-L BIBREF18 in this paper.
1907.06292
Introduction </s> Social Ġmedia Ġis Ġnow Ġbecoming Ġan Ġimportant Ġreal - time Ġinformation Ġsource , Ġespecially Ġduring Ġnatural Ġdisasters Ġand Ġemergencies . ĠIt Ġis Ġnow Ġvery Ġcommon Ġfor Ġtraditional Ġnews Ġmedia Ġto Ġfrequently Ġprobe Ġusers Ġand Ġresort Ġto Ġsocial Ġmedia Ġplatforms Ġto Ġobtain Ġreal - time Ġdevelopments Ġof Ġevents . ĠAccording Ġto Ġa Ġrecent Ġsurvey Ġby ĠPew ĠResearch ĠCenter , Ġin Ġ2017 , Ġmore Ġthan Ġtwo - thirds Ġof ĠAmericans Ġread Ġsome Ġof Ġtheir Ġnews Ġon Ġsocial Ġmedia . ĠEven Ġfor ĠAmerican Ġpeople Ġwho Ġare Ġ50 Ġor Ġolder , ĠIN LINE FORM 0 Ġof Ġthem Ġreport Ġgetting Ġnews Ġfrom Ġsocial Ġmedia , Ġwhich Ġis ĠIN LINE FORM 1 Ġpoints Ġhigher Ġthan Ġthe Ġnumber Ġin Ġ2016 . ĠAmong Ġall Ġmajor Ġsocial Ġmedia Ġsites , ĠTwitter Ġis Ġmost Ġfrequently Ġused Ġas Ġa Ġnews Ġsource , Ġwith ĠIN LINE FORM 2 Ġof Ġits Ġusers Ġobtaining Ġtheir Ġnews Ġfrom ĠTwitter . ĠAll Ġthese Ġstatistical Ġfacts Ġsuggest Ġthat Ġunderstanding Ġuser - generated Ġnoisy Ġsocial Ġmedia Ġtext Ġfrom ĠTwitter Ġis Ġa Ġsignificant Ġtask . </s> In Ġrecent Ġyears , Ġwhile Ġseveral Ġtools Ġfor Ġcore Ġnatural Ġlanguage Ġunderstanding Ġtasks Ġinvolving Ġsynt actic Ġand Ġsemantic Ġanalysis Ġhave Ġbeen Ġdeveloped Ġfor Ġnoisy Ġsocial Ġmedia Ġtext ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġthere Ġis Ġlittle Ġwork Ġon Ġquestion Ġanswering Ġor Ġreading Ġcomprehension Ġover Ġsocial Ġmedia , Ġwith Ġthe Ġprimary Ġbottleneck Ġbeing Ġthe Ġlack Ġof Ġavailable Ġdatasets . ĠWe Ġobserve Ġthat Ġrecently Ġproposed ĠQ A Ġdatasets Ġusually Ġfocus Ġon Ġformal Ġdomains , Ġe . g . ĠCNN / Daily Mail ĠB IB REF 4 Ġand ĠNews Q A ĠB IB REF 5 Ġon Ġnews Ġarticles ; ĠS Qu AD ĠB IB REF 6 Ġand ĠWiki M ovies ĠB IB REF 7 Ġthat Ġuse ĠWikipedia . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġthe Ġfirst Ġlarge - scale Ġdataset Ġfor ĠQ A Ġover Ġsocial Ġmedia Ġdata . ĠRather Ġthan Ġna ively Ġobtaining Ġtweets Ġfrom ĠTwitter Ġusing Ġthe ĠTwitter ĠAPI Ġwhich Ġcan Ġyield Ġirrelevant Ġtweets Ġwith Ġno Ġvaluable Ġinformation , Ġwe Ġrestrict Ġourselves Ġonly Ġto Ġtweets Ġwhich Ġhave Ġbeen Ġused Ġby Ġjournalists Ġin Ġnews Ġarticles Ġthus Ġimplicitly Ġimplying Ġthat Ġsuch Ġtweets Ġcontain Ġuseful Ġand Ġrelevant Ġinformation . ĠTo Ġobtain Ġsuch Ġrelevant Ġtweets , Ġwe Ġcrawled Ġthousands Ġof Ġnews Ġarticles Ġthat Ġinclude Ġtweet Ġquotations Ġand Ġthen Ġemployed Ġcrowd - s ourcing Ġto Ġelicit Ġquestions Ġand Ġanswers Ġbased Ġon Ġthese Ġevent - aligned Ġtweets . ĠTable ĠT AB REF 3 Ġgives Ġan Ġexample Ġfrom Ġour ĠTweet Q A Ġdataset . ĠIt Ġshows Ġthat ĠQ A Ġover Ġtweets Ġraises Ġchallenges Ġnot Ġonly Ġbecause Ġof Ġthe Ġinformal Ġnature Ġof Ġoral - style Ġtexts Ġ( e . g . Ġinfer ring Ġthe Ġanswer Ġfrom Ġmultiple Ġshort Ġsentences , Ġlike Ġthe Ġphrase ĠâĢ ľ so Ġyoung âĢ Ŀ Ġthat Ġforms Ġan Ġindependent Ġsentence Ġin Ġthe Ġexample ), Ġbut Ġalso Ġfrom Ġtweet - specific Ġexpressions Ġ( such Ġas Ġinfer ring Ġthat Ġit Ġis ĠâĢ ľ Jay ĠSean âĢ Ŀ Ġfeeling Ġsad Ġabout ĠPaul 's Ġdeath Ġbecause Ġhe Ġposted Ġthe Ġtweet ). </s> Furthermore , Ġwe Ġshow Ġthe Ġdistinctive Ġnature Ġof ĠTweet Q A Ġby Ġcomparing Ġthe Ġcollected Ġdata Ġwith Ġtraditional ĠQ A Ġdatasets Ġcollected Ġprimarily Ġfrom Ġformal Ġdomains . ĠIn Ġparticular , Ġwe Ġdemonstrate Ġempir ically Ġthat Ġthree Ġstrong Ġneural Ġmodels Ġwhich Ġachieve Ġgood Ġperformance Ġon Ġformal Ġdata Ġdo Ġnot Ġgeneral ize Ġwell Ġto Ġsocial Ġmedia Ġdata , Ġbringing Ġout Ġchallenges Ġto Ġdeveloping ĠQ A Ġsystems Ġthat Ġwork Ġwell Ġon Ġsocial Ġmedia Ġdomains . </s> In Ġsummary , Ġour Ġcontributions Ġare : </s> Tweet Q A </s> In Ġthis Ġsection , Ġwe Ġfirst Ġdescribe Ġthe Ġthree - step Ġdata Ġcollection Ġprocess Ġof ĠTweet Q A : Ġtweet Ġcrawling , Ġquestion - answer Ġwriting Ġand Ġanswer Ġvalidation . ĠNext , Ġwe Ġdefine Ġthe Ġspecific Ġtask Ġof ĠTweet Q A Ġand Ġdiscuss Ġseveral Ġevaluation Ġmetrics . ĠTo Ġbetter Ġunderstand Ġthe Ġcharacteristics Ġof Ġthe ĠTweet Q A Ġtask , Ġwe Ġalso Ġinclude Ġour Ġanalysis Ġon Ġthe Ġanswer Ġand Ġquestion Ġcharacteristics Ġusing Ġa Ġsubset Ġof ĠQ A Ġpairs Ġfrom Ġthe Ġdevelopment Ġset . </s> Data ĠCollection </s> One Ġmajor Ġchallenge Ġof Ġbuilding Ġa ĠQ A Ġdataset Ġon Ġtweets Ġis Ġthe Ġsp arsity Ġof Ġinformative Ġtweets . ĠMany Ġusers Ġwrite Ġtweets Ġto Ġexpress Ġtheir Ġfeelings Ġor Ġemotions Ġabout Ġtheir Ġpersonal Ġlives . ĠThese Ġtweets Ġare Ġgenerally Ġunin form ative Ġand Ġalso Ġvery Ġdifficult Ġto Ġask Ġquestions Ġabout . ĠGiven Ġthe Ġlinguistic Ġvariance Ġof Ġtweets , Ġit Ġis Ġgenerally Ġhard Ġto Ġdirectly Ġdistinguish Ġthose Ġtweets Ġfrom Ġinformative Ġones . ĠIn Ġterms Ġof Ġthis , Ġrather Ġthan Ġstarting Ġfrom ĠTwitter ĠAPI ĠSearch , Ġwe Ġlook Ġinto Ġthe Ġarchived Ġsnapshots Ġof Ġtwo Ġmajor Ġnews Ġwebsites Ġ( CNN , ĠNBC ), Ġand Ġthen Ġextract Ġthe Ġtweet Ġblocks Ġthat Ġare Ġembedded Ġin Ġthe Ġnews Ġarticles . ĠIn Ġorder Ġto Ġget Ġenough Ġdata , Ġwe Ġfirst Ġextract Ġthe ĠURLs Ġof Ġall Ġsection Ġpages Ġ( e . g . ĠWorld , ĠPolitics , ĠMoney , ĠTech ) Ġfrom Ġthe Ġsnapshot Ġof Ġeach Ġhome Ġpage Ġand Ġthen Ġcrawl Ġall Ġarticles Ġwith Ġtweets Ġfrom Ġthese Ġsection Ġpages . ĠNote Ġthat Ġanother Ġpossible Ġway Ġto Ġcollect Ġinformative Ġtweets Ġis Ġto Ġdownload Ġthe Ġtweets Ġthat Ġare Ġposted Ġby Ġthe Ġofficial ĠTwitter Ġaccounts Ġof Ġnews Ġmedia . ĠHowever , Ġthese Ġtweets Ġare Ġoften Ġjust Ġthe Ġsumm aries Ġof Ġnews Ġarticles , Ġwhich Ġare Ġwritten Ġin Ġformal Ġtext . ĠAs Ġour Ġfocus Ġis Ġto Ġdevelop Ġa Ġdataset Ġfor ĠQ A Ġon Ġinformal Ġsocial Ġmedia Ġtext , Ġwe Ġdo Ġnot Ġconsider Ġthis Ġapproach . </s> After Ġwe Ġextracted Ġtweets Ġfrom Ġarchived Ġnews Ġarticles , Ġwe Ġobserved Ġthat Ġthere Ġis Ġstill Ġa Ġportion Ġof Ġtweets Ġthat Ġhave Ġvery Ġsimple Ġsemantic Ġstructures Ġand Ġthus Ġare Ġvery Ġdifficult Ġto Ġraise Ġmeaningful Ġquestions . ĠAn Ġexample Ġof Ġsuch Ġtweets Ġcan Ġbe Ġlike : ĠâĢ ľ W anted Ġto Ġshare Ġthis Ġtoday Ġ- Ġ@ IA m Steve Har vey ". ĠThis Ġtweet Ġis Ġactually Ġtalking Ġabout Ġan Ġimage Ġattached Ġto Ġthis Ġtweet . ĠSome Ġother Ġtweets Ġwith Ġsimple Ġtext Ġstructures Ġmay Ġtalk Ġabout Ġan Ġinserted Ġlink Ġor Ġeven Ġvideos . ĠTo Ġfilter Ġout Ġthese Ġtweets Ġthat Ġheavily Ġrely Ġon Ġattached Ġmedia Ġto Ġconvey Ġinformation , Ġwe Ġutilize Ġa Ġstate - of - the - art Ġsemantic Ġrole Ġlabeling Ġmodel Ġtrained Ġon ĠCo N LL - 2005 ĠB IB REF 15 Ġto Ġanalyze Ġthe Ġpredicate - argument Ġstructure Ġof Ġthe Ġtweets Ġcollected Ġfrom Ġnews Ġarticles Ġand Ġkeep Ġonly Ġthe Ġtweets Ġwith Ġmore Ġthan Ġtwo Ġlabeled Ġarguments . ĠThis Ġfiltering Ġprocess Ġalso Ġautomatically Ġfilters Ġout Ġmost Ġof Ġthe Ġshort Ġtweets . ĠFor Ġthe Ġtweets Ġcollected Ġfrom ĠCNN , ĠIN LINE FORM 0 Ġof Ġthem Ġwere Ġfiltered Ġvia Ġsemantic Ġrole Ġlabeling . ĠFor Ġtweets Ġfrom ĠNBC , ĠIN LINE FORM 1 Ġof Ġthe Ġtweets Ġwere Ġfiltered . </s> We Ġthen Ġuse ĠAmazon ĠMechanical ĠTurk Ġto Ġcollect Ġquestion - answer Ġpairs Ġfor Ġthe Ġfiltered Ġtweets . ĠFor Ġeach ĠHuman ĠIntelligence ĠTask Ġ( H IT ), Ġwe Ġask Ġthe Ġworker Ġto Ġread Ġthree Ġtweets Ġand Ġwrite Ġtwo Ġquestion - answer Ġpairs Ġfor Ġeach Ġtweet . ĠTo Ġensure Ġthe Ġquality , Ġwe Ġrequire Ġthe Ġworkers Ġto Ġbe Ġlocated Ġin Ġmajor ĠEnglish Ġspeaking Ġcountries Ġ( i . e . ĠCanada , ĠUS , Ġand ĠUK ) Ġand Ġhave Ġan Ġacceptance Ġrate Ġlarger Ġthan ĠIN LINE FORM 0 Ġ. ĠSince Ġwe Ġuse Ġtweets Ġas Ġcontext , Ġlots Ġof Ġimportant Ġinformation Ġare Ġcontained Ġin Ġhasht ags Ġor Ġeven Ġem oj is . ĠInstead Ġof Ġonly Ġshowing Ġthe Ġtext Ġto Ġthe Ġworkers , Ġwe Ġuse Ġjavascript Ġto Ġdirectly Ġembed Ġthe Ġwhole Ġtweet Ġinto Ġeach ĠHIT . ĠThis Ġgives Ġworkers Ġthe Ġsame Ġexperience Ġas Ġreading Ġtweets Ġvia Ġweb Ġbrowsers Ġand Ġhelp Ġthem Ġto Ġbetter Ġcompose Ġquestions . </s> To Ġavoid Ġtrivial Ġquestions Ġthat Ġcan Ġbe Ġsimply Ġanswered Ġby Ġsuperficial Ġtext Ġmatching Ġmethods Ġor Ġtoo Ġchallenging Ġquestions Ġthat Ġrequire Ġbackground Ġknowledge . ĠWe Ġexplicitly Ġstate Ġthe Ġfollowing Ġitems Ġin Ġthe ĠHIT Ġinstructions Ġfor Ġquestion Ġwriting : </s> No ĠYes - no Ġquestions Ġshould Ġbe Ġasked . </s> The Ġquestion Ġshould Ġhave Ġat Ġleast Ġfive Ġwords . </s> V ideos , Ġimages Ġor Ġinserted Ġlinks Ġshould Ġnot Ġbe Ġconsidered . </s> No Ġbackground Ġknowledge Ġshould Ġbe Ġrequired Ġto Ġanswer Ġthe Ġquestion . </s> To Ġhelp Ġthe Ġworkers Ġbetter Ġfollow Ġthe Ġinstructions , Ġwe Ġalso Ġinclude Ġa Ġrepresentative Ġexample Ġshowing Ġboth Ġgood Ġand Ġbad Ġquestions Ġor Ġanswers Ġin Ġour Ġinstructions . ĠFigure ĠFIG REF 14 Ġshows Ġthe Ġexample Ġwe Ġuse Ġto Ġguide Ġthe Ġworkers . </s> As Ġfor Ġthe Ġanswers , Ġsince Ġthe Ġcontext Ġwe Ġconsider Ġis Ġrelatively Ġshorter Ġthan Ġthe Ġcontext Ġof Ġprevious Ġdatasets , Ġwe Ġdo Ġnot Ġrestrict Ġthe Ġanswers Ġto Ġbe Ġin Ġthe Ġtweet , Ġotherwise , Ġthe Ġtask Ġmay Ġpotentially Ġbe Ġsimplified Ġas Ġa Ġclassification Ġproblem . ĠThe Ġworkers Ġare Ġallowed Ġto Ġwrite Ġtheir Ġanswers Ġin Ġtheir Ġown Ġwords . ĠWe Ġjust Ġrequire Ġthe Ġanswers Ġto Ġbe Ġbrief Ġand Ġcan Ġbe Ġdirectly Ġinferred Ġfrom Ġthe Ġtweets . </s> After Ġwe Ġretrieve Ġthe ĠQ A Ġpairs Ġfrom Ġall ĠHIT s , Ġwe Ġconduct Ġfurther Ġpost - fil tering Ġto Ġfilter Ġout Ġthe Ġpairs Ġfrom Ġworkers Ġthat Ġobviously Ġdo Ġnot Ġfollow Ġinstructions . ĠWe Ġremove ĠQ A Ġpairs Ġwith Ġyes / no Ġanswers . ĠQuestions Ġwith Ġless Ġthan Ġfive Ġwords Ġare Ġalso Ġfiltered Ġout . ĠThis Ġprocess Ġfiltered ĠIN LINE FORM 0 Ġof Ġthe ĠQ A Ġpairs . ĠThe Ġdataset Ġnow Ġincludes Ġ10 , 8 98 Ġarticles , Ġ17 , 794 Ġtweets , Ġand Ġ13 , 757 Ġcrowds ourced Ġquestion - answer Ġpairs . ĠThe Ġcollected ĠQ A Ġpairs Ġwill Ġbe Ġdirectly Ġavailable Ġto Ġthe Ġpublic , Ġand Ġwe Ġwill Ġprovide Ġa Ġscript Ġto Ġdownload Ġthe Ġoriginal Ġtweets Ġand Ġdetailed Ġdocumentation Ġon Ġhow Ġwe Ġbuild Ġour Ġdataset . ĠAlso Ġnote Ġthat Ġsince Ġwe Ġkeep Ġthe Ġoriginal Ġnews Ġarticle Ġand Ġnews Ġtitles Ġfor Ġeach Ġtweet , Ġour Ġdataset Ġcan Ġalso Ġbe Ġused Ġto Ġexplore Ġmore Ġchallenging Ġgeneration Ġtasks . ĠTable ĠT AB REF 19 Ġshows Ġthe Ġstatistics Ġof Ġour Ġcurrent Ġcollection , Ġand Ġthe Ġfrequency Ġof Ġdifferent Ġtypes Ġof Ġquestions Ġis Ġshown Ġin ĠTable ĠT AB REF 21 Ġ. ĠAll ĠQ A Ġpairs Ġwere Ġwritten Ġby Ġ49 2 Ġindividual Ġworkers . </s> For Ġthe Ġpurposes Ġof Ġhuman Ġperformance Ġevaluation Ġand Ġinter - annot ator Ġagreement Ġchecking , Ġwe Ġlaunch Ġa Ġdifferent Ġset Ġof ĠHIT s Ġto Ġask Ġworkers Ġto Ġanswer Ġquestions Ġin Ġthe Ġtest Ġand Ġdevelopment Ġset . ĠThe Ġworkers Ġare Ġshown Ġwith Ġthe Ġtweet Ġblocks Ġas Ġwell Ġas Ġthe Ġquestions Ġcollected Ġin Ġthe Ġprevious Ġstep . ĠAt Ġthis Ġstep , Ġworkers Ġare Ġallowed Ġto Ġlabel Ġthe Ġquestions Ġas ĠâĢ ľ NA " Ġif Ġthey Ġthink Ġthe Ġquestions Ġare Ġnot Ġanswer able . ĠWe Ġfind Ġthat ĠIN LINE FORM 0 Ġof Ġthe Ġquestions Ġare Ġlabeled Ġas Ġun answer able Ġby Ġthe Ġworkers Ġ( for ĠS Qu AD , Ġthe Ġratio Ġis ĠIN LINE FORM 1 Ġ). ĠSince Ġthe Ġanswers Ġcollected Ġat Ġthis Ġstep Ġand Ġprevious Ġstep Ġare Ġwritten Ġby Ġdifferent Ġworkers , Ġthe Ġanswers Ġcan Ġbe Ġwritten Ġin Ġdifferent Ġtext Ġforms Ġeven Ġthey Ġare Ġsem antically Ġequal Ġto Ġeach Ġother . ĠFor Ġexample , Ġone Ġanswer Ġcan Ġbe ĠâĢ ľ Hillary ĠClinton âĢ Ŀ Ġwhile Ġthe Ġother Ġis ĠâĢ ľ @ Hillary Clinton âĢ Ŀ . ĠAs Ġit Ġis Ġnot Ġstraightforward Ġto Ġautomatically Ġcalculate Ġthe Ġoverall Ġagreement , Ġwe Ġmanually Ġcheck Ġthe Ġagreement Ġon Ġa Ġsubset Ġof Ġ200 Ġrandom Ġsamples Ġfrom Ġthe Ġdevelopment Ġset Ġand Ġask Ġan Ġindependent Ġhuman Ġmoderator Ġto Ġverify Ġthe Ġresult . ĠIt Ġturns Ġout Ġthat ĠIN LINE FORM 2 Ġof Ġthe Ġanswers Ġpairs Ġare Ġsem antically Ġequivalent , ĠIN LINE FORM 3 Ġof Ġthem Ġare Ġpartially Ġequivalent Ġ( one Ġof Ġthem Ġis Ġincomplete ) Ġand ĠIN LINE FORM 4 Ġare Ġtotally Ġinconsistent . ĠThe Ġanswers Ġcollected Ġat Ġthis Ġstep Ġare Ġalso Ġused Ġto Ġmeasure Ġthe Ġhuman Ġperformance . ĠWe Ġhave Ġ59 Ġindividual Ġworkers Ġparticipated Ġin Ġthis Ġprocess . </s> Task Ġand ĠEvaluation </s> As Ġdescribed Ġin Ġthe Ġquestion - answer Ġwriting Ġprocess , Ġthe Ġanswers Ġin Ġour Ġdataset Ġare Ġdifferent Ġfrom Ġthose Ġin Ġsome Ġexisting Ġextract ive Ġdatasets . ĠThus Ġwe Ġconsider Ġthe Ġtask Ġof Ġanswer Ġgeneration Ġfor ĠTweet Q A Ġand Ġwe Ġuse Ġseveral Ġstandard Ġmetrics Ġfor Ġnatural Ġlanguage Ġgeneration Ġto Ġevaluate ĠQ A Ġsystems Ġon Ġour Ġdataset , Ġnamely Ġwe Ġconsider ĠB LE U - 1 ĠB IB REF 16 Ġ, ĠMeteor ĠB IB REF 17 Ġand ĠRouge - L ĠB IB REF 18 Ġin Ġthis Ġpaper . </s> To Ġevaluate Ġmachine Ġsystems , Ġwe Ġcompute Ġthe Ġscores Ġusing Ġboth Ġthe Ġoriginal Ġanswer Ġand Ġvalidation Ġanswer Ġas Ġreferences . ĠFor Ġhuman Ġperformance , Ġwe Ġuse Ġthe Ġvalidation Ġanswers Ġas Ġgenerated Ġones Ġand Ġthe Ġoriginal Ġanswers Ġas Ġreferences Ġto Ġcalculate Ġthe Ġscores . </s> Analysis </s> In Ġthis Ġsection , Ġwe Ġanalyze Ġour Ġdataset Ġand Ġoutline Ġthe Ġkey Ġproperties Ġthat Ġdistinguish Ġit Ġfrom Ġstandard ĠQ A Ġdatasets Ġlike ĠS Qu AD ĠB IB REF 6 Ġ. ĠFirst , Ġour Ġdataset Ġis Ġderived Ġfrom Ġsocial Ġmedia Ġtext Ġwhich Ġcan Ġbe Ġquite Ġinformal Ġand Ġuser - centric Ġas Ġopposed Ġto ĠS Qu AD Ġwhich Ġis Ġderived Ġfrom ĠWikipedia Ġand Ġhence Ġmore Ġformal Ġin Ġnature . ĠWe Ġobserve Ġthat Ġthe Ġshared Ġvocabulary Ġbetween ĠS Qu AD Ġand ĠTweet Q A Ġis Ġonly ĠIN LINE FORM 0 Ġ, Ġsuggesting Ġa Ġsignificant Ġdifference Ġin Ġtheir Ġlex ical Ġcontent . ĠFigure ĠFIG REF 25 Ġshows Ġthe Ġ1000 Ġmost Ġdistinctive Ġwords Ġin Ġeach Ġdomain Ġas Ġextracted Ġfrom ĠS Qu AD Ġand ĠTweet Q A . ĠNote Ġthe Ġstark Ġdifferences Ġin Ġthe Ġwords Ġseen Ġin Ġthe ĠTweet Q A Ġdataset , Ġwhich Ġinclude Ġa Ġlarge Ġnumber Ġof Ġuser Ġaccounts Ġwith Ġa Ġheavy Ġtail . ĠExamples Ġinclude Ġ@ re ald on ald trump , Ġ@ j ds utter , Ġ@ just ink irk land Ġand Ġ# c nn world cup , Ġ# gold eng l ob es . ĠIn Ġcontrast , Ġthe ĠS Qu AD Ġdataset Ġrarely Ġhas Ġus ern ames Ġor Ġhasht ags Ġthat Ġare Ġused Ġto Ġsignify Ġevents Ġor Ġrefer Ġto Ġthe Ġauthors . ĠIt Ġis Ġalso Ġworth Ġnoting Ġthat Ġthe Ġdata Ġcollected Ġfrom Ġsocial Ġmedia Ġcan Ġnot Ġonly Ġcapture Ġevents Ġand Ġdevelopments Ġin Ġreal - time Ġbut Ġalso Ġcapture Ġindividual Ġopinions Ġand Ġthus Ġrequires Ġreasoning Ġrelated Ġto Ġthe Ġauthors hip Ġof Ġthe Ġcontent Ġas Ġis Ġillustrated Ġin ĠTable ĠT AB REF 3 Ġ. ĠIn Ġaddition , Ġwhile ĠS Qu AD Ġrequires Ġall Ġanswers Ġto Ġbe Ġspans Ġfrom Ġthe Ġgiven Ġpassage , Ġwe Ġdo Ġnot Ġenforce Ġany Ġsuch Ġrestriction Ġand Ġanswers Ġcan Ġbe Ġfree - form Ġtext . ĠIn Ġfact , Ġwe Ġobserved Ġthat ĠIN LINE FORM 1 Ġof Ġour ĠQ A Ġpairs Ġconsists Ġof Ġanswers Ġwhich Ġdo Ġnot Ġhave Ġan Ġexact Ġsubst ring Ġmatching Ġwith Ġtheir Ġcorresponding Ġpassages . ĠAll Ġof Ġthe Ġabove Ġdistinguishing Ġfactors Ġhave Ġimplications Ġto Ġexisting Ġmodels Ġwhich Ġwe Ġanalyze Ġin Ġupcoming Ġsections . </s> We Ġconduct Ġanalysis Ġon Ġa Ġsubset Ġof ĠTweet Q A Ġto Ġget Ġa Ġbetter Ġunderstanding Ġof Ġthe Ġkind Ġof Ġreasoning Ġskills Ġthat Ġare Ġrequired Ġto Ġanswer Ġthese Ġquestions . ĠWe Ġsample Ġ150 Ġquestions Ġfrom Ġthe Ġdevelopment Ġset , Ġthen Ġmanually Ġlabel Ġtheir Ġreasoning Ġcategories . ĠTable ĠT AB REF 26 Ġshows Ġthe Ġanalysis Ġresults . ĠWe Ġuse Ġsome Ġof Ġthe Ġcategories Ġin ĠS Qu AD ĠB IB REF 6 Ġand Ġalso Ġproposes Ġsome Ġtweet - specific Ġreasoning Ġtypes . </s> Our Ġfirst Ġobservation Ġis Ġthat Ġalmost Ġhalf Ġof Ġthe Ġquestions Ġonly Ġrequire Ġthe Ġability Ġto Ġidentify Ġparaph r ases . ĠAlthough Ġmost Ġof Ġthe ĠâĢ ľ par aph r asing Ġonly âĢ Ŀ Ġquestions Ġare Ġconsidered Ġas Ġfairly Ġeasy Ġquestions , Ġwe Ġfind Ġthat Ġa Ġsignificant Ġamount Ġ( about Ġ3 / 4 ) Ġof Ġthese Ġquestions Ġare Ġasked Ġabout Ġevent - related Ġtopics , Ġsuch Ġas Ġinformation Ġabout ĠâĢ ľ who Ġdid Ġwhat Ġto Ġwhom , Ġwhen Ġand Ġwhere âĢ Ŀ . ĠThis Ġis Ġactually Ġconsistent Ġwith Ġour Ġmotivation Ġto Ġcreate ĠTweet Q A , Ġas Ġwe Ġexpect Ġthis Ġdataset Ġcould Ġbe Ġused Ġto Ġdevelop Ġsystems Ġthat Ġautomatically Ġcollect Ġinformation Ġabout Ġreal - time Ġevents . </s> Apart Ġfrom Ġthese Ġquestions , Ġthere Ġare Ġalso Ġa Ġgroup Ġof Ġquestions Ġthat Ġrequire Ġunderstanding Ġcommon Ġsense , Ġdeep Ġsemantics Ġ( i . e . Ġthe Ġanswers Ġcannot Ġbe Ġderived Ġfrom Ġthe Ġliteral Ġmeanings Ġof Ġthe Ġtweets ), Ġand Ġrelations Ġof Ġsentences Ġ( including Ġco - reference Ġresolution ), Ġwhich Ġare Ġalso Ġappeared Ġin Ġother ĠRC Ġdatasets ĠB IB REF 6 Ġ. ĠOn Ġthe Ġother Ġhand , Ġthe ĠTweet Q A Ġalso Ġhas Ġits Ġunique Ġproperties . ĠSpecifically , Ġa Ġsignificant Ġamount Ġof Ġquestions Ġrequire Ġcertain Ġreasoning Ġskills Ġthat Ġare Ġspecific Ġto Ġsocial Ġmedia Ġdata : </s> [ no item se p ] </s> Understanding Ġauthors hip : ĠSince Ġtweets Ġare Ġhighly Ġpersonal , Ġit Ġis Ġcritical Ġto Ġunderstand Ġhow Ġquestions / t we ets Ġrelated Ġto Ġthe Ġauthors . </s> O ral ĠEnglish Ġ& ĠTweet ĠEnglish : ĠTwe ets Ġare Ġoften Ġoral Ġand Ġinformal . ĠQ A Ġover Ġtweets Ġrequires Ġthe Ġunderstanding Ġof Ġcommon Ġoral ĠEnglish . ĠOur ĠTweet Q A Ġalso Ġrequires Ġunderstanding Ġsome Ġtweet - specific ĠEnglish , Ġlike Ġconversation - style ĠEnglish . </s> Understanding Ġof Ġuser ĠIDs Ġ& Ġhasht ags : ĠTwe ets Ġoften Ġcontains Ġuser ĠIDs Ġand Ġhasht ags , Ġwhich Ġare Ġsingle Ġspecial Ġtokens . ĠUnderstanding Ġthese Ġspecial Ġtokens Ġis Ġimportant Ġto Ġanswer Ġperson - Ġor Ġevent - related Ġquestions . </s> Exper iments </s> To Ġshow Ġthe Ġchallenge Ġof ĠTweet Q A Ġfor Ġexisting Ġapproaches , Ġwe Ġconsider Ġfour Ġrepresentative Ġmethods Ġas Ġbas elines . ĠFor Ġdata Ġprocessing , Ġwe Ġfirst Ġremove Ġthe ĠURLs Ġin Ġthe Ġtweets Ġand Ġthen Ġtoken ize Ġthe ĠQ A Ġpairs Ġand Ġtweets Ġusing ĠNL T K . ĠThis Ġprocess Ġis Ġconsistent Ġfor Ġall Ġbas elines . </s> Query ĠMatch ing ĠBas eline </s> We Ġfirst Ġconsider Ġa Ġsimple Ġquery Ġmatching Ġbaseline Ġsimilar Ġto Ġthe ĠIR Ġbaseline Ġin ĠK oc isk 2017 The NR . ĠBut Ġinstead Ġof Ġonly Ġconsidering Ġseveral Ġgenres Ġof Ġspans Ġas Ġpotential Ġanswers , Ġwe Ġtry Ġto Ġmatch Ġthe Ġquestion Ġwith Ġall Ġpossible Ġspans Ġin Ġthe Ġtweet Ġcontext Ġand Ġchoose Ġthe Ġspan Ġwith Ġthe Ġhighest ĠB LE U - 1 Ġscore Ġas Ġthe Ġfinal Ġanswer , Ġwhich Ġfollows Ġthe Ġmethod Ġand Ġimplementation Ġof Ġanswer Ġspan Ġselection Ġfor Ġopen - domain ĠQ A ĠB IB REF 19 Ġ. ĠWe Ġinclude Ġthis Ġbaseline Ġto Ġshow Ġthat ĠTweet Q A Ġis Ġa Ġnont riv ial Ġtask Ġwhich Ġcannot Ġbe Ġeasily Ġsolved Ġwith Ġsuperficial Ġtext Ġmatching . </s> Ne ural ĠBas elines </s> We Ġthen Ġexplore Ġthree Ġtypical Ġneural Ġmodels Ġthat Ġperform Ġwell Ġon Ġexisting Ġformal - text Ġdatasets . ĠOne Ġtakes Ġa Ġgener ative Ġperspective Ġand Ġlearns Ġto Ġdecode Ġthe Ġanswer Ġconditioned Ġon Ġthe Ġquestion Ġand Ġcontext , Ġwhile Ġthe Ġothers Ġlearns Ġto Ġextract Ġa Ġtext Ġspan Ġfrom Ġthe Ġcontext Ġthat Ġbest Ġanswers Ġthe Ġquestion . </s> R NN - based Ġenc oder - dec oder Ġmodels ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġhave Ġbeen Ġwidely Ġused Ġfor Ġnatural Ġlanguage Ġgeneration Ġtasks . ĠHere Ġwe Ġconsider Ġa Ġrecently Ġproposed Ġgener ative Ġmodel ĠB IB REF 22 Ġthat Ġfirst Ġenc odes Ġthe Ġcontext Ġand Ġquestion Ġinto Ġa Ġmulti - pers pect ive Ġmemory Ġvia Ġfour Ġdifferent Ġneural Ġmatching Ġlayers , Ġthen Ġdec odes Ġthe Ġanswer Ġusing Ġan Ġattention - based Ġmodel Ġequipped Ġwith Ġboth Ġcopy Ġand Ġcoverage Ġmechanisms . ĠThe Ġmodel Ġis Ġtrained Ġon Ġour Ġdataset Ġfor Ġ15 Ġepoch s Ġand Ġwe Ġchoose Ġthe Ġmodel Ġparameters Ġthat Ġachieve Ġthe Ġbest ĠB LE U - 1 Ġscore Ġon Ġthe Ġdevelopment Ġset . </s> Unlike Ġthe Ġaforementioned Ġgener ative Ġmodel , Ġthe ĠBi - D irection al ĠAttention ĠFlow Ġ( Bi DA F ) ĠB IB REF 23 Ġnetwork Ġlearns Ġto Ġdirectly Ġpredict Ġthe Ġanswer Ġspan Ġin Ġthe Ġcontext . ĠBi DA F Ġfirst Ġutilizes Ġmulti - level Ġembed ding Ġlayers Ġto Ġencode Ġboth Ġthe Ġquestion Ġand Ġcontext , Ġthen Ġuses Ġbi - direction al Ġattention Ġflow Ġto Ġget Ġa Ġquery - aware Ġcontext Ġrepresentation , Ġwhich Ġis Ġfurther Ġmodeled Ġby Ġan ĠR NN Ġlayer Ġto Ġmake Ġthe Ġspan Ġpredictions . ĠSince Ġour ĠTweet Q A Ġdoes Ġnot Ġhave Ġlabeled Ġanswer Ġspans Ġas Ġin ĠS Qu AD , Ġwe Ġneed Ġto Ġuse Ġthe Ġhuman - written Ġanswers Ġto Ġretrieve Ġthe Ġanswer - span Ġlabels Ġfor Ġtraining . ĠTo Ġget Ġthe Ġapproximate Ġanswer Ġspans , Ġwe Ġconsider Ġthe Ġsame Ġmatching Ġapproach Ġas Ġin Ġthe Ġquery Ġmatching Ġbaseline . ĠBut Ġinstead Ġof Ġusing Ġquestions Ġto Ġdo Ġmatching , Ġwe Ġuse Ġthe Ġhuman - written Ġanswers Ġto Ġget Ġthe Ġspans Ġthat Ġachieve Ġthe Ġbest ĠB LE U - 1 Ġscores . </s> This Ġis Ġanother Ġextract ive ĠRC Ġmodel Ġthat Ġbenefits Ġfrom Ġthe Ġrecent Ġadvance Ġin Ġpret rained Ġgeneral Ġlanguage Ġenc od ers ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ. ĠIn Ġour Ġwork , Ġwe Ġselect Ġthe ĠB ERT Ġmodel ĠB IB REF 25 Ġwhich Ġhas Ġachieved Ġthe Ġbest Ġperformance Ġon ĠS Qu AD . ĠIn Ġour Ġexperiments , Ġwe Ġuse Ġthe ĠPy Tor ch Ġreim plementation Ġof Ġthe Ġunc ased Ġbase Ġmodel . ĠThe Ġbatch Ġsize Ġis Ġset Ġas Ġ12 Ġand Ġwe Ġfine - t une Ġthe Ġmodel Ġfor Ġ2 Ġepoch s Ġwith Ġlearning Ġrate Ġ3 e - 5 . </s> Overall ĠPerformance </s> We Ġtest Ġthe Ġperformance Ġof Ġall Ġbaseline Ġsystems Ġusing Ġthe Ġthree Ġgener ative Ġmetrics Ġmentioned Ġin ĠSection ĠSEC REF 22 Ġ. ĠAs Ġshown Ġin ĠTable ĠT AB REF 40 Ġ, Ġthere Ġis Ġa Ġlarge Ġperformance Ġgap Ġbetween Ġhuman Ġperformance Ġand Ġall Ġbaseline Ġmethods , Ġincluding ĠB ERT , Ġwhich Ġhas Ġachieved Ġsuperhuman Ġperformance Ġon ĠS Qu AD . ĠThis Ġconfirms Ġthan ĠTweet Q A Ġis Ġmore Ġchallenging Ġthan Ġformal - test ĠRC Ġtasks . </s> We Ġalso Ġshow Ġthe Ġupper Ġbound Ġof Ġthe Ġextract ive Ġmodels Ġ( den oted Ġas ĠExtract - U pper ). ĠIn Ġthe Ġupper Ġbound Ġmethod , Ġthe Ġanswers Ġare Ġdefined Ġas Ġn - gram s Ġfrom Ġthe Ġtweets Ġthat Ġmaximize Ġthe ĠB LE U - 1 / MET E OR / R OU GE - L Ġcompared Ġto Ġthe Ġannot ated Ġground truth . ĠFrom Ġthe Ġresults , Ġwe Ġcan Ġsee Ġthat Ġthe ĠB ERT Ġmodel Ġstill Ġl ags Ġbehind Ġthe Ġupper Ġbound Ġsignificantly , Ġshowing Ġgreat Ġpotential Ġfor Ġfuture Ġresearch . ĠIt Ġis Ġalso Ġinteresting Ġto Ġsee Ġthat Ġthe ĠHuman Ġperformance Ġis Ġslightly Ġworse Ġcompared Ġto Ġthe Ġupper Ġbound . ĠThis Ġindicates Ġ( 1 ) Ġthe Ġdifficulty Ġof Ġour Ġproblem Ġalso Ġexists Ġfor Ġhuman - be ings Ġand Ġ( 2 ) Ġfor Ġthe Ġanswer Ġverification Ġprocess , Ġthe Ġworkers Ġtend Ġto Ġalso Ġextract Ġtexts Ġfrom Ġtweets Ġas Ġanswers . </s> According Ġto Ġthe Ġcomparison Ġbetween Ġthe Ġtwo Ġnon - pret raining Ġbas elines , Ġour Ġgener ative Ġbaseline Ġyields Ġbetter Ġresults Ġthan ĠBi DA F . ĠWe Ġbelieve Ġthis Ġis Ġlargely Ġdue Ġto Ġthe Ġabstract ive Ġnature Ġof Ġour Ġdataset , Ġsince Ġthe Ġworkers Ġcan Ġsometimes Ġwrite Ġthe Ġanswers Ġusing Ġtheir Ġown Ġwords . </s> Performance ĠAnalysis Ġover ĠHuman - Label ed ĠQuestion ĠTypes </s> To Ġbetter Ġunderstand Ġthe Ġdifficulty Ġof Ġthe ĠTweet Q A Ġtask Ġfor Ġcurrent Ġneural Ġmodels , Ġwe Ġanalyze Ġthe Ġdecomp osed Ġmodel Ġperformance Ġon Ġthe Ġdifferent Ġkinds Ġof Ġquestions Ġthat Ġrequire Ġdifferent Ġtypes Ġof Ġreasoning Ġ( we Ġtested Ġon Ġthe Ġsubset Ġwhich Ġhas Ġbeen Ġused Ġfor Ġthe Ġanalysis Ġin ĠTable ĠT AB REF 26 Ġ). ĠTable ĠT AB REF 42 Ġshows Ġthe Ġresults Ġof Ġthe Ġbest Ġperformed Ġnon - pret raining Ġand Ġpret raining Ġapproach , Ġi . e ., Ġthe Ġgener ative ĠQ A Ġbaseline Ġand Ġthe Ġfine - tun ed ĠB ERT . ĠOur Ġfull Ġcomparison Ġincluding Ġthe ĠBi DA F Ġperformance Ġand Ġevaluation Ġon Ġmore Ġmetrics Ġcan Ġbe Ġfound Ġin ĠAppendix ĠSEC REF 7 Ġ. ĠFollowing Ġprevious ĠRC Ġresearch , Ġwe Ġalso Ġinclude Ġanalysis Ġon Ġautomatically - label ed Ġquestion Ġtypes Ġin ĠAppendix ĠSEC REF 8 Ġ. </s> As Ġindicated Ġby Ġthe Ġresults Ġon ĠMET E OR Ġand ĠR OU GE - L Ġ( also Ġindicated Ġby Ġa Ġthird Ġmetric , ĠB LE U - 1 , Ġas Ġshown Ġin ĠAppendix ĠSEC REF 7 Ġ), Ġboth Ġbas elines Ġperform Ġworse Ġon Ġquestions Ġthat Ġrequire Ġthe Ġunderstanding Ġdeep Ġsemantics Ġand Ġuser ID & hash tags . ĠThe Ġformer Ġkind Ġof Ġquestions Ġalso Ġappear Ġin Ġother Ġbenchmarks Ġand Ġis Ġknown Ġto Ġbe Ġchallenging Ġfor Ġmany Ġcurrent Ġmodels . ĠThe Ġsecond Ġkind Ġof Ġquestions Ġis Ġtweet - specific Ġand Ġis Ġrelated Ġto Ġspecific Ġproperties Ġof Ġsocial Ġmedia Ġdata . ĠSince Ġboth Ġmodels Ġare Ġdesigned Ġfor Ġformal - text Ġpassages Ġand Ġthere Ġis Ġno Ġspecial Ġtreatment Ġfor Ġunderstanding Ġuser ĠIDs Ġand Ġhasht ags , Ġthe Ġperformance Ġis Ġseverely Ġlimited Ġon Ġthe Ġquestions Ġrequiring Ġsuch Ġreasoning Ġabilities . ĠWe Ġbelieve Ġthat Ġgood Ġsegment ation , Ġdis amb ig uation Ġand Ġlinking Ġtools Ġdeveloped Ġby Ġthe Ġsocial Ġmedia Ġcommunity Ġfor Ġprocessing Ġthe Ġuser IDs Ġand Ġhasht ags Ġwill Ġsignificantly Ġhelp Ġthese Ġquestion Ġtypes . </s> Besides Ġthe Ġeasy Ġquestions Ġrequiring Ġmainly Ġparaph r asing Ġskill , Ġwe Ġalso Ġfind Ġthat Ġthe Ġquestions Ġrequiring Ġthe Ġunderstanding Ġof Ġauthors hip Ġand Ġoral / t weet ĠEnglish Ġhabits Ġare Ġnot Ġvery Ġdifficult . ĠWe Ġthink Ġthis Ġis Ġdue Ġto Ġthe Ġreason Ġthat , Ġexcept Ġfor Ġthese Ġtweet - specific Ġtokens , Ġthe Ġrest Ġparts Ġof Ġthe Ġquestions Ġare Ġrather Ġsimple , Ġwhich Ġmay Ġrequire Ġonly Ġsimple Ġreasoning Ġskill Ġ( e . g . Ġparaph r asing ). </s> Although ĠB ERT Ġwas Ġdemonstrated Ġto Ġbe Ġa Ġpowerful Ġtool Ġfor Ġreading Ġcomprehension , Ġthis Ġis Ġthe Ġfirst Ġtime Ġa Ġdetailed Ġanalysis Ġhas Ġbeen Ġdone Ġon Ġits Ġreasoning Ġskills . ĠFrom Ġthe Ġresults , Ġthe Ġhuge Ġimprovement Ġof ĠB ERT Ġmainly Ġcomes Ġfrom Ġtwo Ġtypes . ĠThe Ġfirst Ġis Ġparaph r asing , Ġwhich Ġis Ġnot Ġsurprising Ġbecause Ġthat Ġa Ġwell Ġpret rained Ġlanguage Ġmodel Ġis Ġexpected Ġto Ġbe Ġable Ġto Ġbetter Ġencode Ġsentences . ĠThus Ġthe Ġderived Ġembed ding Ġspace Ġcould Ġwork Ġbetter Ġfor Ġsentence Ġcomparison . ĠThe Ġsecond Ġtype Ġis Ġcommons ense , Ġwhich Ġis Ġconsistent Ġwith Ġthe Ġgood Ġperformance Ġof ĠB ERT ĠB IB REF 25 Ġon ĠSW AG ĠB IB REF 26 Ġ. ĠWe Ġbelieve Ġthat Ġthis Ġprovides Ġfurther Ġevidence Ġabout Ġthe Ġconnection Ġbetween Ġlarge - sc aled Ġdeep Ġneural Ġlanguage Ġmodel Ġand Ġcertain Ġkinds Ġof Ġcommons ense . </s> Conclusion </s> We Ġpresent Ġthe Ġfirst Ġdataset Ġfor ĠQ A Ġon Ġsocial Ġmedia Ġdata Ġby Ġleveraging Ġnews Ġmedia Ġand Ġcrowds ourcing . ĠThe Ġproposed Ġdataset Ġinforms Ġus Ġof Ġthe Ġdistinct iveness Ġof Ġsocial Ġmedia Ġfrom Ġformal Ġdomains Ġin Ġthe Ġcontext Ġof ĠQ A . ĠSpecifically , Ġwe Ġfind Ġthat ĠQ A Ġon Ġsocial Ġmedia Ġrequires Ġsystems Ġto Ġcomprehend Ġsocial Ġmedia Ġspecific Ġlinguistic Ġpatterns Ġlike Ġinform ality , Ġhasht ags , Ġus ern ames , Ġand Ġauthors hip . ĠThese Ġdistinguishing Ġlinguistic Ġfactors Ġbring Ġup Ġimportant Ġproblems Ġfor Ġthe Ġresearch Ġof ĠQ A Ġthat Ġcurrently Ġfocuses Ġon Ġformal Ġtext . ĠWe Ġsee Ġour Ġdataset Ġas Ġa Ġfirst Ġstep Ġtowards Ġenabling Ġnot Ġonly Ġa Ġdeeper Ġunderstanding Ġof Ġnatural Ġlanguage Ġin Ġsocial Ġmedia Ġbut Ġalso Ġrich Ġapplications Ġthat Ġcan Ġextract Ġessential Ġreal - time Ġknowledge Ġfrom Ġsocial Ġmedia . </s> Full Ġresults Ġof ĠPerformance ĠAnalysis Ġover ĠHuman - Label ed ĠQuestion ĠTypes </s> Table ĠT AB REF 45 Ġgives Ġour Ġfull Ġevaluation Ġon Ġhuman Ġannot ated Ġquestion Ġtypes . </s> Compared Ġwith Ġthe ĠBi DA F Ġmodel , Ġone Ġinteresting Ġobservation Ġis Ġthat Ġthe Ġgener ative Ġbaseline Ġgets Ġmuch Ġworse Ġresults Ġon Ġambiguous Ġquestions . ĠWe Ġconjecture Ġthat Ġalthough Ġthese Ġquestions Ġare Ġmeaningless , Ġthey Ġstill Ġhave Ġmany Ġwords Ġthat Ġoverl apped Ġwith Ġthe Ġcontexts . ĠThis Ġcan Ġgive ĠBi DA F Ġpotential Ġadvantage Ġover Ġthe Ġgener ative Ġbaseline . </s> Performance ĠAnalysis Ġover ĠAutom atically - Label ed ĠQuestion ĠTypes </s> Besides Ġthe Ġanalysis Ġon Ġdifferent Ġreasoning Ġtypes , Ġwe Ġalso Ġlook Ġinto Ġthe Ġperformance Ġover Ġquestions Ġwith Ġdifferent Ġfirst Ġtokens Ġin Ġthe Ġdevelopment Ġset , Ġwhich Ġprovide Ġus Ġan Ġautomatic Ġcategor ization Ġof Ġquestions . ĠAccording Ġto Ġthe Ġresults Ġin ĠTable ĠT AB REF 46 Ġ, Ġthe Ġthree Ġneural Ġbas elines Ġall Ġperform Ġthe Ġbest Ġon ĠâĢ ľ Who âĢ Ŀ Ġand ĠâĢ ľ Where âĢ Ŀ Ġquestions , Ġto Ġwhich Ġthe Ġanswers Ġare Ġoften Ġnamed Ġentities . ĠSince Ġthe Ġtweet Ġcontexts Ġare Ġshort , Ġthere Ġare Ġonly Ġa Ġsmall Ġnumber Ġof Ġnamed Ġentities Ġto Ġchoose Ġfrom , Ġwhich Ġcould Ġmake Ġthe Ġanswer Ġpattern Ġeasy Ġto Ġlearn . ĠOn Ġthe Ġother Ġhand , Ġthe Ġneural Ġmodels Ġfail Ġto Ġperform Ġwell Ġon Ġthe ĠâĢ ľ Why âĢ Ŀ Ġquestions , Ġand Ġthe Ġresults Ġof Ġneural Ġbas elines Ġare Ġeven Ġworse Ġthan Ġthat Ġof Ġthe Ġmatching Ġbaseline . ĠWe Ġfind Ġthat Ġthese Ġquestions Ġgenerally Ġhave Ġlonger Ġanswer Ġphrases Ġthan Ġother Ġtypes Ġof Ġquestions , Ġwith Ġthe Ġaverage Ġanswer Ġlength Ġbeing Ġ3 . 74 Ġcompared Ġto Ġ2 . 13 Ġfor Ġany Ġother Ġtypes . ĠAlso , Ġsince Ġall Ġthe Ġanswers Ġare Ġwritten Ġby Ġhumans Ġinstead Ġof Ġjust Ġspans Ġfrom Ġthe Ġcontext , Ġthese Ġabstract ive Ġanswers Ġcan Ġmake Ġit Ġeven Ġharder Ġfor Ġcurrent Ġmodels Ġto Ġhandle . ĠWe Ġalso Ġobserve Ġthat Ġwhen Ġpeople Ġwrite ĠâĢ ľ Why âĢ Ŀ Ġquestions , Ġthey Ġtend Ġto Ġcopy Ġword Ġspans Ġfrom Ġthe Ġtweet , Ġpotentially Ġmaking Ġthe Ġtask Ġeasier Ġfor Ġthe Ġmatching Ġbaseline .
Introduction Social media is now becoming an important real-time information source, especially during natural disasters and emergencies. It is now very common for traditional news media to frequently probe users and resort to social media platforms to obtain real-time developments of events. According to a recent survey by Pew Research Center, in 2017, more than two-thirds of Americans read some of their news on social media. Even for American people who are 50 or older, INLINEFORM0 of them report getting news from social media, which is INLINEFORM1 points higher than the number in 2016. Among all major social media sites, Twitter is most frequently used as a news source, with INLINEFORM2 of its users obtaining their news from Twitter. All these statistical facts suggest that understanding user-generated noisy social media text from Twitter is a significant task. In recent years, while several tools for core natural language understanding tasks involving syntactic and semantic analysis have been developed for noisy social media text BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , there is little work on question answering or reading comprehension over social media, with the primary bottleneck being the lack of available datasets. We observe that recently proposed QA datasets usually focus on formal domains, e.g. CNN/DailyMail BIBREF4 and NewsQA BIBREF5 on news articles; SQuAD BIBREF6 and WikiMovies BIBREF7 that use Wikipedia. In this paper, we propose the first large-scale dataset for QA over social media data. Rather than naively obtaining tweets from Twitter using the Twitter API which can yield irrelevant tweets with no valuable information, we restrict ourselves only to tweets which have been used by journalists in news articles thus implicitly implying that such tweets contain useful and relevant information. To obtain such relevant tweets, we crawled thousands of news articles that include tweet quotations and then employed crowd-sourcing to elicit questions and answers based on these event-aligned tweets. Table TABREF3 gives an example from our TweetQA dataset. It shows that QA over tweets raises challenges not only because of the informal nature of oral-style texts (e.g. inferring the answer from multiple short sentences, like the phrase “so young” that forms an independent sentence in the example), but also from tweet-specific expressions (such as inferring that it is “Jay Sean” feeling sad about Paul's death because he posted the tweet). Furthermore, we show the distinctive nature of TweetQA by comparing the collected data with traditional QA datasets collected primarily from formal domains. In particular, we demonstrate empirically that three strong neural models which achieve good performance on formal data do not generalize well to social media data, bringing out challenges to developing QA systems that work well on social media domains. In summary, our contributions are: TweetQA In this section, we first describe the three-step data collection process of TweetQA: tweet crawling, question-answer writing and answer validation. Next, we define the specific task of TweetQA and discuss several evaluation metrics. To better understand the characteristics of the TweetQA task, we also include our analysis on the answer and question characteristics using a subset of QA pairs from the development set. Data Collection One major challenge of building a QA dataset on tweets is the sparsity of informative tweets. Many users write tweets to express their feelings or emotions about their personal lives. These tweets are generally uninformative and also very difficult to ask questions about. Given the linguistic variance of tweets, it is generally hard to directly distinguish those tweets from informative ones. In terms of this, rather than starting from Twitter API Search, we look into the archived snapshots of two major news websites (CNN, NBC), and then extract the tweet blocks that are embedded in the news articles. In order to get enough data, we first extract the URLs of all section pages (e.g. World, Politics, Money, Tech) from the snapshot of each home page and then crawl all articles with tweets from these section pages. Note that another possible way to collect informative tweets is to download the tweets that are posted by the official Twitter accounts of news media. However, these tweets are often just the summaries of news articles, which are written in formal text. As our focus is to develop a dataset for QA on informal social media text, we do not consider this approach. After we extracted tweets from archived news articles, we observed that there is still a portion of tweets that have very simple semantic structures and thus are very difficult to raise meaningful questions. An example of such tweets can be like: “Wanted to share this today - @IAmSteveHarvey". This tweet is actually talking about an image attached to this tweet. Some other tweets with simple text structures may talk about an inserted link or even videos. To filter out these tweets that heavily rely on attached media to convey information, we utilize a state-of-the-art semantic role labeling model trained on CoNLL-2005 BIBREF15 to analyze the predicate-argument structure of the tweets collected from news articles and keep only the tweets with more than two labeled arguments. This filtering process also automatically filters out most of the short tweets. For the tweets collected from CNN, INLINEFORM0 of them were filtered via semantic role labeling. For tweets from NBC, INLINEFORM1 of the tweets were filtered. We then use Amazon Mechanical Turk to collect question-answer pairs for the filtered tweets. For each Human Intelligence Task (HIT), we ask the worker to read three tweets and write two question-answer pairs for each tweet. To ensure the quality, we require the workers to be located in major English speaking countries (i.e. Canada, US, and UK) and have an acceptance rate larger than INLINEFORM0 . Since we use tweets as context, lots of important information are contained in hashtags or even emojis. Instead of only showing the text to the workers, we use javascript to directly embed the whole tweet into each HIT. This gives workers the same experience as reading tweets via web browsers and help them to better compose questions. To avoid trivial questions that can be simply answered by superficial text matching methods or too challenging questions that require background knowledge. We explicitly state the following items in the HIT instructions for question writing: No Yes-no questions should be asked. The question should have at least five words. Videos, images or inserted links should not be considered. No background knowledge should be required to answer the question. To help the workers better follow the instructions, we also include a representative example showing both good and bad questions or answers in our instructions. Figure FIGREF14 shows the example we use to guide the workers. As for the answers, since the context we consider is relatively shorter than the context of previous datasets, we do not restrict the answers to be in the tweet, otherwise, the task may potentially be simplified as a classification problem. The workers are allowed to write their answers in their own words. We just require the answers to be brief and can be directly inferred from the tweets. After we retrieve the QA pairs from all HITs, we conduct further post-filtering to filter out the pairs from workers that obviously do not follow instructions. We remove QA pairs with yes/no answers. Questions with less than five words are also filtered out. This process filtered INLINEFORM0 of the QA pairs. The dataset now includes 10,898 articles, 17,794 tweets, and 13,757 crowdsourced question-answer pairs. The collected QA pairs will be directly available to the public, and we will provide a script to download the original tweets and detailed documentation on how we build our dataset. Also note that since we keep the original news article and news titles for each tweet, our dataset can also be used to explore more challenging generation tasks. Table TABREF19 shows the statistics of our current collection, and the frequency of different types of questions is shown in Table TABREF21 . All QA pairs were written by 492 individual workers. For the purposes of human performance evaluation and inter-annotator agreement checking, we launch a different set of HITs to ask workers to answer questions in the test and development set. The workers are shown with the tweet blocks as well as the questions collected in the previous step. At this step, workers are allowed to label the questions as “NA" if they think the questions are not answerable. We find that INLINEFORM0 of the questions are labeled as unanswerable by the workers (for SQuAD, the ratio is INLINEFORM1 ). Since the answers collected at this step and previous step are written by different workers, the answers can be written in different text forms even they are semantically equal to each other. For example, one answer can be “Hillary Clinton” while the other is “@HillaryClinton”. As it is not straightforward to automatically calculate the overall agreement, we manually check the agreement on a subset of 200 random samples from the development set and ask an independent human moderator to verify the result. It turns out that INLINEFORM2 of the answers pairs are semantically equivalent, INLINEFORM3 of them are partially equivalent (one of them is incomplete) and INLINEFORM4 are totally inconsistent. The answers collected at this step are also used to measure the human performance. We have 59 individual workers participated in this process. Task and Evaluation As described in the question-answer writing process, the answers in our dataset are different from those in some existing extractive datasets. Thus we consider the task of answer generation for TweetQA and we use several standard metrics for natural language generation to evaluate QA systems on our dataset, namely we consider BLEU-1 BIBREF16 , Meteor BIBREF17 and Rouge-L BIBREF18 in this paper. To evaluate machine systems, we compute the scores using both the original answer and validation answer as references. For human performance, we use the validation answers as generated ones and the original answers as references to calculate the scores. Analysis In this section, we analyze our dataset and outline the key properties that distinguish it from standard QA datasets like SQuAD BIBREF6 . First, our dataset is derived from social media text which can be quite informal and user-centric as opposed to SQuAD which is derived from Wikipedia and hence more formal in nature. We observe that the shared vocabulary between SQuAD and TweetQA is only INLINEFORM0 , suggesting a significant difference in their lexical content. Figure FIGREF25 shows the 1000 most distinctive words in each domain as extracted from SQuAD and TweetQA. Note the stark differences in the words seen in the TweetQA dataset, which include a large number of user accounts with a heavy tail. Examples include @realdonaldtrump, @jdsutter, @justinkirkland and #cnnworldcup, #goldenglobes. In contrast, the SQuAD dataset rarely has usernames or hashtags that are used to signify events or refer to the authors. It is also worth noting that the data collected from social media can not only capture events and developments in real-time but also capture individual opinions and thus requires reasoning related to the authorship of the content as is illustrated in Table TABREF3 . In addition, while SQuAD requires all answers to be spans from the given passage, we do not enforce any such restriction and answers can be free-form text. In fact, we observed that INLINEFORM1 of our QA pairs consists of answers which do not have an exact substring matching with their corresponding passages. All of the above distinguishing factors have implications to existing models which we analyze in upcoming sections. We conduct analysis on a subset of TweetQA to get a better understanding of the kind of reasoning skills that are required to answer these questions. We sample 150 questions from the development set, then manually label their reasoning categories. Table TABREF26 shows the analysis results. We use some of the categories in SQuAD BIBREF6 and also proposes some tweet-specific reasoning types. Our first observation is that almost half of the questions only require the ability to identify paraphrases. Although most of the “paraphrasing only” questions are considered as fairly easy questions, we find that a significant amount (about 3/4) of these questions are asked about event-related topics, such as information about “who did what to whom, when and where”. This is actually consistent with our motivation to create TweetQA, as we expect this dataset could be used to develop systems that automatically collect information about real-time events. Apart from these questions, there are also a group of questions that require understanding common sense, deep semantics (i.e. the answers cannot be derived from the literal meanings of the tweets), and relations of sentences (including co-reference resolution), which are also appeared in other RC datasets BIBREF6 . On the other hand, the TweetQA also has its unique properties. Specifically, a significant amount of questions require certain reasoning skills that are specific to social media data: [noitemsep] Understanding authorship: Since tweets are highly personal, it is critical to understand how questions/tweets related to the authors. Oral English & Tweet English: Tweets are often oral and informal. QA over tweets requires the understanding of common oral English. Our TweetQA also requires understanding some tweet-specific English, like conversation-style English. Understanding of user IDs & hashtags: Tweets often contains user IDs and hashtags, which are single special tokens. Understanding these special tokens is important to answer person- or event-related questions. Experiments To show the challenge of TweetQA for existing approaches, we consider four representative methods as baselines. For data processing, we first remove the URLs in the tweets and then tokenize the QA pairs and tweets using NLTK. This process is consistent for all baselines. Query Matching Baseline We first consider a simple query matching baseline similar to the IR baseline in Kocisk2017TheNR. But instead of only considering several genres of spans as potential answers, we try to match the question with all possible spans in the tweet context and choose the span with the highest BLEU-1 score as the final answer, which follows the method and implementation of answer span selection for open-domain QA BIBREF19 . We include this baseline to show that TweetQA is a nontrivial task which cannot be easily solved with superficial text matching. Neural Baselines We then explore three typical neural models that perform well on existing formal-text datasets. One takes a generative perspective and learns to decode the answer conditioned on the question and context, while the others learns to extract a text span from the context that best answers the question. RNN-based encoder-decoder models BIBREF20 , BIBREF21 have been widely used for natural language generation tasks. Here we consider a recently proposed generative model BIBREF22 that first encodes the context and question into a multi-perspective memory via four different neural matching layers, then decodes the answer using an attention-based model equipped with both copy and coverage mechanisms. The model is trained on our dataset for 15 epochs and we choose the model parameters that achieve the best BLEU-1 score on the development set. Unlike the aforementioned generative model, the Bi-Directional Attention Flow (BiDAF) BIBREF23 network learns to directly predict the answer span in the context. BiDAF first utilizes multi-level embedding layers to encode both the question and context, then uses bi-directional attention flow to get a query-aware context representation, which is further modeled by an RNN layer to make the span predictions. Since our TweetQA does not have labeled answer spans as in SQuAD, we need to use the human-written answers to retrieve the answer-span labels for training. To get the approximate answer spans, we consider the same matching approach as in the query matching baseline. But instead of using questions to do matching, we use the human-written answers to get the spans that achieve the best BLEU-1 scores. This is another extractive RC model that benefits from the recent advance in pretrained general language encoders BIBREF24 , BIBREF25 . In our work, we select the BERT model BIBREF25 which has achieved the best performance on SQuAD. In our experiments, we use the PyTorch reimplementation of the uncased base model. The batch size is set as 12 and we fine-tune the model for 2 epochs with learning rate 3e-5. Overall Performance We test the performance of all baseline systems using the three generative metrics mentioned in Section SECREF22 . As shown in Table TABREF40 , there is a large performance gap between human performance and all baseline methods, including BERT, which has achieved superhuman performance on SQuAD. This confirms than TweetQA is more challenging than formal-test RC tasks. We also show the upper bound of the extractive models (denoted as Extract-Upper). In the upper bound method, the answers are defined as n-grams from the tweets that maximize the BLEU-1/METEOR/ROUGE-L compared to the annotated groundtruth. From the results, we can see that the BERT model still lags behind the upper bound significantly, showing great potential for future research. It is also interesting to see that the Human performance is slightly worse compared to the upper bound. This indicates (1) the difficulty of our problem also exists for human-beings and (2) for the answer verification process, the workers tend to also extract texts from tweets as answers. According to the comparison between the two non-pretraining baselines, our generative baseline yields better results than BiDAF. We believe this is largely due to the abstractive nature of our dataset, since the workers can sometimes write the answers using their own words. Performance Analysis over Human-Labeled Question Types To better understand the difficulty of the TweetQA task for current neural models, we analyze the decomposed model performance on the different kinds of questions that require different types of reasoning (we tested on the subset which has been used for the analysis in Table TABREF26 ). Table TABREF42 shows the results of the best performed non-pretraining and pretraining approach, i.e., the generative QA baseline and the fine-tuned BERT. Our full comparison including the BiDAF performance and evaluation on more metrics can be found in Appendix SECREF7 . Following previous RC research, we also include analysis on automatically-labeled question types in Appendix SECREF8 . As indicated by the results on METEOR and ROUGE-L (also indicated by a third metric, BLEU-1, as shown in Appendix SECREF7 ), both baselines perform worse on questions that require the understanding deep semantics and userID&hashtags. The former kind of questions also appear in other benchmarks and is known to be challenging for many current models. The second kind of questions is tweet-specific and is related to specific properties of social media data. Since both models are designed for formal-text passages and there is no special treatment for understanding user IDs and hashtags, the performance is severely limited on the questions requiring such reasoning abilities. We believe that good segmentation, disambiguation and linking tools developed by the social media community for processing the userIDs and hashtags will significantly help these question types. Besides the easy questions requiring mainly paraphrasing skill, we also find that the questions requiring the understanding of authorship and oral/tweet English habits are not very difficult. We think this is due to the reason that, except for these tweet-specific tokens, the rest parts of the questions are rather simple, which may require only simple reasoning skill (e.g. paraphrasing). Although BERT was demonstrated to be a powerful tool for reading comprehension, this is the first time a detailed analysis has been done on its reasoning skills. From the results, the huge improvement of BERT mainly comes from two types. The first is paraphrasing, which is not surprising because that a well pretrained language model is expected to be able to better encode sentences. Thus the derived embedding space could work better for sentence comparison. The second type is commonsense, which is consistent with the good performance of BERT BIBREF25 on SWAG BIBREF26 . We believe that this provides further evidence about the connection between large-scaled deep neural language model and certain kinds of commonsense. Conclusion We present the first dataset for QA on social media data by leveraging news media and crowdsourcing. The proposed dataset informs us of the distinctiveness of social media from formal domains in the context of QA. Specifically, we find that QA on social media requires systems to comprehend social media specific linguistic patterns like informality, hashtags, usernames, and authorship. These distinguishing linguistic factors bring up important problems for the research of QA that currently focuses on formal text. We see our dataset as a first step towards enabling not only a deeper understanding of natural language in social media but also rich applications that can extract essential real-time knowledge from social media. Full results of Performance Analysis over Human-Labeled Question Types Table TABREF45 gives our full evaluation on human annotated question types. Compared with the BiDAF model, one interesting observation is that the generative baseline gets much worse results on ambiguous questions. We conjecture that although these questions are meaningless, they still have many words that overlapped with the contexts. This can give BiDAF potential advantage over the generative baseline. Performance Analysis over Automatically-Labeled Question Types Besides the analysis on different reasoning types, we also look into the performance over questions with different first tokens in the development set, which provide us an automatic categorization of questions. According to the results in Table TABREF46 , the three neural baselines all perform the best on “Who” and “Where” questions, to which the answers are often named entities. Since the tweet contexts are short, there are only a small number of named entities to choose from, which could make the answer pattern easy to learn. On the other hand, the neural models fail to perform well on the “Why” questions, and the results of neural baselines are even worse than that of the matching baseline. We find that these questions generally have longer answer phrases than other types of questions, with the average answer length being 3.74 compared to 2.13 for any other types. Also, since all the answers are written by humans instead of just spans from the context, these abstractive answers can make it even harder for current models to handle. We also observe that when people write “Why” questions, they tend to copy word spans from the tweet, potentially making the task easier for the matching baseline.
What evaluation metrics do they use?
53dfcd5d7d2a81855ec1728f0d8e6e24c5638f1e
What Ġevaluation Ġmetrics Ġdo Ġthey Ġuse ?
BLE U - 1 , ĠMeteor Ġ, ĠRouge - L Ġ
<s> What Ġevaluation Ġmetrics Ġdo Ġthey Ġuse ? </s> Introduction </s> Social Ġmedia Ġis Ġnow Ġbecoming Ġan Ġimportant Ġreal - time Ġinformation Ġsource , Ġespecially Ġduring Ġnatural Ġdisasters Ġand Ġemergencies . ĠIt Ġis Ġnow Ġvery Ġcommon Ġfor Ġtraditional Ġnews Ġmedia Ġto Ġfrequently Ġprobe Ġusers Ġand Ġresort Ġto Ġsocial Ġmedia Ġplatforms Ġto Ġobtain Ġreal - time Ġdevelopments Ġof Ġevents . ĠAccording Ġto Ġa Ġrecent Ġsurvey Ġby ĠPew ĠResearch ĠCenter , Ġin Ġ2017 , Ġmore Ġthan Ġtwo - thirds Ġof ĠAmericans Ġread Ġsome Ġof Ġtheir Ġnews Ġon Ġsocial Ġmedia . ĠEven Ġfor ĠAmerican Ġpeople Ġwho Ġare Ġ50 Ġor Ġolder , ĠIN LINE FORM 0 Ġof Ġthem Ġreport Ġgetting Ġnews Ġfrom Ġsocial Ġmedia , Ġwhich Ġis ĠIN LINE FORM 1 Ġpoints Ġhigher Ġthan Ġthe Ġnumber Ġin Ġ2016 . ĠAmong Ġall Ġmajor Ġsocial Ġmedia Ġsites , ĠTwitter Ġis Ġmost Ġfrequently Ġused Ġas Ġa Ġnews Ġsource , Ġwith ĠIN LINE FORM 2 Ġof Ġits Ġusers Ġobtaining Ġtheir Ġnews Ġfrom ĠTwitter . ĠAll Ġthese Ġstatistical Ġfacts Ġsuggest Ġthat Ġunderstanding Ġuser - generated Ġnoisy Ġsocial Ġmedia Ġtext Ġfrom ĠTwitter Ġis Ġa Ġsignificant Ġtask . </s> In Ġrecent Ġyears , Ġwhile Ġseveral Ġtools Ġfor Ġcore Ġnatural Ġlanguage Ġunderstanding Ġtasks Ġinvolving Ġsynt actic Ġand Ġsemantic Ġanalysis Ġhave Ġbeen Ġdeveloped Ġfor Ġnoisy Ġsocial Ġmedia Ġtext ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġthere Ġis Ġlittle Ġwork Ġon Ġquestion Ġanswering Ġor Ġreading Ġcomprehension Ġover Ġsocial Ġmedia , Ġwith Ġthe Ġprimary Ġbottleneck Ġbeing Ġthe Ġlack Ġof Ġavailable Ġdatasets . ĠWe Ġobserve Ġthat Ġrecently Ġproposed ĠQ A Ġdatasets Ġusually Ġfocus Ġon Ġformal Ġdomains , Ġe . g . ĠCNN / Daily Mail ĠB IB REF 4 Ġand ĠNews Q A ĠB IB REF 5 Ġon Ġnews Ġarticles ; ĠS Qu AD ĠB IB REF 6 Ġand ĠWiki M ovies ĠB IB REF 7 Ġthat Ġuse ĠWikipedia . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġthe Ġfirst Ġlarge - scale Ġdataset Ġfor ĠQ A Ġover Ġsocial Ġmedia Ġdata . ĠRather Ġthan Ġna ively Ġobtaining Ġtweets Ġfrom ĠTwitter Ġusing Ġthe ĠTwitter ĠAPI Ġwhich Ġcan Ġyield Ġirrelevant Ġtweets Ġwith Ġno Ġvaluable Ġinformation , Ġwe Ġrestrict Ġourselves Ġonly Ġto Ġtweets Ġwhich Ġhave Ġbeen Ġused Ġby Ġjournalists Ġin Ġnews Ġarticles Ġthus Ġimplicitly Ġimplying Ġthat Ġsuch Ġtweets Ġcontain Ġuseful Ġand Ġrelevant Ġinformation . ĠTo Ġobtain Ġsuch Ġrelevant Ġtweets , Ġwe Ġcrawled Ġthousands Ġof Ġnews Ġarticles Ġthat Ġinclude Ġtweet Ġquotations Ġand Ġthen Ġemployed Ġcrowd - s ourcing Ġto Ġelicit Ġquestions Ġand Ġanswers Ġbased Ġon Ġthese Ġevent - aligned Ġtweets . ĠTable ĠT AB REF 3 Ġgives Ġan Ġexample Ġfrom Ġour ĠTweet Q A Ġdataset . ĠIt Ġshows Ġthat ĠQ A Ġover Ġtweets Ġraises Ġchallenges Ġnot Ġonly Ġbecause Ġof Ġthe Ġinformal Ġnature Ġof Ġoral - style Ġtexts Ġ( e . g . Ġinfer ring Ġthe Ġanswer Ġfrom Ġmultiple Ġshort Ġsentences , Ġlike Ġthe Ġphrase ĠâĢ ľ so Ġyoung âĢ Ŀ Ġthat Ġforms Ġan Ġindependent Ġsentence Ġin Ġthe Ġexample ), Ġbut Ġalso Ġfrom Ġtweet - specific Ġexpressions Ġ( such Ġas Ġinfer ring Ġthat Ġit Ġis ĠâĢ ľ Jay ĠSean âĢ Ŀ Ġfeeling Ġsad Ġabout ĠPaul 's Ġdeath Ġbecause Ġhe Ġposted Ġthe Ġtweet ). </s> Furthermore , Ġwe Ġshow Ġthe Ġdistinctive Ġnature Ġof ĠTweet Q A Ġby Ġcomparing Ġthe Ġcollected Ġdata Ġwith Ġtraditional ĠQ A Ġdatasets Ġcollected Ġprimarily Ġfrom Ġformal Ġdomains . ĠIn Ġparticular , Ġwe Ġdemonstrate Ġempir ically Ġthat Ġthree Ġstrong Ġneural Ġmodels Ġwhich Ġachieve Ġgood Ġperformance Ġon Ġformal Ġdata Ġdo Ġnot Ġgeneral ize Ġwell Ġto Ġsocial Ġmedia Ġdata , Ġbringing Ġout Ġchallenges Ġto Ġdeveloping ĠQ A Ġsystems Ġthat Ġwork Ġwell Ġon Ġsocial Ġmedia Ġdomains . </s> In Ġsummary , Ġour Ġcontributions Ġare : </s> Tweet Q A </s> In Ġthis Ġsection , Ġwe Ġfirst Ġdescribe Ġthe Ġthree - step Ġdata Ġcollection Ġprocess Ġof ĠTweet Q A : Ġtweet Ġcrawling , Ġquestion - answer Ġwriting Ġand Ġanswer Ġvalidation . ĠNext , Ġwe Ġdefine Ġthe Ġspecific Ġtask Ġof ĠTweet Q A Ġand Ġdiscuss Ġseveral Ġevaluation Ġmetrics . ĠTo Ġbetter Ġunderstand Ġthe Ġcharacteristics Ġof Ġthe ĠTweet Q A Ġtask , Ġwe Ġalso Ġinclude Ġour Ġanalysis Ġon Ġthe Ġanswer Ġand Ġquestion Ġcharacteristics Ġusing Ġa Ġsubset Ġof ĠQ A Ġpairs Ġfrom Ġthe Ġdevelopment Ġset . </s> Data ĠCollection </s> One Ġmajor Ġchallenge Ġof Ġbuilding Ġa ĠQ A Ġdataset Ġon Ġtweets Ġis Ġthe Ġsp arsity Ġof Ġinformative Ġtweets . ĠMany Ġusers Ġwrite Ġtweets Ġto Ġexpress Ġtheir Ġfeelings Ġor Ġemotions Ġabout Ġtheir Ġpersonal Ġlives . ĠThese Ġtweets Ġare Ġgenerally Ġunin form ative Ġand Ġalso Ġvery Ġdifficult Ġto Ġask Ġquestions Ġabout . ĠGiven Ġthe Ġlinguistic Ġvariance Ġof Ġtweets , Ġit Ġis Ġgenerally Ġhard Ġto Ġdirectly Ġdistinguish Ġthose Ġtweets Ġfrom Ġinformative Ġones . ĠIn Ġterms Ġof Ġthis , Ġrather Ġthan Ġstarting Ġfrom ĠTwitter ĠAPI ĠSearch , Ġwe Ġlook Ġinto Ġthe Ġarchived Ġsnapshots Ġof Ġtwo Ġmajor Ġnews Ġwebsites Ġ( CNN , ĠNBC ), Ġand Ġthen Ġextract Ġthe Ġtweet Ġblocks Ġthat Ġare Ġembedded Ġin Ġthe Ġnews Ġarticles . ĠIn Ġorder Ġto Ġget Ġenough Ġdata , Ġwe Ġfirst Ġextract Ġthe ĠURLs Ġof Ġall Ġsection Ġpages Ġ( e . g . ĠWorld , ĠPolitics , ĠMoney , ĠTech ) Ġfrom Ġthe Ġsnapshot Ġof Ġeach Ġhome Ġpage Ġand Ġthen Ġcrawl Ġall Ġarticles Ġwith Ġtweets Ġfrom Ġthese Ġsection Ġpages . ĠNote Ġthat Ġanother Ġpossible Ġway Ġto Ġcollect Ġinformative Ġtweets Ġis Ġto Ġdownload Ġthe Ġtweets Ġthat Ġare Ġposted Ġby Ġthe Ġofficial ĠTwitter Ġaccounts Ġof Ġnews Ġmedia . ĠHowever , Ġthese Ġtweets Ġare Ġoften Ġjust Ġthe Ġsumm aries Ġof Ġnews Ġarticles , Ġwhich Ġare Ġwritten Ġin Ġformal Ġtext . ĠAs Ġour Ġfocus Ġis Ġto Ġdevelop Ġa Ġdataset Ġfor ĠQ A Ġon Ġinformal Ġsocial Ġmedia Ġtext , Ġwe Ġdo Ġnot Ġconsider Ġthis Ġapproach . </s> After Ġwe Ġextracted Ġtweets Ġfrom Ġarchived Ġnews Ġarticles , Ġwe Ġobserved Ġthat Ġthere Ġis Ġstill Ġa Ġportion Ġof Ġtweets Ġthat Ġhave Ġvery Ġsimple Ġsemantic Ġstructures Ġand Ġthus Ġare Ġvery Ġdifficult Ġto Ġraise Ġmeaningful Ġquestions . ĠAn Ġexample Ġof Ġsuch Ġtweets Ġcan Ġbe Ġlike : ĠâĢ ľ W anted Ġto Ġshare Ġthis Ġtoday Ġ- Ġ@ IA m Steve Har vey ". ĠThis Ġtweet Ġis Ġactually Ġtalking Ġabout Ġan Ġimage Ġattached Ġto Ġthis Ġtweet . ĠSome Ġother Ġtweets Ġwith Ġsimple Ġtext Ġstructures Ġmay Ġtalk Ġabout Ġan Ġinserted Ġlink Ġor Ġeven Ġvideos . ĠTo Ġfilter Ġout Ġthese Ġtweets Ġthat Ġheavily Ġrely Ġon Ġattached Ġmedia Ġto Ġconvey Ġinformation , Ġwe Ġutilize Ġa Ġstate - of - the - art Ġsemantic Ġrole Ġlabeling Ġmodel Ġtrained Ġon ĠCo N LL - 2005 ĠB IB REF 15 Ġto Ġanalyze Ġthe Ġpredicate - argument Ġstructure Ġof Ġthe Ġtweets Ġcollected Ġfrom Ġnews Ġarticles Ġand Ġkeep Ġonly Ġthe Ġtweets Ġwith Ġmore Ġthan Ġtwo Ġlabeled Ġarguments . ĠThis Ġfiltering Ġprocess Ġalso Ġautomatically Ġfilters Ġout Ġmost Ġof Ġthe Ġshort Ġtweets . ĠFor Ġthe Ġtweets Ġcollected Ġfrom ĠCNN , ĠIN LINE FORM 0 Ġof Ġthem Ġwere Ġfiltered Ġvia Ġsemantic Ġrole Ġlabeling . ĠFor Ġtweets Ġfrom ĠNBC , ĠIN LINE FORM 1 Ġof Ġthe Ġtweets Ġwere Ġfiltered . </s> We Ġthen Ġuse ĠAmazon ĠMechanical ĠTurk Ġto Ġcollect Ġquestion - answer Ġpairs Ġfor Ġthe Ġfiltered Ġtweets . ĠFor Ġeach ĠHuman ĠIntelligence ĠTask Ġ( H IT ), Ġwe Ġask Ġthe Ġworker Ġto Ġread Ġthree Ġtweets Ġand Ġwrite Ġtwo Ġquestion - answer Ġpairs Ġfor Ġeach Ġtweet . ĠTo Ġensure Ġthe Ġquality , Ġwe Ġrequire Ġthe Ġworkers Ġto Ġbe Ġlocated Ġin Ġmajor ĠEnglish Ġspeaking Ġcountries Ġ( i . e . ĠCanada , ĠUS , Ġand ĠUK ) Ġand Ġhave Ġan Ġacceptance Ġrate Ġlarger Ġthan ĠIN LINE FORM 0 Ġ. ĠSince Ġwe Ġuse Ġtweets Ġas Ġcontext , Ġlots Ġof Ġimportant Ġinformation Ġare Ġcontained Ġin Ġhasht ags Ġor Ġeven Ġem oj is . ĠInstead Ġof Ġonly Ġshowing Ġthe Ġtext Ġto Ġthe Ġworkers , Ġwe Ġuse Ġjavascript Ġto Ġdirectly Ġembed Ġthe Ġwhole Ġtweet Ġinto Ġeach ĠHIT . ĠThis Ġgives Ġworkers Ġthe Ġsame Ġexperience Ġas Ġreading Ġtweets Ġvia Ġweb Ġbrowsers Ġand Ġhelp Ġthem Ġto Ġbetter Ġcompose Ġquestions . </s> To Ġavoid Ġtrivial Ġquestions Ġthat Ġcan Ġbe Ġsimply Ġanswered Ġby Ġsuperficial Ġtext Ġmatching Ġmethods Ġor Ġtoo Ġchallenging Ġquestions Ġthat Ġrequire Ġbackground Ġknowledge . ĠWe Ġexplicitly Ġstate Ġthe Ġfollowing Ġitems Ġin Ġthe ĠHIT Ġinstructions Ġfor Ġquestion Ġwriting : </s> No ĠYes - no Ġquestions Ġshould Ġbe Ġasked . </s> The Ġquestion Ġshould Ġhave Ġat Ġleast Ġfive Ġwords . </s> V ideos , Ġimages Ġor Ġinserted Ġlinks Ġshould Ġnot Ġbe Ġconsidered . </s> No Ġbackground Ġknowledge Ġshould Ġbe Ġrequired Ġto Ġanswer Ġthe Ġquestion . </s> To Ġhelp Ġthe Ġworkers Ġbetter Ġfollow Ġthe Ġinstructions , Ġwe Ġalso Ġinclude Ġa Ġrepresentative Ġexample Ġshowing Ġboth Ġgood Ġand Ġbad Ġquestions Ġor Ġanswers Ġin Ġour Ġinstructions . ĠFigure ĠFIG REF 14 Ġshows Ġthe Ġexample Ġwe Ġuse Ġto Ġguide Ġthe Ġworkers . </s> As Ġfor Ġthe Ġanswers , Ġsince Ġthe Ġcontext Ġwe Ġconsider Ġis Ġrelatively Ġshorter Ġthan Ġthe Ġcontext Ġof Ġprevious Ġdatasets , Ġwe Ġdo Ġnot Ġrestrict Ġthe Ġanswers Ġto Ġbe Ġin Ġthe Ġtweet , Ġotherwise , Ġthe Ġtask Ġmay Ġpotentially Ġbe Ġsimplified Ġas Ġa Ġclassification Ġproblem . ĠThe Ġworkers Ġare Ġallowed Ġto Ġwrite Ġtheir Ġanswers Ġin Ġtheir Ġown Ġwords . ĠWe Ġjust Ġrequire Ġthe Ġanswers Ġto Ġbe Ġbrief Ġand Ġcan Ġbe Ġdirectly Ġinferred Ġfrom Ġthe Ġtweets . </s> After Ġwe Ġretrieve Ġthe ĠQ A Ġpairs Ġfrom Ġall ĠHIT s , Ġwe Ġconduct Ġfurther Ġpost - fil tering Ġto Ġfilter Ġout Ġthe Ġpairs Ġfrom Ġworkers Ġthat Ġobviously Ġdo Ġnot Ġfollow Ġinstructions . ĠWe Ġremove ĠQ A Ġpairs Ġwith Ġyes / no Ġanswers . ĠQuestions Ġwith Ġless Ġthan Ġfive Ġwords Ġare Ġalso Ġfiltered Ġout . ĠThis Ġprocess Ġfiltered ĠIN LINE FORM 0 Ġof Ġthe ĠQ A Ġpairs . ĠThe Ġdataset Ġnow Ġincludes Ġ10 , 8 98 Ġarticles , Ġ17 , 794 Ġtweets , Ġand Ġ13 , 757 Ġcrowds ourced Ġquestion - answer Ġpairs . ĠThe Ġcollected ĠQ A Ġpairs Ġwill Ġbe Ġdirectly Ġavailable Ġto Ġthe Ġpublic , Ġand Ġwe Ġwill Ġprovide Ġa Ġscript Ġto Ġdownload Ġthe Ġoriginal Ġtweets Ġand Ġdetailed Ġdocumentation Ġon Ġhow Ġwe Ġbuild Ġour Ġdataset . ĠAlso Ġnote Ġthat Ġsince Ġwe Ġkeep Ġthe Ġoriginal Ġnews Ġarticle Ġand Ġnews Ġtitles Ġfor Ġeach Ġtweet , Ġour Ġdataset Ġcan Ġalso Ġbe Ġused Ġto Ġexplore Ġmore Ġchallenging Ġgeneration Ġtasks . ĠTable ĠT AB REF 19 Ġshows Ġthe Ġstatistics Ġof Ġour Ġcurrent Ġcollection , Ġand Ġthe Ġfrequency Ġof Ġdifferent Ġtypes Ġof Ġquestions Ġis Ġshown Ġin ĠTable ĠT AB REF 21 Ġ. ĠAll ĠQ A Ġpairs Ġwere Ġwritten Ġby Ġ49 2 Ġindividual Ġworkers . </s> For Ġthe Ġpurposes Ġof Ġhuman Ġperformance Ġevaluation Ġand Ġinter - annot ator Ġagreement Ġchecking , Ġwe Ġlaunch Ġa Ġdifferent Ġset Ġof ĠHIT s Ġto Ġask Ġworkers Ġto Ġanswer Ġquestions Ġin Ġthe Ġtest Ġand Ġdevelopment Ġset . ĠThe Ġworkers Ġare Ġshown Ġwith Ġthe Ġtweet Ġblocks Ġas Ġwell Ġas Ġthe Ġquestions Ġcollected Ġin Ġthe Ġprevious Ġstep . ĠAt Ġthis Ġstep , Ġworkers Ġare Ġallowed Ġto Ġlabel Ġthe Ġquestions Ġas ĠâĢ ľ NA " Ġif Ġthey Ġthink Ġthe Ġquestions Ġare Ġnot Ġanswer able . ĠWe Ġfind Ġthat ĠIN LINE FORM 0 Ġof Ġthe Ġquestions Ġare Ġlabeled Ġas Ġun answer able Ġby Ġthe Ġworkers Ġ( for ĠS Qu AD , Ġthe Ġratio Ġis ĠIN LINE FORM 1 Ġ). ĠSince Ġthe Ġanswers Ġcollected Ġat Ġthis Ġstep Ġand Ġprevious Ġstep Ġare Ġwritten Ġby Ġdifferent Ġworkers , Ġthe Ġanswers Ġcan Ġbe Ġwritten Ġin Ġdifferent Ġtext Ġforms Ġeven Ġthey Ġare Ġsem antically Ġequal Ġto Ġeach Ġother . ĠFor Ġexample , Ġone Ġanswer Ġcan Ġbe ĠâĢ ľ Hillary ĠClinton âĢ Ŀ Ġwhile Ġthe Ġother Ġis ĠâĢ ľ @ Hillary Clinton âĢ Ŀ . ĠAs Ġit Ġis Ġnot Ġstraightforward Ġto Ġautomatically Ġcalculate Ġthe Ġoverall Ġagreement , Ġwe Ġmanually Ġcheck Ġthe Ġagreement Ġon Ġa Ġsubset Ġof Ġ200 Ġrandom Ġsamples Ġfrom Ġthe Ġdevelopment Ġset Ġand Ġask Ġan Ġindependent Ġhuman Ġmoderator Ġto Ġverify Ġthe Ġresult . ĠIt Ġturns Ġout Ġthat ĠIN LINE FORM 2 Ġof Ġthe Ġanswers Ġpairs Ġare Ġsem antically Ġequivalent , ĠIN LINE FORM 3 Ġof Ġthem Ġare Ġpartially Ġequivalent Ġ( one Ġof Ġthem Ġis Ġincomplete ) Ġand ĠIN LINE FORM 4 Ġare Ġtotally Ġinconsistent . ĠThe Ġanswers Ġcollected Ġat Ġthis Ġstep Ġare Ġalso Ġused Ġto Ġmeasure Ġthe Ġhuman Ġperformance . ĠWe Ġhave Ġ59 Ġindividual Ġworkers Ġparticipated Ġin Ġthis Ġprocess . </s> Task Ġand ĠEvaluation </s> As Ġdescribed Ġin Ġthe Ġquestion - answer Ġwriting Ġprocess , Ġthe Ġanswers Ġin Ġour Ġdataset Ġare Ġdifferent Ġfrom Ġthose Ġin Ġsome Ġexisting Ġextract ive Ġdatasets . ĠThus Ġwe Ġconsider Ġthe Ġtask Ġof Ġanswer Ġgeneration Ġfor ĠTweet Q A Ġand Ġwe Ġuse Ġseveral Ġstandard Ġmetrics Ġfor Ġnatural Ġlanguage Ġgeneration Ġto Ġevaluate ĠQ A Ġsystems Ġon Ġour Ġdataset , Ġnamely Ġwe Ġconsider ĠB LE U - 1 ĠB IB REF 16 Ġ, ĠMeteor ĠB IB REF 17 Ġand ĠRouge - L ĠB IB REF 18 Ġin Ġthis Ġpaper . </s> To Ġevaluate Ġmachine Ġsystems , Ġwe Ġcompute Ġthe Ġscores Ġusing Ġboth Ġthe Ġoriginal Ġanswer Ġand Ġvalidation Ġanswer Ġas Ġreferences . ĠFor Ġhuman Ġperformance , Ġwe Ġuse Ġthe Ġvalidation Ġanswers Ġas Ġgenerated Ġones Ġand Ġthe Ġoriginal Ġanswers Ġas Ġreferences Ġto Ġcalculate Ġthe Ġscores . </s> Analysis </s> In Ġthis Ġsection , Ġwe Ġanalyze Ġour Ġdataset Ġand Ġoutline Ġthe Ġkey Ġproperties Ġthat Ġdistinguish Ġit Ġfrom Ġstandard ĠQ A Ġdatasets Ġlike ĠS Qu AD ĠB IB REF 6 Ġ. ĠFirst , Ġour Ġdataset Ġis Ġderived Ġfrom Ġsocial Ġmedia Ġtext Ġwhich Ġcan Ġbe Ġquite Ġinformal Ġand Ġuser - centric Ġas Ġopposed Ġto ĠS Qu AD Ġwhich Ġis Ġderived Ġfrom ĠWikipedia Ġand Ġhence Ġmore Ġformal Ġin Ġnature . ĠWe Ġobserve Ġthat Ġthe Ġshared Ġvocabulary Ġbetween ĠS Qu AD Ġand ĠTweet Q A Ġis Ġonly ĠIN LINE FORM 0 Ġ, Ġsuggesting Ġa Ġsignificant Ġdifference Ġin Ġtheir Ġlex ical Ġcontent . ĠFigure ĠFIG REF 25 Ġshows Ġthe Ġ1000 Ġmost Ġdistinctive Ġwords Ġin Ġeach Ġdomain Ġas Ġextracted Ġfrom ĠS Qu AD Ġand ĠTweet Q A . ĠNote Ġthe Ġstark Ġdifferences Ġin Ġthe Ġwords Ġseen Ġin Ġthe ĠTweet Q A Ġdataset , Ġwhich Ġinclude Ġa Ġlarge Ġnumber Ġof Ġuser Ġaccounts Ġwith Ġa Ġheavy Ġtail . ĠExamples Ġinclude Ġ@ re ald on ald trump , Ġ@ j ds utter , Ġ@ just ink irk land Ġand Ġ# c nn world cup , Ġ# gold eng l ob es . ĠIn Ġcontrast , Ġthe ĠS Qu AD Ġdataset Ġrarely Ġhas Ġus ern ames Ġor Ġhasht ags Ġthat Ġare Ġused Ġto Ġsignify Ġevents Ġor Ġrefer Ġto Ġthe Ġauthors . ĠIt Ġis Ġalso Ġworth Ġnoting Ġthat Ġthe Ġdata Ġcollected Ġfrom Ġsocial Ġmedia Ġcan Ġnot Ġonly Ġcapture Ġevents Ġand Ġdevelopments Ġin Ġreal - time Ġbut Ġalso Ġcapture Ġindividual Ġopinions Ġand Ġthus Ġrequires Ġreasoning Ġrelated Ġto Ġthe Ġauthors hip Ġof Ġthe Ġcontent Ġas Ġis Ġillustrated Ġin ĠTable ĠT AB REF 3 Ġ. ĠIn Ġaddition , Ġwhile ĠS Qu AD Ġrequires Ġall Ġanswers Ġto Ġbe Ġspans Ġfrom Ġthe Ġgiven Ġpassage , Ġwe Ġdo Ġnot Ġenforce Ġany Ġsuch Ġrestriction Ġand Ġanswers Ġcan Ġbe Ġfree - form Ġtext . ĠIn Ġfact , Ġwe Ġobserved Ġthat ĠIN LINE FORM 1 Ġof Ġour ĠQ A Ġpairs Ġconsists Ġof Ġanswers Ġwhich Ġdo Ġnot Ġhave Ġan Ġexact Ġsubst ring Ġmatching Ġwith Ġtheir Ġcorresponding Ġpassages . ĠAll Ġof Ġthe Ġabove Ġdistinguishing Ġfactors Ġhave Ġimplications Ġto Ġexisting Ġmodels Ġwhich Ġwe Ġanalyze Ġin Ġupcoming Ġsections . </s> We Ġconduct Ġanalysis Ġon Ġa Ġsubset Ġof ĠTweet Q A Ġto Ġget Ġa Ġbetter Ġunderstanding Ġof Ġthe Ġkind Ġof Ġreasoning Ġskills Ġthat Ġare Ġrequired Ġto Ġanswer Ġthese Ġquestions . ĠWe Ġsample Ġ150 Ġquestions Ġfrom Ġthe Ġdevelopment Ġset , Ġthen Ġmanually Ġlabel Ġtheir Ġreasoning Ġcategories . ĠTable ĠT AB REF 26 Ġshows Ġthe Ġanalysis Ġresults . ĠWe Ġuse Ġsome Ġof Ġthe Ġcategories Ġin ĠS Qu AD ĠB IB REF 6 Ġand Ġalso Ġproposes Ġsome Ġtweet - specific Ġreasoning Ġtypes . </s> Our Ġfirst Ġobservation Ġis Ġthat Ġalmost Ġhalf Ġof Ġthe Ġquestions Ġonly Ġrequire Ġthe Ġability Ġto Ġidentify Ġparaph r ases . ĠAlthough Ġmost Ġof Ġthe ĠâĢ ľ par aph r asing Ġonly âĢ Ŀ Ġquestions Ġare Ġconsidered Ġas Ġfairly Ġeasy Ġquestions , Ġwe Ġfind Ġthat Ġa Ġsignificant Ġamount Ġ( about Ġ3 / 4 ) Ġof Ġthese Ġquestions Ġare Ġasked Ġabout Ġevent - related Ġtopics , Ġsuch Ġas Ġinformation Ġabout ĠâĢ ľ who Ġdid Ġwhat Ġto Ġwhom , Ġwhen Ġand Ġwhere âĢ Ŀ . ĠThis Ġis Ġactually Ġconsistent Ġwith Ġour Ġmotivation Ġto Ġcreate ĠTweet Q A , Ġas Ġwe Ġexpect Ġthis Ġdataset Ġcould Ġbe Ġused Ġto Ġdevelop Ġsystems Ġthat Ġautomatically Ġcollect Ġinformation Ġabout Ġreal - time Ġevents . </s> Apart Ġfrom Ġthese Ġquestions , Ġthere Ġare Ġalso Ġa Ġgroup Ġof Ġquestions Ġthat Ġrequire Ġunderstanding Ġcommon Ġsense , Ġdeep Ġsemantics Ġ( i . e . Ġthe Ġanswers Ġcannot Ġbe Ġderived Ġfrom Ġthe Ġliteral Ġmeanings Ġof Ġthe Ġtweets ), Ġand Ġrelations Ġof Ġsentences Ġ( including Ġco - reference Ġresolution ), Ġwhich Ġare Ġalso Ġappeared Ġin Ġother ĠRC Ġdatasets ĠB IB REF 6 Ġ. ĠOn Ġthe Ġother Ġhand , Ġthe ĠTweet Q A Ġalso Ġhas Ġits Ġunique Ġproperties . ĠSpecifically , Ġa Ġsignificant Ġamount Ġof Ġquestions Ġrequire Ġcertain Ġreasoning Ġskills Ġthat Ġare Ġspecific Ġto Ġsocial Ġmedia Ġdata : </s> [ no item se p ] </s> Understanding Ġauthors hip : ĠSince Ġtweets Ġare Ġhighly Ġpersonal , Ġit Ġis Ġcritical Ġto Ġunderstand Ġhow Ġquestions / t we ets Ġrelated Ġto Ġthe Ġauthors . </s> O ral ĠEnglish Ġ& ĠTweet ĠEnglish : ĠTwe ets Ġare Ġoften Ġoral Ġand Ġinformal . ĠQ A Ġover Ġtweets Ġrequires Ġthe Ġunderstanding Ġof Ġcommon Ġoral ĠEnglish . ĠOur ĠTweet Q A Ġalso Ġrequires Ġunderstanding Ġsome Ġtweet - specific ĠEnglish , Ġlike Ġconversation - style ĠEnglish . </s> Understanding Ġof Ġuser ĠIDs Ġ& Ġhasht ags : ĠTwe ets Ġoften Ġcontains Ġuser ĠIDs Ġand Ġhasht ags , Ġwhich Ġare Ġsingle Ġspecial Ġtokens . ĠUnderstanding Ġthese Ġspecial Ġtokens Ġis Ġimportant Ġto Ġanswer Ġperson - Ġor Ġevent - related Ġquestions . </s> Exper iments </s> To Ġshow Ġthe Ġchallenge Ġof ĠTweet Q A Ġfor Ġexisting Ġapproaches , Ġwe Ġconsider Ġfour Ġrepresentative Ġmethods Ġas Ġbas elines . ĠFor Ġdata Ġprocessing , Ġwe Ġfirst Ġremove Ġthe ĠURLs Ġin Ġthe Ġtweets Ġand Ġthen Ġtoken ize Ġthe ĠQ A Ġpairs Ġand Ġtweets Ġusing ĠNL T K . ĠThis Ġprocess Ġis Ġconsistent Ġfor Ġall Ġbas elines . </s> Query ĠMatch ing ĠBas eline </s> We Ġfirst Ġconsider Ġa Ġsimple Ġquery Ġmatching Ġbaseline Ġsimilar Ġto Ġthe ĠIR Ġbaseline Ġin ĠK oc isk 2017 The NR . ĠBut Ġinstead Ġof Ġonly Ġconsidering Ġseveral Ġgenres Ġof Ġspans Ġas Ġpotential Ġanswers , Ġwe Ġtry Ġto Ġmatch Ġthe Ġquestion Ġwith Ġall Ġpossible Ġspans Ġin Ġthe Ġtweet Ġcontext Ġand Ġchoose Ġthe Ġspan Ġwith Ġthe Ġhighest ĠB LE U - 1 Ġscore Ġas Ġthe Ġfinal Ġanswer , Ġwhich Ġfollows Ġthe Ġmethod Ġand Ġimplementation Ġof Ġanswer Ġspan Ġselection Ġfor Ġopen - domain ĠQ A ĠB IB REF 19 Ġ. ĠWe Ġinclude Ġthis Ġbaseline Ġto Ġshow Ġthat ĠTweet Q A Ġis Ġa Ġnont riv ial Ġtask Ġwhich Ġcannot Ġbe Ġeasily Ġsolved Ġwith Ġsuperficial Ġtext Ġmatching . </s> Ne ural ĠBas elines </s> We Ġthen Ġexplore Ġthree Ġtypical Ġneural Ġmodels Ġthat Ġperform Ġwell Ġon Ġexisting Ġformal - text Ġdatasets . ĠOne Ġtakes Ġa Ġgener ative Ġperspective Ġand Ġlearns Ġto Ġdecode Ġthe Ġanswer Ġconditioned Ġon Ġthe Ġquestion Ġand Ġcontext , Ġwhile Ġthe Ġothers Ġlearns Ġto Ġextract Ġa Ġtext Ġspan Ġfrom Ġthe Ġcontext Ġthat Ġbest Ġanswers Ġthe Ġquestion . </s> R NN - based Ġenc oder - dec oder Ġmodels ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġhave Ġbeen Ġwidely Ġused Ġfor Ġnatural Ġlanguage Ġgeneration Ġtasks . ĠHere Ġwe Ġconsider Ġa Ġrecently Ġproposed Ġgener ative Ġmodel ĠB IB REF 22 Ġthat Ġfirst Ġenc odes Ġthe Ġcontext Ġand Ġquestion Ġinto Ġa Ġmulti - pers pect ive Ġmemory Ġvia Ġfour Ġdifferent Ġneural Ġmatching Ġlayers , Ġthen Ġdec odes Ġthe Ġanswer Ġusing Ġan Ġattention - based Ġmodel Ġequipped Ġwith Ġboth Ġcopy Ġand Ġcoverage Ġmechanisms . ĠThe Ġmodel Ġis Ġtrained Ġon Ġour Ġdataset Ġfor Ġ15 Ġepoch s Ġand Ġwe Ġchoose Ġthe Ġmodel Ġparameters Ġthat Ġachieve Ġthe Ġbest ĠB LE U - 1 Ġscore Ġon Ġthe Ġdevelopment Ġset . </s> Unlike Ġthe Ġaforementioned Ġgener ative Ġmodel , Ġthe ĠBi - D irection al ĠAttention ĠFlow Ġ( Bi DA F ) ĠB IB REF 23 Ġnetwork Ġlearns Ġto Ġdirectly Ġpredict Ġthe Ġanswer Ġspan Ġin Ġthe Ġcontext . ĠBi DA F Ġfirst Ġutilizes Ġmulti - level Ġembed ding Ġlayers Ġto Ġencode Ġboth Ġthe Ġquestion Ġand Ġcontext , Ġthen Ġuses Ġbi - direction al Ġattention Ġflow Ġto Ġget Ġa Ġquery - aware Ġcontext Ġrepresentation , Ġwhich Ġis Ġfurther Ġmodeled Ġby Ġan ĠR NN Ġlayer Ġto Ġmake Ġthe Ġspan Ġpredictions . ĠSince Ġour ĠTweet Q A Ġdoes Ġnot Ġhave Ġlabeled Ġanswer Ġspans Ġas Ġin ĠS Qu AD , Ġwe Ġneed Ġto Ġuse Ġthe Ġhuman - written Ġanswers Ġto Ġretrieve Ġthe Ġanswer - span Ġlabels Ġfor Ġtraining . ĠTo Ġget Ġthe Ġapproximate Ġanswer Ġspans , Ġwe Ġconsider Ġthe Ġsame Ġmatching Ġapproach Ġas Ġin Ġthe Ġquery Ġmatching Ġbaseline . ĠBut Ġinstead Ġof Ġusing Ġquestions Ġto Ġdo Ġmatching , Ġwe Ġuse Ġthe Ġhuman - written Ġanswers Ġto Ġget Ġthe Ġspans Ġthat Ġachieve Ġthe Ġbest ĠB LE U - 1 Ġscores . </s> This Ġis Ġanother Ġextract ive ĠRC Ġmodel Ġthat Ġbenefits Ġfrom Ġthe Ġrecent Ġadvance Ġin Ġpret rained Ġgeneral Ġlanguage Ġenc od ers ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ. ĠIn Ġour Ġwork , Ġwe Ġselect Ġthe ĠB ERT Ġmodel ĠB IB REF 25 Ġwhich Ġhas Ġachieved Ġthe Ġbest Ġperformance Ġon ĠS Qu AD . ĠIn Ġour Ġexperiments , Ġwe Ġuse Ġthe ĠPy Tor ch Ġreim plementation Ġof Ġthe Ġunc ased Ġbase Ġmodel . ĠThe Ġbatch Ġsize Ġis Ġset Ġas Ġ12 Ġand Ġwe Ġfine - t une Ġthe Ġmodel Ġfor Ġ2 Ġepoch s Ġwith Ġlearning Ġrate Ġ3 e - 5 . </s> Overall ĠPerformance </s> We Ġtest Ġthe Ġperformance Ġof Ġall Ġbaseline Ġsystems Ġusing Ġthe Ġthree Ġgener ative Ġmetrics Ġmentioned Ġin ĠSection ĠSEC REF 22 Ġ. ĠAs Ġshown Ġin ĠTable ĠT AB REF 40 Ġ, Ġthere Ġis Ġa Ġlarge Ġperformance Ġgap Ġbetween Ġhuman Ġperformance Ġand Ġall Ġbaseline Ġmethods , Ġincluding ĠB ERT , Ġwhich Ġhas Ġachieved Ġsuperhuman Ġperformance Ġon ĠS Qu AD . ĠThis Ġconfirms Ġthan ĠTweet Q A Ġis Ġmore Ġchallenging Ġthan Ġformal - test ĠRC Ġtasks . </s> We Ġalso Ġshow Ġthe Ġupper Ġbound Ġof Ġthe Ġextract ive Ġmodels Ġ( den oted Ġas ĠExtract - U pper ). ĠIn Ġthe Ġupper Ġbound Ġmethod , Ġthe Ġanswers Ġare Ġdefined Ġas Ġn - gram s Ġfrom Ġthe Ġtweets Ġthat Ġmaximize Ġthe ĠB LE U - 1 / MET E OR / R OU GE - L Ġcompared Ġto Ġthe Ġannot ated Ġground truth . ĠFrom Ġthe Ġresults , Ġwe Ġcan Ġsee Ġthat Ġthe ĠB ERT Ġmodel Ġstill Ġl ags Ġbehind Ġthe Ġupper Ġbound Ġsignificantly , Ġshowing Ġgreat Ġpotential Ġfor Ġfuture Ġresearch . ĠIt Ġis Ġalso Ġinteresting Ġto Ġsee Ġthat Ġthe ĠHuman Ġperformance Ġis Ġslightly Ġworse Ġcompared Ġto Ġthe Ġupper Ġbound . ĠThis Ġindicates Ġ( 1 ) Ġthe Ġdifficulty Ġof Ġour Ġproblem Ġalso Ġexists Ġfor Ġhuman - be ings Ġand Ġ( 2 ) Ġfor Ġthe Ġanswer Ġverification Ġprocess , Ġthe Ġworkers Ġtend Ġto Ġalso Ġextract Ġtexts Ġfrom Ġtweets Ġas Ġanswers . </s> According Ġto Ġthe Ġcomparison Ġbetween Ġthe Ġtwo Ġnon - pret raining Ġbas elines , Ġour Ġgener ative Ġbaseline Ġyields Ġbetter Ġresults Ġthan ĠBi DA F . ĠWe Ġbelieve Ġthis Ġis Ġlargely Ġdue Ġto Ġthe Ġabstract ive Ġnature Ġof Ġour Ġdataset , Ġsince Ġthe Ġworkers Ġcan Ġsometimes Ġwrite Ġthe Ġanswers Ġusing Ġtheir Ġown Ġwords . </s> Performance ĠAnalysis Ġover ĠHuman - Label ed ĠQuestion ĠTypes </s> To Ġbetter Ġunderstand Ġthe Ġdifficulty Ġof Ġthe ĠTweet Q A Ġtask Ġfor Ġcurrent Ġneural Ġmodels , Ġwe Ġanalyze Ġthe Ġdecomp osed Ġmodel Ġperformance Ġon Ġthe Ġdifferent Ġkinds Ġof Ġquestions Ġthat Ġrequire Ġdifferent Ġtypes Ġof Ġreasoning Ġ( we Ġtested Ġon Ġthe Ġsubset Ġwhich Ġhas Ġbeen Ġused Ġfor Ġthe Ġanalysis Ġin ĠTable ĠT AB REF 26 Ġ). ĠTable ĠT AB REF 42 Ġshows Ġthe Ġresults Ġof Ġthe Ġbest Ġperformed Ġnon - pret raining Ġand Ġpret raining Ġapproach , Ġi . e ., Ġthe Ġgener ative ĠQ A Ġbaseline Ġand Ġthe Ġfine - tun ed ĠB ERT . ĠOur Ġfull Ġcomparison Ġincluding Ġthe ĠBi DA F Ġperformance Ġand Ġevaluation Ġon Ġmore Ġmetrics Ġcan Ġbe Ġfound Ġin ĠAppendix ĠSEC REF 7 Ġ. ĠFollowing Ġprevious ĠRC Ġresearch , Ġwe Ġalso Ġinclude Ġanalysis Ġon Ġautomatically - label ed Ġquestion Ġtypes Ġin ĠAppendix ĠSEC REF 8 Ġ. </s> As Ġindicated Ġby Ġthe Ġresults Ġon ĠMET E OR Ġand ĠR OU GE - L Ġ( also Ġindicated Ġby Ġa Ġthird Ġmetric , ĠB LE U - 1 , Ġas Ġshown Ġin ĠAppendix ĠSEC REF 7 Ġ), Ġboth Ġbas elines Ġperform Ġworse Ġon Ġquestions Ġthat Ġrequire Ġthe Ġunderstanding Ġdeep Ġsemantics Ġand Ġuser ID & hash tags . ĠThe Ġformer Ġkind Ġof Ġquestions Ġalso Ġappear Ġin Ġother Ġbenchmarks Ġand Ġis Ġknown Ġto Ġbe Ġchallenging Ġfor Ġmany Ġcurrent Ġmodels . ĠThe Ġsecond Ġkind Ġof Ġquestions Ġis Ġtweet - specific Ġand Ġis Ġrelated Ġto Ġspecific Ġproperties Ġof Ġsocial Ġmedia Ġdata . ĠSince Ġboth Ġmodels Ġare Ġdesigned Ġfor Ġformal - text Ġpassages Ġand Ġthere Ġis Ġno Ġspecial Ġtreatment Ġfor Ġunderstanding Ġuser ĠIDs Ġand Ġhasht ags , Ġthe Ġperformance Ġis Ġseverely Ġlimited Ġon Ġthe Ġquestions Ġrequiring Ġsuch Ġreasoning Ġabilities . ĠWe Ġbelieve Ġthat Ġgood Ġsegment ation , Ġdis amb ig uation Ġand Ġlinking Ġtools Ġdeveloped Ġby Ġthe Ġsocial Ġmedia Ġcommunity Ġfor Ġprocessing Ġthe Ġuser IDs Ġand Ġhasht ags Ġwill Ġsignificantly Ġhelp Ġthese Ġquestion Ġtypes . </s> Besides Ġthe Ġeasy Ġquestions Ġrequiring Ġmainly Ġparaph r asing Ġskill , Ġwe Ġalso Ġfind Ġthat Ġthe Ġquestions Ġrequiring Ġthe Ġunderstanding Ġof Ġauthors hip Ġand Ġoral / t weet ĠEnglish Ġhabits Ġare Ġnot Ġvery Ġdifficult . ĠWe Ġthink Ġthis Ġis Ġdue Ġto Ġthe Ġreason Ġthat , Ġexcept Ġfor Ġthese Ġtweet - specific Ġtokens , Ġthe Ġrest Ġparts Ġof Ġthe Ġquestions Ġare Ġrather Ġsimple , Ġwhich Ġmay Ġrequire Ġonly Ġsimple Ġreasoning Ġskill Ġ( e . g . Ġparaph r asing ). </s> Although ĠB ERT Ġwas Ġdemonstrated Ġto Ġbe Ġa Ġpowerful Ġtool Ġfor Ġreading Ġcomprehension , Ġthis Ġis Ġthe Ġfirst Ġtime Ġa Ġdetailed Ġanalysis Ġhas Ġbeen Ġdone Ġon Ġits Ġreasoning Ġskills . ĠFrom Ġthe Ġresults , Ġthe Ġhuge Ġimprovement Ġof ĠB ERT Ġmainly Ġcomes Ġfrom Ġtwo Ġtypes . ĠThe Ġfirst Ġis Ġparaph r asing , Ġwhich Ġis Ġnot Ġsurprising Ġbecause Ġthat Ġa Ġwell Ġpret rained Ġlanguage Ġmodel Ġis Ġexpected Ġto Ġbe Ġable Ġto Ġbetter Ġencode Ġsentences . ĠThus Ġthe Ġderived Ġembed ding Ġspace Ġcould Ġwork Ġbetter Ġfor Ġsentence Ġcomparison . ĠThe Ġsecond Ġtype Ġis Ġcommons ense , Ġwhich Ġis Ġconsistent Ġwith Ġthe Ġgood Ġperformance Ġof ĠB ERT ĠB IB REF 25 Ġon ĠSW AG ĠB IB REF 26 Ġ. ĠWe Ġbelieve Ġthat Ġthis Ġprovides Ġfurther Ġevidence Ġabout Ġthe Ġconnection Ġbetween Ġlarge - sc aled Ġdeep Ġneural Ġlanguage Ġmodel Ġand Ġcertain Ġkinds Ġof Ġcommons ense . </s> Conclusion </s> We Ġpresent Ġthe Ġfirst Ġdataset Ġfor ĠQ A Ġon Ġsocial Ġmedia Ġdata Ġby Ġleveraging Ġnews Ġmedia Ġand Ġcrowds ourcing . ĠThe Ġproposed Ġdataset Ġinforms Ġus Ġof Ġthe Ġdistinct iveness Ġof Ġsocial Ġmedia Ġfrom Ġformal Ġdomains Ġin Ġthe Ġcontext Ġof ĠQ A . ĠSpecifically , Ġwe Ġfind Ġthat ĠQ A Ġon Ġsocial Ġmedia Ġrequires Ġsystems Ġto Ġcomprehend Ġsocial Ġmedia Ġspecific Ġlinguistic Ġpatterns Ġlike Ġinform ality , Ġhasht ags , Ġus ern ames , Ġand Ġauthors hip . ĠThese Ġdistinguishing Ġlinguistic Ġfactors Ġbring Ġup Ġimportant Ġproblems Ġfor Ġthe Ġresearch Ġof ĠQ A Ġthat Ġcurrently Ġfocuses Ġon Ġformal Ġtext . ĠWe Ġsee Ġour Ġdataset Ġas Ġa Ġfirst Ġstep Ġtowards Ġenabling Ġnot Ġonly Ġa Ġdeeper Ġunderstanding Ġof Ġnatural Ġlanguage Ġin Ġsocial Ġmedia Ġbut Ġalso Ġrich Ġapplications Ġthat Ġcan Ġextract Ġessential Ġreal - time Ġknowledge Ġfrom Ġsocial Ġmedia . </s> Full Ġresults Ġof ĠPerformance ĠAnalysis Ġover ĠHuman - Label ed ĠQuestion ĠTypes </s> Table ĠT AB REF 45 Ġgives Ġour Ġfull Ġevaluation Ġon Ġhuman Ġannot ated Ġquestion Ġtypes . </s> Compared Ġwith Ġthe ĠBi DA F Ġmodel , Ġone Ġinteresting Ġobservation Ġis Ġthat Ġthe Ġgener ative Ġbaseline Ġgets Ġmuch Ġworse Ġresults Ġon Ġambiguous Ġquestions . ĠWe Ġconjecture Ġthat Ġalthough Ġthese Ġquestions Ġare Ġmeaningless , Ġthey Ġstill Ġhave Ġmany Ġwords Ġthat Ġoverl apped Ġwith Ġthe Ġcontexts . ĠThis Ġcan Ġgive ĠBi DA F Ġpotential Ġadvantage Ġover Ġthe Ġgener ative Ġbaseline . </s> Performance ĠAnalysis Ġover ĠAutom atically - Label ed ĠQuestion ĠTypes </s> Besides Ġthe Ġanalysis Ġon Ġdifferent Ġreasoning Ġtypes , Ġwe Ġalso Ġlook Ġinto Ġthe Ġperformance Ġover Ġquestions Ġwith Ġdifferent Ġfirst Ġtokens Ġin Ġthe Ġdevelopment Ġset , Ġwhich Ġprovide Ġus Ġan Ġautomatic Ġcategor ization Ġof Ġquestions . ĠAccording Ġto Ġthe Ġresults Ġin ĠTable ĠT AB REF 46 Ġ, Ġthe Ġthree Ġneural Ġbas elines Ġall Ġperform Ġthe Ġbest Ġon ĠâĢ ľ Who âĢ Ŀ Ġand ĠâĢ ľ Where âĢ Ŀ Ġquestions , Ġto Ġwhich Ġthe Ġanswers Ġare Ġoften Ġnamed Ġentities . ĠSince Ġthe Ġtweet Ġcontexts Ġare Ġshort , Ġthere Ġare Ġonly Ġa Ġsmall Ġnumber Ġof Ġnamed Ġentities Ġto Ġchoose Ġfrom , Ġwhich Ġcould Ġmake Ġthe Ġanswer Ġpattern Ġeasy Ġto Ġlearn . ĠOn Ġthe Ġother Ġhand , Ġthe Ġneural Ġmodels Ġfail Ġto Ġperform Ġwell Ġon Ġthe ĠâĢ ľ Why âĢ Ŀ Ġquestions , Ġand Ġthe Ġresults Ġof Ġneural Ġbas elines Ġare Ġeven Ġworse Ġthan Ġthat Ġof Ġthe Ġmatching Ġbaseline . ĠWe Ġfind Ġthat Ġthese Ġquestions Ġgenerally Ġhave Ġlonger Ġanswer Ġphrases Ġthan Ġother Ġtypes Ġof Ġquestions , Ġwith Ġthe Ġaverage Ġanswer Ġlength Ġbeing Ġ3 . 74 Ġcompared Ġto Ġ2 . 13 Ġfor Ġany Ġother Ġtypes . ĠAlso , Ġsince Ġall Ġthe Ġanswers Ġare Ġwritten Ġby Ġhumans Ġinstead Ġof Ġjust Ġspans Ġfrom Ġthe Ġcontext , Ġthese Ġabstract ive Ġanswers Ġcan Ġmake Ġit Ġeven Ġharder Ġfor Ġcurrent Ġmodels Ġto Ġhandle . ĠWe Ġalso Ġobserve Ġthat Ġwhen Ġpeople Ġwrite ĠâĢ ľ Why âĢ Ŀ Ġquestions , Ġthey Ġtend Ġto Ġcopy Ġword Ġspans Ġfrom Ġthe Ġtweet , Ġpotentially Ġmaking Ġthe Ġtask Ġeasier Ġfor Ġthe Ġmatching Ġbaseline .