 `Conversations Gone Awry' dataset, subreddit ChangeMyView
EXTRACTIVE
An expanded version of the existing 'Conversations Gone Awry' dataset and the ChangeMyView dataset, a subreddit whose only annotation is whether the conversation required action by the Reddit moderators. 
ABSTRACTIVE
To test the effectiveness of this new architecture in forecasting derailment of online conversations, we develop and distribute two new datasets. The first triples in size the highly curated `Conversations Gone Awry' dataset BIBREF9, where civil-starting Wikipedia Talk Page conversations are crowd-labeled according to whether they eventually lead to personal attacks; the second relies on in-the-wild moderation of the popular subreddit ChangeMyView, where the aim is to forecast whether a discussion will later be subject to moderator action due to “rude or hostile” behavior. In both datasets, our model outperforms existing fixed-window approaches, as well as simpler sequential baselines that cannot account for inter-comment relations. Furthermore, by virtue of its online processing of the conversation, our system can provide substantial prior notice of upcoming derailment, triggering on average 3 comments (or 3 hours) before an overtly toxic comment is posted.
We consider two datasets, representing related but slightly different forecasting tasks. The first dataset is an expanded version of the annotated Wikipedia conversations dataset from BIBREF9. This dataset uses carefully-controlled crowdsourced labels, strictly filtered to ensure the conversations are civil up to the moment of a personal attack. This is a useful property for the purposes of model analysis, and hence we focus on this as our primary dataset. However, we are conscious of the possibility that these strict labels may not fully capture the kind of behavior that moderators care about in practice. We therefore introduce a secondary dataset, constructed from the subreddit ChangeMyView (CMV) that does not use post-hoc annotations. Instead, the prediction task is to forecast whether the conversation will be subject to moderator action in the future. To test the effectiveness of this new architecture in forecasting derailment of online conversations, we develop and distribute two new datasets. The first triples in size the highly curated `Conversations Gone Awry' dataset BIBREF9, where civil-starting Wikipedia Talk Page conversations are crowd-labeled according to whether they eventually lead to personal attacks; the second relies on in-the-wild moderation of the popular subreddit ChangeMyView, where the aim is to forecast whether a discussion will later be subject to moderator action due to “rude or hostile” behavior. In both datasets, our model outperforms existing fixed-window approaches, as well as simpler sequential baselines that cannot account for inter-comment relations. Furthermore, by virtue of its online processing of the conversation, our system can provide substantial prior notice of upcoming derailment, triggering on average 3 comments (or 3 hours) before an overtly toxic comment is posted.
1909.01362
Introduction </s> âĢ ľ Ch Ã© Ġsa etta Ġpre vis a Ġv ien Ġpi Ã ¹ Ġlent a . âĢ Ŀ </s> âĢĵ ĠDante ĠAl igh ieri , ĠDiv ina ĠCom media , ĠParad iso </s> Ant is ocial Ġbehavior Ġis Ġa Ġpersistent Ġproblem Ġplag uing Ġonline Ġconversation Ġplatforms ; Ġit Ġis Ġboth Ġwidespread ĠB IB REF 0 Ġand Ġpotentially Ġdamaging Ġto Ġmental Ġand Ġemotional Ġhealth ĠB IB REF 1 , ĠB IB REF 2 . ĠThe Ġstrain Ġthis Ġphenomenon Ġputs Ġon Ġcommunity Ġmaintain ers Ġhas Ġsparked Ġrecent Ġinterest Ġin Ġcomputational Ġapproaches Ġfor Ġassisting Ġhuman Ġmoderators . </s> Prior Ġwork Ġin Ġthis Ġdirection Ġhas Ġlargely Ġfocused Ġon Ġpost - h oc Ġidentification Ġof Ġvarious Ġkinds Ġof Ġantis ocial Ġbehavior , Ġincluding Ġhate Ġspeech ĠB IB REF 3 , ĠB IB REF 4 , Ġharassment ĠB IB REF 5 , Ġpersonal Ġattacks ĠB IB REF 6 , Ġand Ġgeneral Ġtoxicity ĠB IB REF 7 . ĠThe Ġfact Ġthat Ġthese Ġapproaches Ġonly Ġidentify Ġantis ocial Ġcontent Ġafter Ġthe Ġfact Ġlimits Ġtheir Ġpractical ity Ġas Ġtools Ġfor Ġassisting Ġpre - empt ive Ġmoderation Ġin Ġconvers ational Ġdomains . </s> Add ressing Ġthis Ġlimitation Ġrequires Ġforecasting Ġthe Ġfuture Ġderail ment Ġof Ġa Ġconversation Ġbased Ġon Ġearly Ġwarning Ġsigns , Ġgiving Ġthe Ġmoderators Ġtime Ġto Ġpotentially Ġintervene Ġbefore Ġany Ġharm Ġis Ġdone Ġ( BI B REF 8 ĠB IB REF 8 , ĠB IB REF 9 ĠB IB REF 9 , Ġsee ĠB IB REF 10 ĠB IB REF 10 Ġfor Ġa Ġdiscussion ). ĠSuch Ġa Ġgoal Ġrecognizes Ġderail ment Ġas Ġemerging Ġfrom Ġthe Ġdevelopment Ġof Ġthe Ġconversation , Ġand Ġbelongs Ġto Ġthe Ġbroader Ġarea Ġof Ġconvers ational Ġforecasting , Ġwhich Ġincludes Ġfuture - pred iction Ġtasks Ġsuch Ġas Ġpredicting Ġthe Ġeventual Ġlength Ġof Ġa Ġconversation ĠB IB REF 11 , Ġwhether Ġa Ġpersuasion Ġattempt Ġwill Ġeventually Ġsucceed ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 14 , Ġwhether Ġteam Ġdiscussions Ġwill Ġeventually Ġlead Ġto Ġan Ġincrease Ġin Ġperformance ĠB IB REF 15 , Ġor Ġwhether Ġongoing Ġcounseling Ġconversations Ġwill Ġeventually Ġbe Ġperceived Ġas Ġhelpful ĠB IB REF 16 . </s> App ro aching Ġsuch Ġconvers ational Ġforecasting Ġproblems , Ġhowever , Ġrequires Ġovercoming Ġseveral Ġinherent Ġmodeling Ġchallenges . ĠFirst , Ġconversations Ġare Ġdynamic Ġand Ġtheir Ġoutcome Ġmight Ġdepend Ġon Ġhow Ġsubsequent Ġcomments Ġinteract Ġwith Ġeach Ġother . ĠConsider Ġthe Ġexample Ġin ĠFigure ĠFIG REF 2 : Ġwhile Ġno Ġindividual Ġcomment Ġis Ġoutright Ġoffensive , Ġa Ġhuman Ġreader Ġcan Ġsense Ġa Ġtension Ġemerging Ġfrom Ġtheir Ġsuccession Ġ( e . g ., Ġdismissive Ġanswers Ġto Ġrepeated Ġquestioning ). ĠThus Ġa Ġforecasting Ġmodel Ġneeds Ġto Ġcapture Ġnot Ġonly Ġthe Ġcontent Ġof Ġeach Ġindividual Ġcomment , Ġbut Ġalso Ġthe Ġrelations Ġbetween Ġcomments . ĠPrevious Ġwork Ġhas Ġlargely Ġrelied Ġon Ġhand - crafted Ġfeatures Ġto Ġcapture Ġsuch Ġrelations âĢĶ e . g ., Ġsimilarity Ġbetween Ġcomments ĠB IB REF 16 , ĠB IB REF 12 Ġor Ġconversation Ġstructure ĠB IB REF 17 , ĠB IB REF 18 âĢĶ , Ġthough Ġneural Ġattention Ġarchitectures Ġhave Ġalso Ġrecently Ġshown Ġpromise ĠB IB REF 19 . </s> The Ġsecond Ġmodeling Ġchallenge Ġstems Ġfrom Ġthe Ġfact Ġthat Ġconversations Ġhave Ġan Ġunknown Ġhorizon : Ġthey Ġcan Ġbe Ġof Ġvarying Ġlengths , Ġand Ġthe Ġto - be - fore cast ed Ġevent Ġcan Ġoccur Ġat Ġany Ġtime . ĠSo Ġwhen Ġis Ġit Ġa Ġgood Ġtime Ġto Ġmake Ġa Ġforecast ? ĠPrior Ġwork Ġhas Ġlargely Ġproposed Ġtwo Ġsolutions , Ġboth Ġresulting Ġin Ġimportant Ġpractical Ġlimitations . ĠOne Ġsolution Ġis Ġto Ġassume Ġ( un real istic ) Ġprior Ġknowledge Ġof Ġwhen Ġthe Ġto - be - fore cast ed Ġevent Ġtakes Ġplace Ġand Ġextract Ġfeatures Ġup Ġto Ġthat Ġpoint ĠB IB REF 20 , ĠB IB REF 8 . ĠAnother Ġcompromising Ġsolution Ġis Ġto Ġextract Ġfeatures Ġfrom Ġa Ġfixed - length Ġwindow , Ġoften Ġat Ġthe Ġstart Ġof Ġthe Ġconversation ĠB IB REF 21 , ĠB IB REF 15 , ĠB IB REF 16 , ĠB IB REF 9 . ĠCho osing Ġa Ġcatch - all Ġwindow - size Ġis Ġhowever Ġimpractical : Ġshort Ġwindows Ġwill Ġmiss Ġinformation Ġin Ġcomments Ġthey Ġdo Ġnot Ġencompass Ġ( e . g ., Ġa Ġwindow Ġof Ġonly Ġtwo Ġcomments Ġwould Ġmiss Ġthe Ġchain Ġof Ġrepeated Ġquestioning Ġin Ġcomments Ġ3 Ġthrough Ġ6 Ġof ĠFigure ĠFIG REF 2 ), Ġwhile Ġlonger Ġwindows Ġrisk Ġmissing Ġthe Ġto - be - fore cast ed Ġevent Ġaltogether Ġif Ġit Ġoccurs Ġbefore Ġthe Ġend Ġof Ġthe Ġwindow , Ġwhich Ġwould Ġprevent Ġearly Ġdetection . </s> In Ġthis Ġwork Ġwe Ġintroduce Ġa Ġmodel Ġfor Ġforecasting Ġconvers ational Ġevents Ġthat Ġover comes Ġboth Ġthese Ġinherent Ġchallenges Ġby Ġprocessing Ġcomments , Ġand Ġtheir Ġrelations , Ġas Ġthey Ġhappen Ġ( i . e ., Ġin Ġan Ġonline Ġfashion ). ĠOur Ġmain Ġinsight Ġis Ġthat Ġmodels Ġwith Ġthese Ġproperties Ġalready Ġexist , Ġalbeit Ġgeared Ġtoward Ġgeneration Ġrather Ġthan Ġprediction : Ġrecent Ġwork Ġin Ġcontext - aware Ġdialog Ġgeneration Ġ( or ĠâĢ ľ chat bots âĢ Ŀ ) Ġhas Ġproposed Ġsequential Ġneural Ġmodels Ġthat Ġmake Ġeffective Ġuse Ġof Ġthe Ġintra - con vers ational Ġdynamics ĠB IB REF 22 , ĠB IB REF 23 , ĠB IB REF 24 , Ġwhile Ġcon com it antly Ġbeing Ġable Ġto Ġprocess Ġthe Ġconversation Ġas Ġit Ġdevelops Ġ( see ĠB IB REF 25 Ġfor Ġa Ġsurvey ). </s> In Ġorder Ġfor Ġthese Ġsystems Ġto Ġperform Ġwell Ġin Ġthe Ġgener ative Ġdomain Ġthey Ġneed Ġto Ġbe Ġtrained Ġon Ġmassive Ġamounts Ġof Ġ( un label ed ) Ġconvers ational Ġdata . ĠThe Ġmain Ġdifficulty Ġin Ġdirectly Ġadapting Ġthese Ġmodels Ġto Ġthe Ġsupervised Ġdomain Ġof Ġconvers ational Ġforecasting Ġis Ġthe Ġrelative Ġscarcity Ġof Ġlabeled Ġdata : Ġfor Ġmost Ġforecasting Ġtasks , Ġat Ġmost Ġa Ġfew Ġthousands Ġlabeled Ġexamples Ġare Ġavailable , Ġinsufficient Ġfor Ġthe Ġnotoriously Ġdata - hung ry Ġsequential Ġneural Ġmodels . </s> To Ġovercome Ġthis Ġdifficulty , Ġwe Ġpropose Ġto Ġdec ouple Ġthe Ġobjective Ġof Ġlearning Ġa Ġneural Ġrepresentation Ġof Ġconvers ational Ġdynamics Ġfrom Ġthe Ġobjective Ġof Ġpredicting Ġfuture Ġevents . ĠThe Ġformer Ġcan Ġbe Ġpre - trained Ġon Ġlarge Ġamounts Ġof Ġun super vised Ġdata , Ġsimilarly Ġto Ġhow Ġchat bots Ġare Ġtrained . ĠThe Ġlatter Ġcan Ġpig gy - back Ġon Ġthe Ġresulting Ġrepresentation Ġafter Ġfine - tun ing Ġit Ġfor Ġclassification Ġusing Ġrelatively Ġsmall Ġlabeled Ġdata . ĠWhile Ġsimilar Ġpre - train - then - fine - t une Ġapproaches Ġhave Ġrecently Ġachieved Ġstate - of - the - art Ġperformance Ġin Ġa Ġnumber Ġof ĠN LP Ġtasks âĢĶ including Ġnatural Ġlanguage Ġinference , Ġquestion Ġanswering , Ġand Ġcommons ense Ġreasoning Ġ( disc ussed Ġin ĠSection ĠSEC REF 2 )âĢĶ to Ġthe Ġbest Ġof Ġour Ġknowledge Ġthis Ġis Ġthe Ġfirst Ġattempt Ġat Ġapplying Ġthis Ġparadigm Ġto Ġconvers ational Ġforecasting . </s> To Ġtest Ġthe Ġeffectiveness Ġof Ġthis Ġnew Ġarchitecture Ġin Ġforecasting Ġderail ment Ġof Ġonline Ġconversations , Ġwe Ġdevelop Ġand Ġdistribute Ġtwo Ġnew Ġdatasets . ĠThe Ġfirst Ġtri ples Ġin Ġsize Ġthe Ġhighly Ġcurated Ġ` Con vers ations ĠGone ĠAw ry ' Ġdataset ĠB IB REF 9 , Ġwhere Ġcivil - starting ĠWikipedia ĠTalk ĠPage Ġconversations Ġare Ġcrowd - label ed Ġaccording Ġto Ġwhether Ġthey Ġeventually Ġlead Ġto Ġpersonal Ġattacks ; Ġthe Ġsecond Ġrelies Ġon Ġin - the - wild Ġmoderation Ġof Ġthe Ġpopular Ġsubreddit ĠChange My View , Ġwhere Ġthe Ġaim Ġis Ġto Ġforecast Ġwhether Ġa Ġdiscussion Ġwill Ġlater Ġbe Ġsubject Ġto Ġmoderator Ġaction Ġdue Ġto ĠâĢ ľ r ude Ġor Ġhostile âĢ Ŀ Ġbehavior . ĠIn Ġboth Ġdatasets , Ġour Ġmodel Ġoutper forms Ġexisting Ġfixed - window Ġapproaches , Ġas Ġwell Ġas Ġsimpler Ġsequential Ġbas elines Ġthat Ġcannot Ġaccount Ġfor Ġinter - comment Ġrelations . ĠFurthermore , Ġby Ġvirtue Ġof Ġits Ġonline Ġprocessing Ġof Ġthe Ġconversation , Ġour Ġsystem Ġcan Ġprovide Ġsubstantial Ġprior Ġnotice Ġof Ġupcoming Ġderail ment , Ġtriggering Ġon Ġaverage Ġ3 Ġcomments Ġ( or Ġ3 Ġhours ) Ġbefore Ġan Ġovertly Ġtoxic Ġcomment Ġis Ġposted . </s> To Ġsummarize , Ġin Ġthis Ġwork Ġwe : </s> introdu ce Ġthe Ġfirst Ġmodel Ġfor Ġforecasting Ġconvers ational Ġevents Ġthat Ġcan Ġcapture Ġthe Ġdynamics Ġof Ġa Ġconversation Ġas Ġit Ġdevelops ; </s> build Ġtwo Ġdiverse Ġdatasets Ġ( one Ġentirely Ġnew , Ġone Ġextending Ġprior Ġwork ) Ġfor Ġthe Ġtask Ġof Ġforecasting Ġderail ment Ġof Ġonline Ġconversations ; </s> comp are Ġthe Ġperformance Ġof Ġour Ġmodel Ġagainst Ġthe Ġcurrent Ġstate - of - the - art , Ġand Ġevaluate Ġits Ġability Ġto Ġprovide Ġearly Ġwarning Ġsigns . </s> Our Ġwork Ġis Ġmotivated Ġby Ġthe Ġgoal Ġof Ġassisting Ġhuman Ġmoderators Ġof Ġonline Ġcommunities Ġby Ġpreempt ively Ġsignaling Ġat - risk Ġconversations Ġthat Ġmight Ġdeserve Ġtheir Ġattention . ĠHowever , Ġwe Ġcaution Ġthat Ġany Ġautomated Ġsystems Ġmight Ġencode Ġor Ġeven Ġamplify Ġthe Ġbiases Ġexisting Ġin Ġthe Ġtraining Ġdata ĠB IB REF 26 , ĠB IB REF 27 , ĠB IB REF 28 , Ġso Ġa Ġpublic - facing Ġimplementation Ġwould Ġneed Ġto Ġbe Ġexhaust ively Ġscrutin ized Ġfor Ġsuch Ġbiases ĠB IB REF 29 . </s> Further ĠRelated ĠWork </s> Ant is ocial Ġbehavior . ĠAnt is ocial Ġbehavior Ġonline Ġcomes Ġin Ġmany Ġforms , Ġincluding Ġharassment ĠB IB REF 30 , Ġcyber bull ying ĠB IB REF 31 , Ġand Ġgeneral Ġaggression ĠB IB REF 32 . ĠPrior Ġwork Ġhas Ġsought Ġto Ġunderstand Ġdifferent Ġaspects Ġof Ġsuch Ġbehavior , Ġincluding Ġits Ġeffect Ġon Ġthe Ġcommunities Ġwhere Ġit Ġhappens ĠB IB REF 33 , ĠB IB REF 34 , Ġthe Ġactors Ġinvolved ĠB IB REF 35 , ĠB IB REF 36 , ĠB IB REF 37 , ĠB IB REF 38 Ġand Ġconnections Ġto Ġthe Ġoutside Ġworld ĠB IB REF 39 . </s> Post - h oc Ġclassification Ġof Ġconversations . ĠThere Ġis Ġa Ġrich Ġbody Ġof Ġprior Ġwork Ġon Ġclass ifying Ġthe Ġoutcome Ġof Ġa Ġconversation Ġafter Ġit Ġhas Ġconcluded , Ġor Ġclass ifying Ġconvers ational Ġevents Ġafter Ġthey Ġhappened . ĠMany Ġexamples Ġexist , Ġbut Ġsome Ġmore Ġclosely Ġrelated Ġto Ġour Ġpresent Ġwork Ġinclude Ġidentifying Ġthe Ġwinner Ġof Ġa Ġdebate ĠB IB REF 40 , ĠB IB REF 41 , ĠB IB REF 42 , Ġidentifying Ġsuccessful Ġnegotiations ĠB IB REF 21 , ĠB IB REF 43 , Ġas Ġwell Ġas Ġdetecting Ġwhether Ġdeception ĠB IB REF 44 , ĠB IB REF 45 , ĠB IB REF 46 Ġor Ġdisagreement ĠB IB REF 47 , ĠB IB REF 48 , ĠB IB REF 49 , ĠB IB REF 50 , ĠB IB REF 51 Ġhas Ġoccurred . </s> Our Ġgoal Ġis Ġdifferent Ġbecause Ġwe Ġwish Ġto Ġforecast Ġconvers ational Ġevents Ġbefore Ġthey Ġhappen Ġand Ġwhile Ġthe Ġconversation Ġis Ġstill Ġongoing Ġ( pot entially Ġallowing Ġfor Ġinterventions ). ĠNote Ġthat Ġsome Ġpost - h oc Ġtasks Ġcan Ġalso Ġbe Ġre - fram ed Ġas Ġforecasting Ġtasks Ġ( assuming Ġthe Ġexistence Ġof Ġnecessary Ġlabels ); Ġfor Ġinstance , Ġpredicting Ġwhether Ġan Ġongoing Ġconversation Ġwill Ġeventually Ġspark Ġdisagreement ĠB IB REF 18 , Ġrather Ġthan Ġdetecting Ġalready - existing Ġdisagreement . </s> Con vers ational Ġforecasting . ĠAs Ġdescribed Ġin ĠSection ĠSEC REF 1 , Ġprior Ġwork Ġon Ġforecasting Ġconvers ational Ġoutcomes Ġand Ġevents Ġhas Ġlargely Ġrelied Ġon Ġhand - crafted Ġfeatures Ġto Ġcapture Ġaspects Ġof Ġconvers ational Ġdynamics . ĠExample Ġfeature Ġsets Ġinclude Ġstatistical Ġmeasures Ġbased Ġon Ġsimilarity Ġbetween Ġutter ances ĠB IB REF 16 , Ġsentiment Ġimbalance ĠB IB REF 20 , Ġflow Ġof Ġideas ĠB IB REF 20 , Ġincrease Ġin Ġhostility ĠB IB REF 8 , Ġreply Ġrate ĠB IB REF 11 Ġand Ġgraph Ġrepresentations Ġof Ġconversations ĠB IB REF 52 , ĠB IB REF 17 . ĠBy Ġcontrast , Ġwe Ġaim Ġto Ġautomatically Ġlearn Ġneural Ġrepresentations Ġof Ġconvers ational Ġdynamics Ġthrough Ġpre - training . </s> Such Ġhand - crafted Ġfeatures Ġare Ġtypically Ġextracted Ġfrom Ġfixed - length Ġwindows Ġof Ġthe Ġconversation , Ġleaving Ġun add ressed Ġthe Ġproblem Ġof Ġunknown Ġhorizon . ĠWhile Ġsome Ġwork Ġhas Ġtrained Ġmultiple Ġmodels Ġfor Ġdifferent Ġwindow - length s ĠB IB REF 8 , ĠB IB REF 18 , Ġthey Ġconsider Ġthese Ġmodels Ġto Ġbe Ġindependent Ġand , Ġas Ġsuch , Ġdo Ġnot Ġaddress Ġthe Ġissue Ġof Ġaggreg ating Ġthem Ġinto Ġa Ġsingle Ġforecast Ġ( i . e ., Ġdeciding Ġat Ġwhat Ġpoint Ġto Ġmake Ġa Ġprediction ). ĠWe Ġimplement Ġa Ġsimple Ġsliding Ġwindows Ġsolution Ġas Ġa Ġbaseline Ġ( Section ĠSEC REF 5 ). </s> Pre - training Ġfor ĠN LP . ĠThe Ġuse Ġof Ġpre - training Ġfor Ġnatural Ġlanguage Ġtasks Ġhas Ġbeen Ġgrowing Ġin Ġpopularity Ġafter Ġrecent Ġbreakthrough s Ġdemonstrating Ġimproved Ġperformance Ġon Ġa Ġwide Ġarray Ġof Ġbenchmark Ġtasks ĠB IB REF 53 , ĠB IB REF 54 . ĠEx isting Ġwork Ġhas Ġgenerally Ġused Ġa Ġlanguage Ġmodeling Ġobjective Ġas Ġthe Ġpre - training Ġobjective ; Ġexamples Ġinclude Ġnext - word Ġprediction ĠB IB REF 55 , Ġsentence Ġaut oen c oding , ĠB IB REF 56 , Ġand Ġmachine Ġtranslation ĠB IB REF 57 . ĠB ERT ĠB IB REF 58 Ġintroduces Ġa Ġvariation Ġon Ġthis Ġin Ġwhich Ġthe Ġgoal Ġis Ġto Ġpredict Ġthe Ġnext Ġsentence Ġin Ġa Ġdocument Ġgiven Ġthe Ġcurrent Ġsentence . ĠOur Ġpre - training Ġobjective Ġis Ġsimilar Ġin Ġspirit , Ġbut Ġoperates Ġat Ġa Ġconversation Ġlevel , Ġrather Ġthan Ġa Ġdocument Ġlevel . ĠWe Ġhence Ġview Ġour Ġobjective Ġas Ġconvers ational Ġmodeling Ġrather Ġthan Ġ( only ) Ġlanguage Ġmodeling . ĠFurthermore , Ġwhile ĠB ERT 's Ġsentence Ġprediction Ġobjective Ġis Ġframed Ġas Ġa Ġmultiple - choice Ġtask , Ġour Ġobjective Ġis Ġframed Ġas Ġa Ġgener ative Ġtask . </s> Der ail ment ĠDat as ets </s> We Ġconsider Ġtwo Ġdatasets , Ġrepresenting Ġrelated Ġbut Ġslightly Ġdifferent Ġforecasting Ġtasks . ĠThe Ġfirst Ġdataset Ġis Ġan Ġexpanded Ġversion Ġof Ġthe Ġannot ated ĠWikipedia Ġconversations Ġdataset Ġfrom ĠB IB REF 9 . ĠThis Ġdataset Ġuses Ġcarefully - controlled Ġcrowds ourced Ġlabels , Ġstrictly Ġfiltered Ġto Ġensure Ġthe Ġconversations Ġare Ġcivil Ġup Ġto Ġthe Ġmoment Ġof Ġa Ġpersonal Ġattack . ĠThis Ġis Ġa Ġuseful Ġproperty Ġfor Ġthe Ġpurposes Ġof Ġmodel Ġanalysis , Ġand Ġhence Ġwe Ġfocus Ġon Ġthis Ġas Ġour Ġprimary Ġdataset . ĠHowever , Ġwe Ġare Ġconscious Ġof Ġthe Ġpossibility Ġthat Ġthese Ġstrict Ġlabels Ġmay Ġnot Ġfully Ġcapture Ġthe Ġkind Ġof Ġbehavior Ġthat Ġmoderators Ġcare Ġabout Ġin Ġpractice . ĠWe Ġtherefore Ġintroduce Ġa Ġsecondary Ġdataset , Ġconstructed Ġfrom Ġthe Ġsubreddit ĠChange My View Ġ( CM V ) Ġthat Ġdoes Ġnot Ġuse Ġpost - h oc Ġannotations . ĠInstead , Ġthe Ġprediction Ġtask Ġis Ġto Ġforecast Ġwhether Ġthe Ġconversation Ġwill Ġbe Ġsubject Ġto Ġmoderator Ġaction Ġin Ġthe Ġfuture . </s> Wikipedia Ġdata . ĠB IB REF 9 's Ġ` Con vers ations ĠGone ĠAw ry ' Ġdataset Ġconsists Ġof Ġ1 , 270 Ġconversations Ġthat Ġtook Ġplace Ġbetween ĠWikipedia Ġeditors Ġon Ġpublicly Ġaccessible Ġtalk Ġpages . ĠThe Ġconversations Ġare Ġsourced Ġfrom Ġthe ĠWiki Con v Ġdataset ĠB IB REF 59 Ġand Ġlabeled Ġby Ġcrowd workers Ġas Ġeither Ġcontaining Ġa Ġpersonal Ġattack Ġfrom Ġwithin Ġ( i . e ., Ġhostile Ġbehavior Ġby Ġone Ġuser Ġin Ġthe Ġconversation Ġdirected Ġtowards Ġanother ) Ġor Ġremaining Ġcivil Ġthroughout . </s> A Ġseries Ġof Ġcontrols Ġare Ġimplemented Ġto Ġprevent Ġmodels Ġfrom Ġpicking Ġup Ġon Ġtrivial Ġcorrelations . ĠTo Ġprevent Ġmodels Ġfrom Ġcapturing Ġtopic - specific Ġinformation Ġ( e . g ., Ġpolitical Ġconversations Ġare Ġmore Ġlikely Ġto Ġderail ), Ġeach Ġattack - containing Ġconversation Ġis Ġpaired Ġwith Ġa Ġclean Ġconversation Ġfrom Ġthe Ġsame Ġtalk Ġpage , Ġwhere Ġthe Ġtalk Ġpage Ġserves Ġas Ġa Ġproxy Ġfor Ġtopic . ĠTo Ġforce Ġmodels Ġto Ġactually Ġcapture Ġconvers ational Ġdynamics Ġrather Ġthan Ġdetecting Ġalready - existing Ġtoxicity , Ġhuman Ġannotations Ġare Ġused Ġto Ġensure Ġthat Ġall Ġcomments Ġpreceding Ġa Ġpersonal Ġattack Ġare Ġcivil . </s> To Ġthe Ġends Ġof Ġmore Ġeffective Ġmodel Ġtraining , Ġwe Ġelected Ġto Ġexpand Ġthe Ġ` Con vers ations ĠGone ĠAw ry ' Ġdataset , Ġusing Ġthe Ġoriginal Ġannotation Ġprocedure . ĠSince Ġwe Ġfound Ġthat Ġthe Ġoriginal Ġdata Ġskewed Ġtowards Ġshorter Ġconversations , Ġwe Ġfocused Ġthis Ġcrowds ourcing Ġrun Ġon Ġlonger Ġconversations : Ġones Ġwith Ġ4 Ġor Ġmore Ġcomments Ġpreceding Ġthe Ġattack . ĠThrough Ġthis Ġadditional Ġcrowds ourcing , Ġwe Ġexpand Ġthe Ġdataset Ġto Ġ4 , 188 Ġconversations , Ġwhich Ġwe Ġare Ġpublicly Ġreleasing Ġas Ġpart Ġof Ġthe ĠCornell ĠConvers ational ĠAnalysis ĠTool kit Ġ( Con vo Kit ). </s> We Ġperform Ġan Ġ80 - 20 - 20 Ġtrain / dev / test Ġsplit , Ġensuring Ġthat Ġpaired Ġconversations Ġend Ġup Ġin Ġthe Ġsame Ġsplit Ġin Ġorder Ġto Ġpreserve Ġthe Ġtopic Ġcontrol . ĠFinally , Ġwe Ġrandomly Ġsample Ġanother Ġ1 Ġmillion Ġconversations Ġfrom ĠWiki Con v Ġto Ġuse Ġfor Ġthe Ġun super vised Ġpre - training Ġof Ġthe Ġgener ative Ġcomponent . </s> Reddit ĠCM V Ġdata . ĠThe ĠCM V Ġdataset Ġis Ġconstructed Ġfrom Ġconversations Ġcollected Ġvia Ġthe ĠReddit ĠAPI . ĠIn Ġcontrast Ġto Ġthe ĠWikipedia - based Ġdataset , Ġwe Ġexplicitly Ġavoid Ġthe Ġuse Ġof Ġpost - h oc Ġannotation . ĠInstead , Ġwe Ġuse Ġas Ġour Ġlabel Ġwhether Ġa Ġconversation Ġeventually Ġhad Ġa Ġcomment Ġremoved Ġby Ġa Ġmoderator Ġfor Ġviolation Ġof ĠRule Ġ2 : ĠâĢ ľ Don 't Ġbe Ġrude Ġor Ġhostile Ġto Ġother Ġusers âĢ Ŀ . </s> Though Ġthe Ġlack Ġof Ġpost - h oc Ġannotation Ġlimits Ġthe Ġdegree Ġto Ġwhich Ġwe Ġcan Ġimpose Ġcontrols Ġon Ġthe Ġdata Ġ( e . g ., Ġsome Ġconversations Ġmay Ġcontain Ġtoxic Ġcomments Ġnot Ġflagged Ġby Ġthe Ġmoderators ) Ġwe Ġdo Ġreproduce Ġas Ġmany Ġof Ġthe ĠWikipedia Ġdata 's Ġcontrols Ġas Ġwe Ġcan . ĠName ly , Ġwe Ġreplicate Ġthe Ġtopic Ġcontrol Ġpairing Ġby Ġchoosing Ġpairs Ġof Ġpositive Ġand Ġnegative Ġexamples Ġthat Ġbelong Ġto Ġthe Ġsame Ġtop - level Ġpost , Ġfollowing ĠB IB REF 12 ; Ġand Ġenforce Ġthat Ġthe Ġremoved Ġcomment Ġwas Ġmade Ġby Ġa Ġuser Ġwho Ġwas Ġpreviously Ġinvolved Ġin Ġthe Ġconversation . ĠThis Ġprocess Ġresults Ġin Ġ6 , 8 42 Ġconversations , Ġto Ġwhich Ġwe Ġagain Ġapply Ġa Ġpair - pres erving Ġ80 - 20 - 20 Ġsplit . ĠFinally , Ġwe Ġgather Ġover Ġ600 , 000 Ġconversations Ġthat Ġdo Ġnot Ġinclude Ġany Ġremoved Ġcomment , Ġfor Ġun super vised Ġpre - training . </s> Con vers ational ĠFore casting ĠModel </s> We Ġnow Ġdescribe Ġour Ġgeneral Ġmodel Ġfor Ġforecasting Ġfuture Ġconvers ational Ġevents . ĠOur Ġmodel Ġintegrates Ġtwo Ġcomponents : Ġ( a ) Ġa Ġgener ative Ġdialog Ġmodel Ġthat Ġlearns Ġto Ġrepresent Ġconvers ational Ġdynamics Ġin Ġan Ġun super vised Ġfashion ; Ġand Ġ( b ) Ġa Ġsupervised Ġcomponent Ġthat Ġfine - t unes Ġthis Ġrepresentation Ġto Ġforecast Ġfuture Ġevents . ĠFigure ĠFIG REF 13 Ġprovides Ġan Ġoverview Ġof Ġthe Ġproposed Ġarchitecture , Ġhence forth ĠCRA FT Ġ( Con vers ational ĠRec urrent ĠArchitecture Ġfor ĠFore cas T ing ). </s> Termin ology . ĠFor Ġmodeling Ġpurposes , Ġwe Ġtreat Ġa Ġconversation Ġas Ġa Ġsequence Ġof Ġ$ N $ Ġcomments Ġ$ C Ġ= Ġ\ l brace Ġc _ 1 , \ d ots Ġ, c _ N \ r brace Ġ$ . ĠEach Ġcomment , Ġin Ġturn , Ġis Ġa Ġsequence Ġof Ġtokens , Ġwhere Ġthe Ġnumber Ġof Ġtokens Ġmay Ġvary Ġfrom Ġcomment Ġto Ġcomment . ĠFor Ġthe Ġ$ n $ - th Ġcomment Ġ($ 1 Ġ\ le Ġn Ġ\ le ĠN ) $, Ġwe Ġlet Ġ$ M _ n $ Ġdenote Ġthe Ġnumber Ġof Ġtokens . ĠThen , Ġa Ġcomment Ġ$ c _ n $ Ġcan Ġbe Ġrepresented Ġas Ġa Ġsequence Ġof Ġ$ M _ n $ Ġtokens : Ġ$ c _ n Ġ= Ġ\ l brace Ġw _ 1 , \ d ots Ġ, w _{ M _ n }\ r brace Ġ$ . </s> Gener ative Ġcomponent . ĠFor Ġthe Ġgener ative Ġcomponent Ġof Ġour Ġmodel , Ġwe Ġuse Ġa Ġhierarchical Ġrecurrent Ġenc oder - dec oder Ġ( HR ED ) Ġarchitecture ĠB IB REF 60 , Ġa Ġmodified Ġversion Ġof Ġthe Ġpopular Ġsequence - to - sequence Ġ( seq 2 seq ) Ġarchitecture ĠB IB REF 61 Ġdesigned Ġto Ġaccount Ġfor Ġdependencies Ġbetween Ġconsecutive Ġinputs . ĠB IB REF 23 Ġshowed Ġthat ĠHR ED Ġcan Ġsuccessfully Ġmodel Ġconvers ational Ġcontext Ġby Ġencoding Ġthe Ġtemporal Ġstructure Ġof Ġpreviously Ġseen Ġcomments , Ġmaking Ġit Ġan Ġideal Ġfit Ġfor Ġour Ġuse Ġcase . ĠHere , Ġwe Ġprovide Ġa Ġhigh - level Ġsummary Ġof Ġthe ĠHR ED Ġarchitecture , Ġdefer ring Ġdeeper Ġtechnical Ġdiscussion Ġto ĠB IB REF 60 Ġand ĠB IB REF 23 . </s> An ĠHR ED Ġdialog Ġmodel Ġconsists Ġof Ġthree Ġcomponents : Ġan Ġutter ance Ġenc oder , Ġa Ġcontext Ġenc oder , Ġand Ġa Ġdec oder . ĠThe Ġutter ance Ġenc oder Ġis Ġresponsible Ġfor Ġgenerating Ġsemantic Ġvector Ġrepresentations Ġof Ġcomments . ĠIt Ġconsists Ġof Ġa Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġthat Ġreads Ġa Ġcomment Ġtoken - by - token , Ġand Ġon Ġeach Ġtoken Ġ$ w _ m $ Ġupdates Ġa Ġhidden Ġstate Ġ$ h ^ {\ text { enc }} $ Ġbased Ġon Ġthe Ġcurrent Ġtoken Ġand Ġthe Ġprevious Ġhidden Ġstate : </s> where Ġ$ f ^ {\ text { R NN }} $ Ġis Ġa Ġnon linear Ġg ating Ġfunction Ġ( our Ġimplementation Ġuses ĠGR U ĠB IB REF 62 ). ĠThe Ġfinal Ġhidden Ġstate Ġ$ h ^ {\ text { enc }} _ M $ Ġcan Ġbe Ġviewed Ġas Ġa Ġvector Ġencoding Ġof Ġthe Ġentire Ġcomment . </s> Running Ġthe Ġenc oder Ġon Ġeach Ġcomment Ġ$ c _ n $ Ġresults Ġin Ġa Ġsequence Ġof Ġ$ N $ Ġvector Ġenc od ings . ĠA Ġsecond Ġenc oder , Ġthe Ġcontext Ġenc oder , Ġis Ġthen Ġrun Ġover Ġthis Ġsequence : </s> Each Ġhidden Ġstate Ġ$ h ^ {\ text { con }} _ n $ Ġcan Ġthen Ġbe Ġviewed Ġas Ġan Ġencoding Ġof Ġthe Ġfull Ġconvers ational Ġcontext Ġup Ġto Ġand Ġincluding Ġthe Ġ$ n $ - th Ġcomment . ĠTo Ġgenerate Ġa Ġresponse Ġto Ġcomment Ġ$ n $, Ġthe Ġcontext Ġencoding Ġ$ h ^ {\ text { con }} _ n $ Ġis Ġused Ġto Ġinitialize Ġthe Ġhidden Ġstate Ġ$ h ^ {\ text { dec }} _{ 0 } $ Ġof Ġa Ġdec oder ĠR NN . ĠThe Ġdec oder Ġproduces Ġa Ġresponse Ġtoken Ġby Ġtoken Ġusing Ġthe Ġfollowing Ġrec urrence : </s> where Ġ$ f ^ {\ text { out }} $ Ġis Ġsome Ġfunction Ġthat Ġoutputs Ġa Ġprobability Ġdistribution Ġover Ġwords ; Ġwe Ġimplement Ġthis Ġusing Ġa Ġsimple Ġfeed forward Ġlayer . ĠIn Ġour Ġimplementation , Ġwe Ġfurther Ġaugment Ġthe Ġdec oder Ġwith Ġattention ĠB IB REF 63 , ĠB IB REF 64 Ġover Ġcontext Ġenc oder Ġstates Ġto Ġhelp Ġcapture Ġlong - term Ġinter - comment Ġdependencies . ĠThis Ġgener ative Ġcomponent Ġcan Ġbe Ġpre - trained Ġusing Ġunl abel ed Ġconvers ational Ġdata . </s> Pred iction Ġcomponent . ĠGiven Ġa Ġpre - trained ĠHR ED Ġdialog Ġmodel , Ġwe Ġaim Ġto Ġextend Ġthe Ġmodel Ġto Ġpredict Ġfrom Ġthe Ġconvers ational Ġcontext Ġwhether Ġthe Ġto - be - fore cast ed Ġevent Ġwill Ġoccur . ĠOur Ġpredictor Ġconsists Ġof Ġa Ġmult il ayer Ġpercept ron Ġ( ML P ) Ġwith Ġ3 Ġfully - connected Ġlayers , Ġleak y ĠRe LU Ġactiv ations Ġbetween Ġlayers , Ġand Ġs igm oid Ġactivation Ġfor Ġoutput . ĠFor Ġeach Ġcomment Ġ$ c _ n $, Ġthe Ġpredictor Ġtakes Ġas Ġinput Ġthe Ġcontext Ġencoding Ġ$ h ^ {\ text { con }} _ n $ Ġand Ġforwards Ġit Ġthrough Ġthe ĠML P Ġlayers , Ġresulting Ġin Ġan Ġoutput Ġscore Ġthat Ġis Ġinterpreted Ġas Ġa Ġprobability Ġ$ p _{ \ text { event }} ( c _{ n + 1 }) $ Ġthat Ġthe Ġto - be - fore cast ed Ġevent Ġwill Ġhappen Ġ( e . g ., Ġthat Ġthe Ġconversation Ġwill Ġderail ). </s> Training Ġthe Ġpredictive Ġcomponent Ġstarts Ġby Ġinitial izing Ġthe Ġweights Ġof Ġthe Ġenc od ers Ġto Ġthe Ġvalues Ġlearned Ġin Ġpre - training . ĠThe Ġmain Ġtraining Ġloop Ġthen Ġworks Ġas Ġfollows : Ġfor Ġeach Ġpositive Ġsample âĢĶ i . e ., Ġa Ġconversation Ġcontaining Ġan Ġinstance Ġof Ġthe Ġto - be - fore cast ed Ġevent Ġ( e . g ., Ġderail ment ) Ġat Ġcomment Ġ$ c _ e $ âĢĶ we Ġfeed Ġthe Ġcontext Ġ$ c _ 1 , \ d ots Ġ, c _{ e - 1 } $ Ġthrough Ġthe Ġenc oder Ġand Ġclass ifier , Ġand Ġcompute Ġcross - ent ropy Ġloss Ġbetween Ġthe Ġclass ifier Ġoutput Ġand Ġexpected Ġoutput Ġof Ġ1 . ĠSimilarly , Ġfor Ġeach Ġnegative Ġsample âĢĶ i . e ., Ġa Ġconversation Ġwhere Ġnone Ġof Ġthe Ġcomments Ġexhibit Ġthe Ġto - be - fore cast ed Ġevent Ġand Ġthat Ġends Ġwith Ġ$ c _ N $ âĢĶ we Ġfeed Ġthe Ġcontext Ġ$ c _ 1 , \ d ots Ġ, c _{ N - 1 } $ Ġthrough Ġthe Ġmodel Ġand Ġcompute Ġloss Ġagainst Ġan Ġexpected Ġoutput Ġof Ġ0 . </s> Note Ġthat Ġthe Ġparameters Ġof Ġthe Ġgener ative Ġcomponent Ġare Ġnot Ġheld Ġfixed Ġduring Ġthis Ġprocess ; Ġinstead , Ġback prop ag ation Ġis Ġallowed Ġto Ġgo Ġall Ġthe Ġway Ġthrough Ġthe Ġenc oder Ġlayers . ĠThis Ġprocess , Ġknown Ġas Ġfine - tun ing , Ġresh apes Ġthe Ġrepresentation Ġlearned Ġduring Ġpre - training Ġto Ġbe Ġmore Ġdirectly Ġuseful Ġto Ġprediction ĠB IB REF 55 . </s> We Ġimplement Ġthe Ġmodel Ġand Ġtraining Ġcode Ġusing ĠPy Tor ch , Ġand Ġwe Ġare Ġpublicly Ġreleasing Ġour Ġimplementation Ġand Ġthe Ġtrained Ġmodels Ġtogether Ġwith Ġthe Ġdata Ġas Ġpart Ġof ĠCon vo Kit . </s> Fore casting ĠDer ail ment </s> We Ġevaluate Ġthe Ġperformance Ġof ĠCRA FT Ġin Ġthe Ġtask Ġof Ġforecasting Ġconvers ational Ġderail ment Ġin Ġboth Ġthe ĠWikipedia Ġand ĠCM V Ġscenarios . ĠTo Ġthis Ġend , Ġfor Ġeach Ġof Ġthese Ġdatasets Ġwe Ġpre - train Ġthe Ġgener ative Ġcomponent Ġon Ġthe Ġunl abel ed Ġportion Ġof Ġthe Ġdata Ġand Ġfine - t une Ġit Ġon Ġthe Ġlabeled Ġtraining Ġsplit Ġ( data Ġsize Ġdetailed Ġin ĠSection ĠSEC REF 3 ). </s> In Ġorder Ġto Ġevaluate Ġour Ġsequential Ġsystem Ġagainst Ġconvers ational - level Ġground Ġtruth , Ġwe Ġneed Ġto Ġaggregate Ġcomment Ġlevel Ġpredictions . ĠIf Ġany Ġcomment Ġin Ġthe Ġconversation Ġtriggers Ġa Ġpositive Ġprediction âĢĶ i . e ., Ġ$ p _{ \ text { event }} ( c _{ n + 1 }) $ Ġis Ġgreater Ġthan Ġa Ġthreshold Ġlearned Ġon Ġthe Ġdevelopment Ġsplit âĢĶ then Ġthe Ġrespective Ġconversation Ġis Ġpredicted Ġto Ġderail . ĠIf Ġthis Ġforecast Ġis Ġtriggered Ġin Ġa Ġconversation Ġthat Ġactually Ġder ails , Ġbut Ġbefore Ġthe Ġderail ment Ġactually Ġhappens , Ġthen Ġthe Ġconversation Ġis Ġcounted Ġas Ġa Ġtrue Ġpositive ; Ġotherwise Ġit Ġis Ġa Ġfalse Ġpositive . ĠIf Ġno Ġpositive Ġpredictions Ġare Ġtriggered Ġfor Ġa Ġconversation , Ġbut Ġit Ġactually Ġder ails Ġthen Ġit Ġcounts Ġas Ġa Ġfalse Ġnegative ; Ġif Ġit Ġdoes Ġnot Ġderail Ġthen Ġit Ġis Ġa Ġtrue Ġnegative . </s> Fixed - length Ġwindow Ġbas elines . ĠWe Ġfirst Ġseek Ġto Ġcompare ĠCRA FT Ġto Ġexisting , Ġfixed - length Ġwindow Ġapproaches Ġto Ġforecasting . ĠTo Ġthis Ġend , Ġwe Ġimplement Ġtwo Ġsuch Ġbas elines : ĠAw ry , Ġwhich Ġis Ġthe Ġstate - of - the - art Ġmethod Ġproposed Ġin ĠB IB REF 9 Ġbased Ġon Ġpragmatic Ġfeatures Ġin Ġthe Ġfirst Ġcomment - reply Ġpair , Ġand ĠBo W , Ġa Ġsimple Ġbag - of - words Ġbaseline Ġthat Ġmakes Ġa Ġprediction Ġusing ĠTF - ID F Ġweighted Ġbag - of - words Ġfeatures Ġextracted Ġfrom Ġthe Ġfirst Ġcomment - reply Ġpair . </s> Online Ġforecasting Ġbas elines . ĠNext , Ġwe Ġconsider Ġsimpler Ġapproaches Ġfor Ġmaking Ġforecasts Ġas Ġthe Ġconversations Ġhappen Ġ( i . e ., Ġin Ġan Ġonline Ġfashion ). ĠFirst , Ġwe Ġpropose ĠCum ulative ĠBo W , Ġa Ġmodel Ġthat Ġrecomp utes Ġbag - of - words Ġfeatures Ġon Ġall Ġcomments Ġseen Ġthus Ġfar Ġevery Ġtime Ġa Ġnew Ġcomment Ġarrives . ĠWhile Ġthis Ġapproach Ġdoes Ġexhibit Ġthe Ġdesired Ġbehavior Ġof Ġproducing Ġupdated Ġpredictions Ġfor Ġeach Ġnew Ġcomment , Ġit Ġfails Ġto Ġaccount Ġfor Ġrelationships Ġbetween Ġcomments . </s> This Ġsimple Ġcumulative Ġapproach Ġcannot Ġbe Ġdirectly Ġextended Ġto Ġmodels Ġwhose Ġfeatures Ġare Ġstrictly Ġbased Ġon Ġa Ġfixed Ġnumber Ġof Ġcomments , Ġlike ĠAw ry . ĠAn Ġalternative Ġis Ġto Ġuse Ġa Ġsliding Ġwindow : Ġfor Ġa Ġfeature Ġset Ġbased Ġon Ġa Ġwindow Ġof Ġ$ W $ Ġcomments , Ġupon Ġeach Ġnew Ġcomment Ġwe Ġcan Ġextract Ġfeatures Ġfrom Ġa Ġwindow Ġcontaining Ġthat Ġcomment Ġand Ġthe Ġ$ W - 1 $ Ġcomments Ġpreceding Ġit . ĠWe Ġapply Ġthis Ġto Ġthe ĠAw ry Ġmethod Ġand Ġcall Ġthis Ġmodel ĠSl iding ĠAw ry . ĠFor Ġboth Ġthese Ġbas elines , Ġwe Ġaggregate Ġcomment - level Ġpredictions Ġin Ġthe Ġsame Ġway Ġas Ġin Ġour Ġmain Ġmodel . </s> C RAFT Ġabl ations . ĠFinally , Ġwe Ġconsider Ġtwo Ġmodified Ġversions Ġof Ġthe ĠCRA FT Ġmodel Ġin Ġorder Ġto Ġevaluate Ġthe Ġimpact Ġof Ġtwo Ġof Ġits Ġkey Ġcomponents : Ġ( 1 ) Ġthe Ġpre - training Ġstep , Ġand Ġ( 2 ) Ġits Ġability Ġto Ġcapture Ġinter - comment Ġdependencies Ġthrough Ġits Ġhierarchical Ġmemory . </s> To Ġevaluate Ġthe Ġimpact Ġof Ġpre - training , Ġwe Ġtrain Ġthe Ġprediction Ġcomponent Ġof ĠCRA FT Ġon Ġonly Ġthe Ġlabeled Ġtraining Ġdata , Ġwithout Ġfirst Ġpre - training Ġthe Ġenc oder Ġlayers Ġwith Ġthe Ġunl abel ed Ġdata . ĠWe Ġfind Ġthat Ġgiven Ġthe Ġrelatively Ġsmall Ġsize Ġof Ġlabeled Ġdata , Ġthis Ġbaseline Ġfails Ġto Ġsuccessfully Ġlearn , Ġand Ġends Ġup Ġperforming Ġat Ġthe Ġlevel Ġof Ġrandom Ġguessing . ĠThis Ġresult Ġunderscores Ġthe Ġneed Ġfor Ġthe Ġpre - training Ġstep Ġthat Ġcan Ġmake Ġuse Ġof Ġunl abel ed Ġdata . </s> To Ġevaluate Ġthe Ġimpact Ġof Ġthe Ġhierarchical Ġmemory , Ġwe Ġimplement Ġa Ġsimplified Ġversion Ġof ĠCRA FT Ġwhere Ġthe Ġmemory Ġsize Ġof Ġthe Ġcontext Ġenc oder Ġis Ġzero Ġ( C RAFT Ġ$ -$ ĠCE ), Ġthus Ġeffectively Ġacting Ġas Ġif Ġthe Ġpre - training Ġcomponent Ġis Ġa Ġvanilla Ġseq 2 seq Ġmodel . ĠIn Ġother Ġwords , Ġthis Ġmodel Ġcannot Ġcapture Ġinter - comment Ġdependencies , Ġand Ġinstead Ġat Ġeach Ġstep Ġmakes Ġa Ġprediction Ġbased Ġonly Ġon Ġthe Ġutter ance Ġencoding Ġof Ġthe Ġlatest Ġcomment . </s> Results . ĠTable ĠT AB REF 17 Ġcompares ĠCRA FT Ġto Ġthe Ġbas elines Ġon Ġthe Ġtest Ġsplits Ġ( random Ġbaseline Ġis Ġ50 %) Ġand Ġillustrates Ġseveral Ġkey Ġfindings . ĠFirst , Ġwe Ġfind Ġthat Ġunsurprisingly , Ġaccounting Ġfor Ġfull Ġconvers ational Ġcontext Ġis Ġindeed Ġhelpful , Ġwith Ġeven Ġthe Ġsimple Ġonline Ġbas elines Ġoutper forming Ġthe Ġfixed - window Ġbas elines . ĠOn Ġboth Ġdatasets , ĠCRA FT Ġoutper forms Ġall Ġbas elines Ġ( including Ġthe Ġother Ġonline Ġmodels ) Ġin Ġterms Ġof Ġaccuracy Ġand ĠF 1 . ĠFurthermore , Ġalthough Ġit Ġloses Ġon Ġprecision Ġ( to ĠCRA FT Ġ$ -$ ĠCE ) Ġand Ġrecall Ġ( to ĠCum ulative ĠBo W ) Ġindividually Ġon Ġthe ĠWikipedia Ġdata , ĠCRA FT Ġhas Ġthe Ġsuperior Ġbalance Ġbetween Ġthe Ġtwo , Ġhaving Ġboth Ġa Ġvisibly Ġhigher Ġprecision - rec all Ġcurve Ġand Ġlarger Ġarea Ġunder Ġthe Ġcurve Ġ( AU PR ) Ġthan Ġthe Ġbas elines Ġ( Figure ĠFIG REF 20 ). ĠThis Ġlatter Ġproperty Ġis Ġparticularly Ġuseful Ġin Ġa Ġpractical Ġsetting , Ġas Ġit Ġallows Ġmoderators Ġto Ġtune Ġmodel Ġperformance Ġto Ġsome Ġdesired Ġprecision Ġwithout Ġhaving Ġto Ġsacrifice Ġas Ġmuch Ġin Ġthe Ġway Ġof Ġrecall Ġ( or Ġvice Ġversa ) Ġcompared Ġto Ġthe Ġbas elines Ġand Ġpre - existing Ġsolutions . </s> Analysis </s> We Ġnow Ġexamine Ġthe Ġbehavior Ġof ĠCRA FT Ġin Ġgreater Ġdetail , Ġto Ġbetter Ġunderstand Ġits Ġbenefits Ġand Ġlimitations . ĠWe Ġspecifically Ġaddress Ġthe Ġfollowing Ġquestions : Ġ( 1 ) ĠHow Ġmuch Ġearly Ġwarning Ġdoes Ġthe Ġthe Ġmodel Ġprovide ? Ġ( 2 ) ĠDoes Ġthe Ġmodel Ġactually Ġlearn Ġan Ġorder - sensitive Ġrepresentation Ġof Ġconvers ational Ġcontext ? </s> Early Ġwarning , Ġbut Ġhow Ġearly ? ĠThe Ġrecent Ġinterest Ġin Ġforecasting Ġantis ocial Ġbehavior Ġhas Ġbeen Ġdriven Ġby Ġa Ġdesire Ġto Ġprovide Ġpre - empt ive , Ġaction able Ġwarning Ġto Ġmoderators . ĠBut Ġdoes Ġour Ġmodel Ġtrigger Ġearly Ġenough Ġfor Ġany Ġsuch Ġpractical Ġgoals ? </s> For Ġeach Ġpersonal Ġattack Ġcorrectly Ġforecast ed Ġby Ġour Ġmodel , Ġwe Ġcount Ġthe Ġnumber Ġof Ġcomments Ġelapsed Ġbetween Ġthe Ġtime Ġthe Ġmodel Ġis Ġfirst Ġtriggered Ġand Ġthe Ġattack . ĠFigure ĠFIG REF 22 Ġshows Ġthe Ġdistribution Ġof Ġthese Ġcounts : Ġon Ġaverage , Ġthe Ġmodel Ġwarns Ġof Ġan Ġattack Ġ3 Ġcomments Ġbefore Ġit Ġactually Ġhappens Ġ( 4 Ġcomments Ġfor ĠCM V ). ĠTo Ġfurther Ġevaluate Ġhow Ġmuch Ġtime Ġthis Ġearly Ġwarning Ġwould Ġgive Ġto Ġthe Ġmoderator , Ġwe Ġalso Ġconsider Ġthe Ġdifference Ġin Ġtim est amps Ġbetween Ġthe Ġcomment Ġwhere Ġthe Ġmodel Ġfirst Ġtriggers Ġand Ġthe Ġcomment Ġcontaining Ġthe Ġactual Ġattack . ĠOver Ġ50 % Ġof Ġconversations Ġget Ġat Ġleast Ġ3 Ġhours Ġof Ġadvance Ġwarning Ġ( 2 Ġhours Ġfor ĠCM V ). ĠMoreover , Ġ39 % Ġof Ġconversations Ġget Ġat Ġleast Ġ12 Ġhours Ġof Ġearly Ġwarning Ġbefore Ġthey Ġderail . </s> Does Ġorder Ġmatter ? ĠOne Ġmotivation Ġbehind Ġthe Ġdesign Ġof Ġour Ġmodel Ġwas Ġthe Ġintuition Ġthat Ġcomments Ġin Ġa Ġconversation Ġare Ġnot Ġindependent Ġevents ; Ġrather , Ġthe Ġorder Ġin Ġwhich Ġthey Ġappear Ġmatters Ġ( e . g ., Ġa Ġblunt Ġcomment Ġfollowed Ġby Ġa Ġpolite Ġone Ġfeels Ġintuitive ly Ġdifferent Ġfrom Ġa Ġpolite Ġcomment Ġfollowed Ġby Ġa Ġblunt Ġone ). ĠBy Ġdesign , ĠCRA FT Ġhas Ġthe Ġcapacity Ġto Ġlearn Ġan Ġorder - sensitive Ġrepresentation Ġof Ġconvers ational Ġcontext , Ġbut Ġhow Ġcan Ġwe Ġknow Ġthat Ġthis Ġcapacity Ġis Ġactually Ġused ? ĠIt Ġis Ġconceivable Ġthat Ġthe Ġmodel Ġis Ġsimply Ġcomputing Ġan Ġorder - ins ensitive ĠâĢ ľ bag - of - features âĢ Ŀ . ĠNeural Ġnetwork Ġmodels Ġare Ġnotorious Ġfor Ġtheir Ġlack Ġof Ġtransparency , Ġpre cluding Ġan Ġanalysis Ġof Ġhow Ġexactly ĠCRA FT Ġmodels Ġconvers ational Ġcontext . ĠNevertheless , Ġthrough Ġtwo Ġsimple Ġexplor atory Ġexperiments , Ġwe Ġseek Ġto Ġshow Ġthat Ġit Ġdoes Ġnot Ġcompletely Ġignore Ġcomment Ġorder . </s> The Ġfirst Ġexperiment Ġfor Ġtesting Ġwhether Ġthe Ġmodel Ġaccounts Ġfor Ġcomment Ġorder Ġis Ġa Ġprefix - sh uff ling Ġexperiment , Ġvisual ized Ġin ĠFigure ĠFIG REF 23 . ĠFor Ġeach Ġconversation Ġthat Ġthe Ġmodel Ġpredicts Ġwill Ġderail , Ġlet Ġ$ t $ Ġdenote Ġthe Ġindex Ġof Ġthe Ġtriggering Ġcomment , Ġi . e ., Ġthe Ġindex Ġwhere Ġthe Ġmodel Ġfirst Ġmade Ġa Ġderail ment Ġforecast . ĠWe Ġthen Ġconstruct Ġsynthetic Ġconversations Ġby Ġtaking Ġthe Ġfirst Ġ$ t - 1 $ Ġcomments Ġ( hen ce forth Ġreferred Ġto Ġas Ġthe Ġprefix ) Ġand Ġrandom izing Ġtheir Ġorder . ĠFinally , Ġwe Ġcount Ġhow Ġoften Ġthe Ġmodel Ġno Ġlonger Ġpredicts Ġderail ment Ġat Ġindex Ġ$ t $ Ġin Ġthe Ġsynthetic Ġconversations . ĠIf Ġthe Ġmodel Ġwere Ġignoring Ġcomment Ġorder , Ġits Ġprediction Ġshould Ġremain Ġunchanged Ġ( as Ġit Ġremains Ġfor Ġthe ĠCum ulative ĠBo W Ġbaseline ), Ġsince Ġthe Ġactual Ġcontent Ġof Ġthe Ġfirst Ġ$ t $ Ġcomments Ġhas Ġnot Ġchanged Ġ( and ĠCRA FT Ġinference Ġis Ġdeter ministic ). ĠWe Ġinstead Ġfind Ġthat Ġin Ġroughly Ġone Ġfifth Ġof Ġcases Ġ( 12 % Ġfor ĠCM V ) Ġthe Ġmodel Ġchanges Ġits Ġprediction Ġon Ġthe Ġsynthetic Ġconversations . ĠThis Ġsuggests Ġthat ĠCRA FT Ġlearns Ġan Ġorder - sensitive Ġrepresentation Ġof Ġcontext , Ġnot Ġa Ġmere ĠâĢ ľ bag - of - features âĢ Ŀ . </s> To Ġmore Ġconcrete ly Ġquantify Ġhow Ġmuch Ġthis Ġorder - sensitive Ġcontext Ġmodeling Ġhelps Ġwith Ġprediction , Ġwe Ġcan Ġactively Ġprevent Ġthe Ġmodel Ġfrom Ġlearning Ġand Ġexploiting Ġany Ġorder - related Ġdynamics . ĠWe Ġachieve Ġthis Ġthrough Ġanother Ġtype Ġof Ġshuff ling Ġexperiment , Ġwhere Ġwe Ġgo Ġback Ġeven Ġfurther Ġand Ġshuffle Ġthe Ġcomment Ġorder Ġin Ġthe Ġconversations Ġused Ġfor Ġpre - training , Ġfine - tun ing Ġand Ġtesting . ĠThis Ġprocedure Ġpreserves Ġthe Ġmodel 's Ġability Ġto Ġcapture Ġsignals Ġpresent Ġwithin Ġthe Ġindividual Ġcomments Ġprocessed Ġso Ġfar , Ġas Ġthe Ġutter ance Ġenc oder Ġis Ġunaffected , Ġbut Ġinhibits Ġit Ġfrom Ġcapturing Ġany Ġmeaningful Ġorder - sensitive Ġdynamics . ĠWe Ġfind Ġthat Ġthis Ġhurts Ġthe Ġmodel 's Ġperformance Ġ( 65 % Ġaccuracy Ġfor ĠWikipedia , Ġ59 . 5 % Ġfor ĠCM V ), Ġlowering Ġit Ġto Ġa Ġlevel Ġsimilar Ġto Ġthat Ġof Ġthe Ġversion Ġwhere Ġwe Ġcompletely Ġdisable Ġthe Ġcontext Ġenc oder . </s> T aken Ġtogether , Ġthese Ġexperiments Ġprovide Ġevidence Ġthat ĠCRA FT Ġuses Ġits Ġcapacity Ġto Ġmodel Ġconvers ational Ġcontext Ġin Ġan Ġorder - sensitive Ġfashion , Ġand Ġthat Ġit Ġmakes Ġeffective Ġuse Ġof Ġthe Ġdynamics Ġwithin . ĠAn Ġimportant Ġavenue Ġfor Ġfuture Ġwork Ġwould Ġbe Ġdeveloping Ġmore Ġtransparent Ġmodels Ġthat Ġcan Ġshed Ġlight Ġon Ġexactly Ġwhat Ġkinds Ġof Ġorder - related Ġfeatures Ġare Ġbeing Ġextracted Ġand Ġhow Ġthey Ġare Ġused Ġin Ġprediction . </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġwork , Ġwe Ġintroduced Ġa Ġmodel Ġfor Ġforecasting Ġconvers ational Ġevents Ġthat Ġprocesses Ġcomments Ġas Ġthey Ġhappen Ġand Ġtakes Ġthe Ġfull Ġconvers ational Ġcontext Ġinto Ġaccount Ġto Ġmake Ġan Ġupdated Ġprediction Ġat Ġeach Ġstep . ĠThis Ġmodel Ġfills Ġa Ġvoid Ġin Ġthe Ġexisting Ġliterature Ġon Ġconvers ational Ġforecasting , Ġsimultaneously Ġaddressing Ġthe Ġdual Ġchallenges Ġof Ġcapturing Ġinter - comment Ġdynamics Ġand Ġdealing Ġwith Ġan Ġunknown Ġhorizon . ĠWe Ġfind Ġthat Ġour Ġmodel Ġachieves Ġstate - of - the - art Ġperformance Ġon Ġthe Ġtask Ġof Ġforecasting Ġderail ment Ġin Ġtwo Ġdifferent Ġdatasets Ġthat Ġwe Ġrelease Ġpublicly . ĠWe Ġfurther Ġshow Ġthat Ġthe Ġresulting Ġsystem Ġcan Ġprovide Ġsubstantial Ġprior Ġnotice Ġof Ġderail ment , Ġopening Ġup Ġthe Ġpotential Ġfor Ġpreempt ive Ġinterventions Ġby Ġhuman Ġmoderators ĠB IB REF 65 . </s> While Ġwe Ġhave Ġfocused Ġspecifically Ġon Ġthe Ġtask Ġof Ġforecasting Ġderail ment , Ġwe Ġview Ġthis Ġwork Ġas Ġa Ġstep Ġtowards Ġa Ġmore Ġgeneral Ġmodel Ġfor Ġreal - time Ġforecasting Ġof Ġother Ġtypes Ġof Ġemer gent Ġproperties Ġof Ġconversations . ĠFollow - up Ġwork Ġcould Ġadapt Ġthe ĠCRA FT Ġarchitecture Ġto Ġaddress Ġother Ġforecasting Ġtasks Ġmentioned Ġin ĠSection ĠSEC REF 2 âĢĶ including Ġthose Ġfor Ġwhich Ġthe Ġoutcome Ġis Ġextr aneous Ġto Ġthe Ġconversation . ĠWe Ġexpect Ġdifferent Ġtasks Ġto Ġbe Ġinformed Ġby Ġdifferent Ġtypes Ġof Ġinter - comment Ġdynamics , Ġand Ġfurther Ġarchitecture Ġextensions Ġcould Ġadd Ġadditional Ġsupervised Ġfine - tun ing Ġin Ġorder Ġto Ġdirect Ġit Ġto Ġfocus Ġon Ġspecific Ġdynamics Ġthat Ġmight Ġbe Ġrelevant Ġto Ġthe Ġtask Ġ( e . g ., Ġexchange Ġof Ġideas Ġbetween Ġinter loc ut ors Ġor Ġst onew alling ). </s> With Ġrespect Ġto Ġforecasting Ġderail ment , Ġthere Ġremain Ġopen Ġquestions Ġregarding Ġwhat Ġhuman Ġmoderators Ġactually Ġdesire Ġfrom Ġan Ġearly - warning Ġsystem , Ġwhich Ġwould Ġaffect Ġthe Ġdesign Ġof Ġa Ġpractical Ġsystem Ġbased Ġon Ġthis Ġwork . ĠFor Ġinstance , Ġhow Ġearly Ġdoes Ġa Ġwarning Ġneed Ġto Ġbe Ġin Ġorder Ġfor Ġmoderators Ġto Ġfind Ġit Ġuseful ? ĠWhat Ġis Ġthe Ġoptimal Ġbalance Ġbetween Ġprecision , Ġrecall , Ġand Ġfalse Ġpositive Ġrate Ġat Ġwhich Ġsuch Ġa Ġsystem Ġis Ġtruly Ġimproving Ġmoderator Ġproductivity Ġrather Ġthan Ġwasting Ġtheir Ġtime Ġthrough Ġfalse Ġpositives ? ĠWhat Ġare Ġthe Ġethical Ġimplications Ġof Ġsuch Ġa Ġsystem ? ĠFollow - up Ġwork Ġcould Ġrun Ġa Ġuser Ġstudy Ġof Ġa Ġprototype Ġsystem Ġwith Ġactual Ġmoderators Ġto Ġaddress Ġthese Ġquestions . </s> A Ġpractical Ġlimitation Ġof Ġthe Ġcurrent Ġanalysis Ġis Ġthat Ġit Ġrelies Ġon Ġbalanced Ġdatasets , Ġwhile Ġderail ment Ġis Ġa Ġrelatively Ġrare Ġevent Ġfor Ġwhich Ġa Ġmore Ġrestrictive Ġtrigger Ġthreshold Ġwould Ġbe Ġappropriate . ĠWhile Ġour Ġanalysis Ġof Ġthe Ġprecision - rec all Ġcurve Ġsuggests Ġthe Ġsystem Ġis Ġrobust Ġacross Ġmultiple Ġthresholds Ġ($ AU PR = 0 . 7 $ ), Ġadditional Ġwork Ġis Ġneeded Ġto Ġestablish Ġwhether Ġthe Ġrecall Ġtrade off Ġwould Ġbe Ġacceptable Ġin Ġpractice . </s> Finally , Ġone Ġmajor Ġlimitation Ġof Ġthe Ġpresent Ġwork Ġis Ġthat Ġit Ġassigns Ġa Ġsingle Ġlabel Ġto Ġeach Ġconversation : Ġdoes Ġit Ġderail Ġor Ġnot ? ĠIn Ġreality , Ġderail ment Ġneed Ġnot Ġspell Ġthe Ġend Ġof Ġa Ġconversation ; Ġit Ġis Ġpossible Ġthat Ġa Ġconversation Ġcould Ġget Ġback Ġon Ġtrack , Ġsuffer Ġa Ġrepeat Ġoccurrence Ġof Ġantis ocial Ġbehavior , Ġor Ġany Ġnumber Ġof Ġother Ġtraject ories . ĠIt Ġwould Ġbe Ġexciting Ġto Ġconsider Ġfiner - gr ained Ġforecasting Ġof Ġconvers ational Ġtraject ories , Ġaccounting Ġfor Ġthe Ġnatural âĢĶ and Ġsometimes Ġchaotic âĢĶ eb b - and - flow Ġof Ġhuman Ġinteractions . </s> Acknowled gements . ĠWe Ġthank ĠCaleb ĠCh iam , ĠLi ye ĠFu , ĠL illian ĠLee , ĠAlexand ru ĠNic ul esc u - M iz il , ĠAndrew ĠWang Ġand ĠJust ine ĠZhang Ġfor Ġinsightful Ġconversations Ġ( with Ġunknown Ġhorizon ), ĠAd ity a ĠJ ha Ġfor Ġhis Ġgreat Ġhelp Ġwith Ġimplementing Ġand Ġrunning Ġthe Ġcrowd - s ourcing Ġtasks , ĠThomas ĠDavidson Ġand ĠClaire ĠLiang Ġfor Ġexplor atory Ġdata Ġannotation , Ġas Ġwell Ġas Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġhelpful Ġcomments . ĠThis Ġwork Ġis Ġsupported Ġin Ġpart Ġby Ġthe ĠNS F ĠCARE ER Ġaward ĠI IS - 17 506 15 Ġand Ġby Ġthe ĠNS F ĠGrant ĠS ES - 174 14 41 .
Introduction “Ché saetta previsa vien più lenta.” – Dante Alighieri, Divina Commedia, Paradiso Antisocial behavior is a persistent problem plaguing online conversation platforms; it is both widespread BIBREF0 and potentially damaging to mental and emotional health BIBREF1, BIBREF2. The strain this phenomenon puts on community maintainers has sparked recent interest in computational approaches for assisting human moderators. Prior work in this direction has largely focused on post-hoc identification of various kinds of antisocial behavior, including hate speech BIBREF3, BIBREF4, harassment BIBREF5, personal attacks BIBREF6, and general toxicity BIBREF7. The fact that these approaches only identify antisocial content after the fact limits their practicality as tools for assisting pre-emptive moderation in conversational domains. Addressing this limitation requires forecasting the future derailment of a conversation based on early warning signs, giving the moderators time to potentially intervene before any harm is done (BIBREF8 BIBREF8, BIBREF9 BIBREF9, see BIBREF10 BIBREF10 for a discussion). Such a goal recognizes derailment as emerging from the development of the conversation, and belongs to the broader area of conversational forecasting, which includes future-prediction tasks such as predicting the eventual length of a conversation BIBREF11, whether a persuasion attempt will eventually succeed BIBREF12, BIBREF13, BIBREF14, whether team discussions will eventually lead to an increase in performance BIBREF15, or whether ongoing counseling conversations will eventually be perceived as helpful BIBREF16. Approaching such conversational forecasting problems, however, requires overcoming several inherent modeling challenges. First, conversations are dynamic and their outcome might depend on how subsequent comments interact with each other. Consider the example in Figure FIGREF2: while no individual comment is outright offensive, a human reader can sense a tension emerging from their succession (e.g., dismissive answers to repeated questioning). Thus a forecasting model needs to capture not only the content of each individual comment, but also the relations between comments. Previous work has largely relied on hand-crafted features to capture such relations—e.g., similarity between comments BIBREF16, BIBREF12 or conversation structure BIBREF17, BIBREF18—, though neural attention architectures have also recently shown promise BIBREF19. The second modeling challenge stems from the fact that conversations have an unknown horizon: they can be of varying lengths, and the to-be-forecasted event can occur at any time. So when is it a good time to make a forecast? Prior work has largely proposed two solutions, both resulting in important practical limitations. One solution is to assume (unrealistic) prior knowledge of when the to-be-forecasted event takes place and extract features up to that point BIBREF20, BIBREF8. Another compromising solution is to extract features from a fixed-length window, often at the start of the conversation BIBREF21, BIBREF15, BIBREF16, BIBREF9. Choosing a catch-all window-size is however impractical: short windows will miss information in comments they do not encompass (e.g., a window of only two comments would miss the chain of repeated questioning in comments 3 through 6 of Figure FIGREF2), while longer windows risk missing the to-be-forecasted event altogether if it occurs before the end of the window, which would prevent early detection. In this work we introduce a model for forecasting conversational events that overcomes both these inherent challenges by processing comments, and their relations, as they happen (i.e., in an online fashion). Our main insight is that models with these properties already exist, albeit geared toward generation rather than prediction: recent work in context-aware dialog generation (or “chatbots”) has proposed sequential neural models that make effective use of the intra-conversational dynamics BIBREF22, BIBREF23, BIBREF24, while concomitantly being able to process the conversation as it develops (see BIBREF25 for a survey). In order for these systems to perform well in the generative domain they need to be trained on massive amounts of (unlabeled) conversational data. The main difficulty in directly adapting these models to the supervised domain of conversational forecasting is the relative scarcity of labeled data: for most forecasting tasks, at most a few thousands labeled examples are available, insufficient for the notoriously data-hungry sequential neural models. To overcome this difficulty, we propose to decouple the objective of learning a neural representation of conversational dynamics from the objective of predicting future events. The former can be pre-trained on large amounts of unsupervised data, similarly to how chatbots are trained. The latter can piggy-back on the resulting representation after fine-tuning it for classification using relatively small labeled data. While similar pre-train-then-fine-tune approaches have recently achieved state-of-the-art performance in a number of NLP tasks—including natural language inference, question answering, and commonsense reasoning (discussed in Section SECREF2)—to the best of our knowledge this is the first attempt at applying this paradigm to conversational forecasting. To test the effectiveness of this new architecture in forecasting derailment of online conversations, we develop and distribute two new datasets. The first triples in size the highly curated `Conversations Gone Awry' dataset BIBREF9, where civil-starting Wikipedia Talk Page conversations are crowd-labeled according to whether they eventually lead to personal attacks; the second relies on in-the-wild moderation of the popular subreddit ChangeMyView, where the aim is to forecast whether a discussion will later be subject to moderator action due to “rude or hostile” behavior. In both datasets, our model outperforms existing fixed-window approaches, as well as simpler sequential baselines that cannot account for inter-comment relations. Furthermore, by virtue of its online processing of the conversation, our system can provide substantial prior notice of upcoming derailment, triggering on average 3 comments (or 3 hours) before an overtly toxic comment is posted. To summarize, in this work we: introduce the first model for forecasting conversational events that can capture the dynamics of a conversation as it develops; build two diverse datasets (one entirely new, one extending prior work) for the task of forecasting derailment of online conversations; compare the performance of our model against the current state-of-the-art, and evaluate its ability to provide early warning signs. Our work is motivated by the goal of assisting human moderators of online communities by preemptively signaling at-risk conversations that might deserve their attention. However, we caution that any automated systems might encode or even amplify the biases existing in the training data BIBREF26, BIBREF27, BIBREF28, so a public-facing implementation would need to be exhaustively scrutinized for such biases BIBREF29. Further Related Work Antisocial behavior. Antisocial behavior online comes in many forms, including harassment BIBREF30, cyberbullying BIBREF31, and general aggression BIBREF32. Prior work has sought to understand different aspects of such behavior, including its effect on the communities where it happens BIBREF33, BIBREF34, the actors involved BIBREF35, BIBREF36, BIBREF37, BIBREF38 and connections to the outside world BIBREF39. Post-hoc classification of conversations. There is a rich body of prior work on classifying the outcome of a conversation after it has concluded, or classifying conversational events after they happened. Many examples exist, but some more closely related to our present work include identifying the winner of a debate BIBREF40, BIBREF41, BIBREF42, identifying successful negotiations BIBREF21, BIBREF43, as well as detecting whether deception BIBREF44, BIBREF45, BIBREF46 or disagreement BIBREF47, BIBREF48, BIBREF49, BIBREF50, BIBREF51 has occurred. Our goal is different because we wish to forecast conversational events before they happen and while the conversation is still ongoing (potentially allowing for interventions). Note that some post-hoc tasks can also be re-framed as forecasting tasks (assuming the existence of necessary labels); for instance, predicting whether an ongoing conversation will eventually spark disagreement BIBREF18, rather than detecting already-existing disagreement. Conversational forecasting. As described in Section SECREF1, prior work on forecasting conversational outcomes and events has largely relied on hand-crafted features to capture aspects of conversational dynamics. Example feature sets include statistical measures based on similarity between utterances BIBREF16, sentiment imbalance BIBREF20, flow of ideas BIBREF20, increase in hostility BIBREF8, reply rate BIBREF11 and graph representations of conversations BIBREF52, BIBREF17. By contrast, we aim to automatically learn neural representations of conversational dynamics through pre-training. Such hand-crafted features are typically extracted from fixed-length windows of the conversation, leaving unaddressed the problem of unknown horizon. While some work has trained multiple models for different window-lengths BIBREF8, BIBREF18, they consider these models to be independent and, as such, do not address the issue of aggregating them into a single forecast (i.e., deciding at what point to make a prediction). We implement a simple sliding windows solution as a baseline (Section SECREF5). Pre-training for NLP. The use of pre-training for natural language tasks has been growing in popularity after recent breakthroughs demonstrating improved performance on a wide array of benchmark tasks BIBREF53, BIBREF54. Existing work has generally used a language modeling objective as the pre-training objective; examples include next-word prediction BIBREF55, sentence autoencoding, BIBREF56, and machine translation BIBREF57. BERT BIBREF58 introduces a variation on this in which the goal is to predict the next sentence in a document given the current sentence. Our pre-training objective is similar in spirit, but operates at a conversation level, rather than a document level. We hence view our objective as conversational modeling rather than (only) language modeling. Furthermore, while BERT's sentence prediction objective is framed as a multiple-choice task, our objective is framed as a generative task. Derailment Datasets We consider two datasets, representing related but slightly different forecasting tasks. The first dataset is an expanded version of the annotated Wikipedia conversations dataset from BIBREF9. This dataset uses carefully-controlled crowdsourced labels, strictly filtered to ensure the conversations are civil up to the moment of a personal attack. This is a useful property for the purposes of model analysis, and hence we focus on this as our primary dataset. However, we are conscious of the possibility that these strict labels may not fully capture the kind of behavior that moderators care about in practice. We therefore introduce a secondary dataset, constructed from the subreddit ChangeMyView (CMV) that does not use post-hoc annotations. Instead, the prediction task is to forecast whether the conversation will be subject to moderator action in the future. Wikipedia data. BIBREF9's `Conversations Gone Awry' dataset consists of 1,270 conversations that took place between Wikipedia editors on publicly accessible talk pages. The conversations are sourced from the WikiConv dataset BIBREF59 and labeled by crowdworkers as either containing a personal attack from within (i.e., hostile behavior by one user in the conversation directed towards another) or remaining civil throughout. A series of controls are implemented to prevent models from picking up on trivial correlations. To prevent models from capturing topic-specific information (e.g., political conversations are more likely to derail), each attack-containing conversation is paired with a clean conversation from the same talk page, where the talk page serves as a proxy for topic. To force models to actually capture conversational dynamics rather than detecting already-existing toxicity, human annotations are used to ensure that all comments preceding a personal attack are civil. To the ends of more effective model training, we elected to expand the `Conversations Gone Awry' dataset, using the original annotation procedure. Since we found that the original data skewed towards shorter conversations, we focused this crowdsourcing run on longer conversations: ones with 4 or more comments preceding the attack. Through this additional crowdsourcing, we expand the dataset to 4,188 conversations, which we are publicly releasing as part of the Cornell Conversational Analysis Toolkit (ConvoKit). We perform an 80-20-20 train/dev/test split, ensuring that paired conversations end up in the same split in order to preserve the topic control. Finally, we randomly sample another 1 million conversations from WikiConv to use for the unsupervised pre-training of the generative component. Reddit CMV data. The CMV dataset is constructed from conversations collected via the Reddit API. In contrast to the Wikipedia-based dataset, we explicitly avoid the use of post-hoc annotation. Instead, we use as our label whether a conversation eventually had a comment removed by a moderator for violation of Rule 2: “Don't be rude or hostile to other users”. Though the lack of post-hoc annotation limits the degree to which we can impose controls on the data (e.g., some conversations may contain toxic comments not flagged by the moderators) we do reproduce as many of the Wikipedia data's controls as we can. Namely, we replicate the topic control pairing by choosing pairs of positive and negative examples that belong to the same top-level post, following BIBREF12; and enforce that the removed comment was made by a user who was previously involved in the conversation. This process results in 6,842 conversations, to which we again apply a pair-preserving 80-20-20 split. Finally, we gather over 600,000 conversations that do not include any removed comment, for unsupervised pre-training. Conversational Forecasting Model We now describe our general model for forecasting future conversational events. Our model integrates two components: (a) a generative dialog model that learns to represent conversational dynamics in an unsupervised fashion; and (b) a supervised component that fine-tunes this representation to forecast future events. Figure FIGREF13 provides an overview of the proposed architecture, henceforth CRAFT (Conversational Recurrent Architecture for ForecasTing). Terminology. For modeling purposes, we treat a conversation as a sequence of $N$ comments $C = \lbrace c_1,\dots ,c_N\rbrace $. Each comment, in turn, is a sequence of tokens, where the number of tokens may vary from comment to comment. For the $n$-th comment ($1 \le n \le N)$, we let $M_n$ denote the number of tokens. Then, a comment $c_n$ can be represented as a sequence of $M_n$ tokens: $c_n = \lbrace w_1,\dots ,w_{M_n}\rbrace $. Generative component. For the generative component of our model, we use a hierarchical recurrent encoder-decoder (HRED) architecture BIBREF60, a modified version of the popular sequence-to-sequence (seq2seq) architecture BIBREF61 designed to account for dependencies between consecutive inputs. BIBREF23 showed that HRED can successfully model conversational context by encoding the temporal structure of previously seen comments, making it an ideal fit for our use case. Here, we provide a high-level summary of the HRED architecture, deferring deeper technical discussion to BIBREF60 and BIBREF23. An HRED dialog model consists of three components: an utterance encoder, a context encoder, and a decoder. The utterance encoder is responsible for generating semantic vector representations of comments. It consists of a recurrent neural network (RNN) that reads a comment token-by-token, and on each token $w_m$ updates a hidden state $h^{\text{enc}}$ based on the current token and the previous hidden state: where $f^{\text{RNN}}$ is a nonlinear gating function (our implementation uses GRU BIBREF62). The final hidden state $h^{\text{enc}}_M$ can be viewed as a vector encoding of the entire comment. Running the encoder on each comment $c_n$ results in a sequence of $N$ vector encodings. A second encoder, the context encoder, is then run over this sequence: Each hidden state $h^{\text{con}}_n$ can then be viewed as an encoding of the full conversational context up to and including the $n$-th comment. To generate a response to comment $n$, the context encoding $h^{\text{con}}_n$ is used to initialize the hidden state $h^{\text{dec}}_{0}$ of a decoder RNN. The decoder produces a response token by token using the following recurrence: where $f^{\text{out}}$ is some function that outputs a probability distribution over words; we implement this using a simple feedforward layer. In our implementation, we further augment the decoder with attention BIBREF63, BIBREF64 over context encoder states to help capture long-term inter-comment dependencies. This generative component can be pre-trained using unlabeled conversational data. Prediction component. Given a pre-trained HRED dialog model, we aim to extend the model to predict from the conversational context whether the to-be-forecasted event will occur. Our predictor consists of a multilayer perceptron (MLP) with 3 fully-connected layers, leaky ReLU activations between layers, and sigmoid activation for output. For each comment $c_n$, the predictor takes as input the context encoding $h^{\text{con}}_n$ and forwards it through the MLP layers, resulting in an output score that is interpreted as a probability $p_{\text{event}}(c_{n+1})$ that the to-be-forecasted event will happen (e.g., that the conversation will derail). Training the predictive component starts by initializing the weights of the encoders to the values learned in pre-training. The main training loop then works as follows: for each positive sample—i.e., a conversation containing an instance of the to-be-forecasted event (e.g., derailment) at comment $c_e$—we feed the context $c_1,\dots ,c_{e-1}$ through the encoder and classifier, and compute cross-entropy loss between the classifier output and expected output of 1. Similarly, for each negative sample—i.e., a conversation where none of the comments exhibit the to-be-forecasted event and that ends with $c_N$—we feed the context $c_1,\dots ,c_{N-1}$ through the model and compute loss against an expected output of 0. Note that the parameters of the generative component are not held fixed during this process; instead, backpropagation is allowed to go all the way through the encoder layers. This process, known as fine-tuning, reshapes the representation learned during pre-training to be more directly useful to prediction BIBREF55. We implement the model and training code using PyTorch, and we are publicly releasing our implementation and the trained models together with the data as part of ConvoKit. Forecasting Derailment We evaluate the performance of CRAFT in the task of forecasting conversational derailment in both the Wikipedia and CMV scenarios. To this end, for each of these datasets we pre-train the generative component on the unlabeled portion of the data and fine-tune it on the labeled training split (data size detailed in Section SECREF3). In order to evaluate our sequential system against conversational-level ground truth, we need to aggregate comment level predictions. If any comment in the conversation triggers a positive prediction—i.e., $p_{\text{event}}(c_{n+1})$ is greater than a threshold learned on the development split—then the respective conversation is predicted to derail. If this forecast is triggered in a conversation that actually derails, but before the derailment actually happens, then the conversation is counted as a true positive; otherwise it is a false positive. If no positive predictions are triggered for a conversation, but it actually derails then it counts as a false negative; if it does not derail then it is a true negative. Fixed-length window baselines. We first seek to compare CRAFT to existing, fixed-length window approaches to forecasting. To this end, we implement two such baselines: Awry, which is the state-of-the-art method proposed in BIBREF9 based on pragmatic features in the first comment-reply pair, and BoW, a simple bag-of-words baseline that makes a prediction using TF-IDF weighted bag-of-words features extracted from the first comment-reply pair. Online forecasting baselines. Next, we consider simpler approaches for making forecasts as the conversations happen (i.e., in an online fashion). First, we propose Cumulative BoW, a model that recomputes bag-of-words features on all comments seen thus far every time a new comment arrives. While this approach does exhibit the desired behavior of producing updated predictions for each new comment, it fails to account for relationships between comments. This simple cumulative approach cannot be directly extended to models whose features are strictly based on a fixed number of comments, like Awry. An alternative is to use a sliding window: for a feature set based on a window of $W$ comments, upon each new comment we can extract features from a window containing that comment and the $W-1$ comments preceding it. We apply this to the Awry method and call this model Sliding Awry. For both these baselines, we aggregate comment-level predictions in the same way as in our main model. CRAFT ablations. Finally, we consider two modified versions of the CRAFT model in order to evaluate the impact of two of its key components: (1) the pre-training step, and (2) its ability to capture inter-comment dependencies through its hierarchical memory. To evaluate the impact of pre-training, we train the prediction component of CRAFT on only the labeled training data, without first pre-training the encoder layers with the unlabeled data. We find that given the relatively small size of labeled data, this baseline fails to successfully learn, and ends up performing at the level of random guessing. This result underscores the need for the pre-training step that can make use of unlabeled data. To evaluate the impact of the hierarchical memory, we implement a simplified version of CRAFT where the memory size of the context encoder is zero (CRAFT $-$ CE), thus effectively acting as if the pre-training component is a vanilla seq2seq model. In other words, this model cannot capture inter-comment dependencies, and instead at each step makes a prediction based only on the utterance encoding of the latest comment. Results. Table TABREF17 compares CRAFT to the baselines on the test splits (random baseline is 50%) and illustrates several key findings. First, we find that unsurprisingly, accounting for full conversational context is indeed helpful, with even the simple online baselines outperforming the fixed-window baselines. On both datasets, CRAFT outperforms all baselines (including the other online models) in terms of accuracy and F1. Furthermore, although it loses on precision (to CRAFT $-$ CE) and recall (to Cumulative BoW) individually on the Wikipedia data, CRAFT has the superior balance between the two, having both a visibly higher precision-recall curve and larger area under the curve (AUPR) than the baselines (Figure FIGREF20). This latter property is particularly useful in a practical setting, as it allows moderators to tune model performance to some desired precision without having to sacrifice as much in the way of recall (or vice versa) compared to the baselines and pre-existing solutions. Analysis We now examine the behavior of CRAFT in greater detail, to better understand its benefits and limitations. We specifically address the following questions: (1) How much early warning does the the model provide? (2) Does the model actually learn an order-sensitive representation of conversational context? Early warning, but how early? The recent interest in forecasting antisocial behavior has been driven by a desire to provide pre-emptive, actionable warning to moderators. But does our model trigger early enough for any such practical goals? For each personal attack correctly forecasted by our model, we count the number of comments elapsed between the time the model is first triggered and the attack. Figure FIGREF22 shows the distribution of these counts: on average, the model warns of an attack 3 comments before it actually happens (4 comments for CMV). To further evaluate how much time this early warning would give to the moderator, we also consider the difference in timestamps between the comment where the model first triggers and the comment containing the actual attack. Over 50% of conversations get at least 3 hours of advance warning (2 hours for CMV). Moreover, 39% of conversations get at least 12 hours of early warning before they derail. Does order matter? One motivation behind the design of our model was the intuition that comments in a conversation are not independent events; rather, the order in which they appear matters (e.g., a blunt comment followed by a polite one feels intuitively different from a polite comment followed by a blunt one). By design, CRAFT has the capacity to learn an order-sensitive representation of conversational context, but how can we know that this capacity is actually used? It is conceivable that the model is simply computing an order-insensitive “bag-of-features”. Neural network models are notorious for their lack of transparency, precluding an analysis of how exactly CRAFT models conversational context. Nevertheless, through two simple exploratory experiments, we seek to show that it does not completely ignore comment order. The first experiment for testing whether the model accounts for comment order is a prefix-shuffling experiment, visualized in Figure FIGREF23. For each conversation that the model predicts will derail, let $t$ denote the index of the triggering comment, i.e., the index where the model first made a derailment forecast. We then construct synthetic conversations by taking the first $t-1$ comments (henceforth referred to as the prefix) and randomizing their order. Finally, we count how often the model no longer predicts derailment at index $t$ in the synthetic conversations. If the model were ignoring comment order, its prediction should remain unchanged (as it remains for the Cumulative BoW baseline), since the actual content of the first $t$ comments has not changed (and CRAFT inference is deterministic). We instead find that in roughly one fifth of cases (12% for CMV) the model changes its prediction on the synthetic conversations. This suggests that CRAFT learns an order-sensitive representation of context, not a mere “bag-of-features”. To more concretely quantify how much this order-sensitive context modeling helps with prediction, we can actively prevent the model from learning and exploiting any order-related dynamics. We achieve this through another type of shuffling experiment, where we go back even further and shuffle the comment order in the conversations used for pre-training, fine-tuning and testing. This procedure preserves the model's ability to capture signals present within the individual comments processed so far, as the utterance encoder is unaffected, but inhibits it from capturing any meaningful order-sensitive dynamics. We find that this hurts the model's performance (65% accuracy for Wikipedia, 59.5% for CMV), lowering it to a level similar to that of the version where we completely disable the context encoder. Taken together, these experiments provide evidence that CRAFT uses its capacity to model conversational context in an order-sensitive fashion, and that it makes effective use of the dynamics within. An important avenue for future work would be developing more transparent models that can shed light on exactly what kinds of order-related features are being extracted and how they are used in prediction. Conclusions and Future Work In this work, we introduced a model for forecasting conversational events that processes comments as they happen and takes the full conversational context into account to make an updated prediction at each step. This model fills a void in the existing literature on conversational forecasting, simultaneously addressing the dual challenges of capturing inter-comment dynamics and dealing with an unknown horizon. We find that our model achieves state-of-the-art performance on the task of forecasting derailment in two different datasets that we release publicly. We further show that the resulting system can provide substantial prior notice of derailment, opening up the potential for preemptive interventions by human moderators BIBREF65. While we have focused specifically on the task of forecasting derailment, we view this work as a step towards a more general model for real-time forecasting of other types of emergent properties of conversations. Follow-up work could adapt the CRAFT architecture to address other forecasting tasks mentioned in Section SECREF2—including those for which the outcome is extraneous to the conversation. We expect different tasks to be informed by different types of inter-comment dynamics, and further architecture extensions could add additional supervised fine-tuning in order to direct it to focus on specific dynamics that might be relevant to the task (e.g., exchange of ideas between interlocutors or stonewalling). With respect to forecasting derailment, there remain open questions regarding what human moderators actually desire from an early-warning system, which would affect the design of a practical system based on this work. For instance, how early does a warning need to be in order for moderators to find it useful? What is the optimal balance between precision, recall, and false positive rate at which such a system is truly improving moderator productivity rather than wasting their time through false positives? What are the ethical implications of such a system? Follow-up work could run a user study of a prototype system with actual moderators to address these questions. A practical limitation of the current analysis is that it relies on balanced datasets, while derailment is a relatively rare event for which a more restrictive trigger threshold would be appropriate. While our analysis of the precision-recall curve suggests the system is robust across multiple thresholds ($AUPR=0.7$), additional work is needed to establish whether the recall tradeoff would be acceptable in practice. Finally, one major limitation of the present work is that it assigns a single label to each conversation: does it derail or not? In reality, derailment need not spell the end of a conversation; it is possible that a conversation could get back on track, suffer a repeat occurrence of antisocial behavior, or any number of other trajectories. It would be exciting to consider finer-grained forecasting of conversational trajectories, accounting for the natural—and sometimes chaotic—ebb-and-flow of human interactions. Acknowledgements. We thank Caleb Chiam, Liye Fu, Lillian Lee, Alexandru Niculescu-Mizil, Andrew Wang and Justine Zhang for insightful conversations (with unknown horizon), Aditya Jha for his great help with implementing and running the crowd-sourcing tasks, Thomas Davidson and Claire Liang for exploratory data annotation, as well as the anonymous reviewers for their helpful comments. This work is supported in part by the NSF CAREER award IIS-1750615 and by the NSF Grant SES-1741441.
What are two datasets model is applied to?
88e5d37617e14d6976cc602a168332fc23644f19
What Ġare Ġtwo Ġdatasets Ġmodel Ġis Ġapplied Ġto ?
Ġ` Con vers ations ĠGone ĠAw ry ' Ġdataset , Ġsubreddit ĠChange My View
<s> What Ġare Ġtwo Ġdatasets Ġmodel Ġis Ġapplied Ġto ? </s> Introduction </s> âĢ ľ Ch Ã© Ġsa etta Ġpre vis a Ġv ien Ġpi Ã ¹ Ġlent a . âĢ Ŀ </s> âĢĵ ĠDante ĠAl igh ieri , ĠDiv ina ĠCom media , ĠParad iso </s> Ant is ocial Ġbehavior Ġis Ġa Ġpersistent Ġproblem Ġplag uing Ġonline Ġconversation Ġplatforms ; Ġit Ġis Ġboth Ġwidespread ĠB IB REF 0 Ġand Ġpotentially Ġdamaging Ġto Ġmental Ġand Ġemotional Ġhealth ĠB IB REF 1 , ĠB IB REF 2 . ĠThe Ġstrain Ġthis Ġphenomenon Ġputs Ġon Ġcommunity Ġmaintain ers Ġhas Ġsparked Ġrecent Ġinterest Ġin Ġcomputational Ġapproaches Ġfor Ġassisting Ġhuman Ġmoderators . </s> Prior Ġwork Ġin Ġthis Ġdirection Ġhas Ġlargely Ġfocused Ġon Ġpost - h oc Ġidentification Ġof Ġvarious Ġkinds Ġof Ġantis ocial Ġbehavior , Ġincluding Ġhate Ġspeech ĠB IB REF 3 , ĠB IB REF 4 , Ġharassment ĠB IB REF 5 , Ġpersonal Ġattacks ĠB IB REF 6 , Ġand Ġgeneral Ġtoxicity ĠB IB REF 7 . ĠThe Ġfact Ġthat Ġthese Ġapproaches Ġonly Ġidentify Ġantis ocial Ġcontent Ġafter Ġthe Ġfact Ġlimits Ġtheir Ġpractical ity Ġas Ġtools Ġfor Ġassisting Ġpre - empt ive Ġmoderation Ġin Ġconvers ational Ġdomains . </s> Add ressing Ġthis Ġlimitation Ġrequires Ġforecasting Ġthe Ġfuture Ġderail ment Ġof Ġa Ġconversation Ġbased Ġon Ġearly Ġwarning Ġsigns , Ġgiving Ġthe Ġmoderators Ġtime Ġto Ġpotentially Ġintervene Ġbefore Ġany Ġharm Ġis Ġdone Ġ( BI B REF 8 ĠB IB REF 8 , ĠB IB REF 9 ĠB IB REF 9 , Ġsee ĠB IB REF 10 ĠB IB REF 10 Ġfor Ġa Ġdiscussion ). ĠSuch Ġa Ġgoal Ġrecognizes Ġderail ment Ġas Ġemerging Ġfrom Ġthe Ġdevelopment Ġof Ġthe Ġconversation , Ġand Ġbelongs Ġto Ġthe Ġbroader Ġarea Ġof Ġconvers ational Ġforecasting , Ġwhich Ġincludes Ġfuture - pred iction Ġtasks Ġsuch Ġas Ġpredicting Ġthe Ġeventual Ġlength Ġof Ġa Ġconversation ĠB IB REF 11 , Ġwhether Ġa Ġpersuasion Ġattempt Ġwill Ġeventually Ġsucceed ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 14 , Ġwhether Ġteam Ġdiscussions Ġwill Ġeventually Ġlead Ġto Ġan Ġincrease Ġin Ġperformance ĠB IB REF 15 , Ġor Ġwhether Ġongoing Ġcounseling Ġconversations Ġwill Ġeventually Ġbe Ġperceived Ġas Ġhelpful ĠB IB REF 16 . </s> App ro aching Ġsuch Ġconvers ational Ġforecasting Ġproblems , Ġhowever , Ġrequires Ġovercoming Ġseveral Ġinherent Ġmodeling Ġchallenges . ĠFirst , Ġconversations Ġare Ġdynamic Ġand Ġtheir Ġoutcome Ġmight Ġdepend Ġon Ġhow Ġsubsequent Ġcomments Ġinteract Ġwith Ġeach Ġother . ĠConsider Ġthe Ġexample Ġin ĠFigure ĠFIG REF 2 : Ġwhile Ġno Ġindividual Ġcomment Ġis Ġoutright Ġoffensive , Ġa Ġhuman Ġreader Ġcan Ġsense Ġa Ġtension Ġemerging Ġfrom Ġtheir Ġsuccession Ġ( e . g ., Ġdismissive Ġanswers Ġto Ġrepeated Ġquestioning ). ĠThus Ġa Ġforecasting Ġmodel Ġneeds Ġto Ġcapture Ġnot Ġonly Ġthe Ġcontent Ġof Ġeach Ġindividual Ġcomment , Ġbut Ġalso Ġthe Ġrelations Ġbetween Ġcomments . ĠPrevious Ġwork Ġhas Ġlargely Ġrelied Ġon Ġhand - crafted Ġfeatures Ġto Ġcapture Ġsuch Ġrelations âĢĶ e . g ., Ġsimilarity Ġbetween Ġcomments ĠB IB REF 16 , ĠB IB REF 12 Ġor Ġconversation Ġstructure ĠB IB REF 17 , ĠB IB REF 18 âĢĶ , Ġthough Ġneural Ġattention Ġarchitectures Ġhave Ġalso Ġrecently Ġshown Ġpromise ĠB IB REF 19 . </s> The Ġsecond Ġmodeling Ġchallenge Ġstems Ġfrom Ġthe Ġfact Ġthat Ġconversations Ġhave Ġan Ġunknown Ġhorizon : Ġthey Ġcan Ġbe Ġof Ġvarying Ġlengths , Ġand Ġthe Ġto - be - fore cast ed Ġevent Ġcan Ġoccur Ġat Ġany Ġtime . ĠSo Ġwhen Ġis Ġit Ġa Ġgood Ġtime Ġto Ġmake Ġa Ġforecast ? ĠPrior Ġwork Ġhas Ġlargely Ġproposed Ġtwo Ġsolutions , Ġboth Ġresulting Ġin Ġimportant Ġpractical Ġlimitations . ĠOne Ġsolution Ġis Ġto Ġassume Ġ( un real istic ) Ġprior Ġknowledge Ġof Ġwhen Ġthe Ġto - be - fore cast ed Ġevent Ġtakes Ġplace Ġand Ġextract Ġfeatures Ġup Ġto Ġthat Ġpoint ĠB IB REF 20 , ĠB IB REF 8 . ĠAnother Ġcompromising Ġsolution Ġis Ġto Ġextract Ġfeatures Ġfrom Ġa Ġfixed - length Ġwindow , Ġoften Ġat Ġthe Ġstart Ġof Ġthe Ġconversation ĠB IB REF 21 , ĠB IB REF 15 , ĠB IB REF 16 , ĠB IB REF 9 . ĠCho osing Ġa Ġcatch - all Ġwindow - size Ġis Ġhowever Ġimpractical : Ġshort Ġwindows Ġwill Ġmiss Ġinformation Ġin Ġcomments Ġthey Ġdo Ġnot Ġencompass Ġ( e . g ., Ġa Ġwindow Ġof Ġonly Ġtwo Ġcomments Ġwould Ġmiss Ġthe Ġchain Ġof Ġrepeated Ġquestioning Ġin Ġcomments Ġ3 Ġthrough Ġ6 Ġof ĠFigure ĠFIG REF 2 ), Ġwhile Ġlonger Ġwindows Ġrisk Ġmissing Ġthe Ġto - be - fore cast ed Ġevent Ġaltogether Ġif Ġit Ġoccurs Ġbefore Ġthe Ġend Ġof Ġthe Ġwindow , Ġwhich Ġwould Ġprevent Ġearly Ġdetection . </s> In Ġthis Ġwork Ġwe Ġintroduce Ġa Ġmodel Ġfor Ġforecasting Ġconvers ational Ġevents Ġthat Ġover comes Ġboth Ġthese Ġinherent Ġchallenges Ġby Ġprocessing Ġcomments , Ġand Ġtheir Ġrelations , Ġas Ġthey Ġhappen Ġ( i . e ., Ġin Ġan Ġonline Ġfashion ). ĠOur Ġmain Ġinsight Ġis Ġthat Ġmodels Ġwith Ġthese Ġproperties Ġalready Ġexist , Ġalbeit Ġgeared Ġtoward Ġgeneration Ġrather Ġthan Ġprediction : Ġrecent Ġwork Ġin Ġcontext - aware Ġdialog Ġgeneration Ġ( or ĠâĢ ľ chat bots âĢ Ŀ ) Ġhas Ġproposed Ġsequential Ġneural Ġmodels Ġthat Ġmake Ġeffective Ġuse Ġof Ġthe Ġintra - con vers ational Ġdynamics ĠB IB REF 22 , ĠB IB REF 23 , ĠB IB REF 24 , Ġwhile Ġcon com it antly Ġbeing Ġable Ġto Ġprocess Ġthe Ġconversation Ġas Ġit Ġdevelops Ġ( see ĠB IB REF 25 Ġfor Ġa Ġsurvey ). </s> In Ġorder Ġfor Ġthese Ġsystems Ġto Ġperform Ġwell Ġin Ġthe Ġgener ative Ġdomain Ġthey Ġneed Ġto Ġbe Ġtrained Ġon Ġmassive Ġamounts Ġof Ġ( un label ed ) Ġconvers ational Ġdata . ĠThe Ġmain Ġdifficulty Ġin Ġdirectly Ġadapting Ġthese Ġmodels Ġto Ġthe Ġsupervised Ġdomain Ġof Ġconvers ational Ġforecasting Ġis Ġthe Ġrelative Ġscarcity Ġof Ġlabeled Ġdata : Ġfor Ġmost Ġforecasting Ġtasks , Ġat Ġmost Ġa Ġfew Ġthousands Ġlabeled Ġexamples Ġare Ġavailable , Ġinsufficient Ġfor Ġthe Ġnotoriously Ġdata - hung ry Ġsequential Ġneural Ġmodels . </s> To Ġovercome Ġthis Ġdifficulty , Ġwe Ġpropose Ġto Ġdec ouple Ġthe Ġobjective Ġof Ġlearning Ġa Ġneural Ġrepresentation Ġof Ġconvers ational Ġdynamics Ġfrom Ġthe Ġobjective Ġof Ġpredicting Ġfuture Ġevents . ĠThe Ġformer Ġcan Ġbe Ġpre - trained Ġon Ġlarge Ġamounts Ġof Ġun super vised Ġdata , Ġsimilarly Ġto Ġhow Ġchat bots Ġare Ġtrained . ĠThe Ġlatter Ġcan Ġpig gy - back Ġon Ġthe Ġresulting Ġrepresentation Ġafter Ġfine - tun ing Ġit Ġfor Ġclassification Ġusing Ġrelatively Ġsmall Ġlabeled Ġdata . ĠWhile Ġsimilar Ġpre - train - then - fine - t une Ġapproaches Ġhave Ġrecently Ġachieved Ġstate - of - the - art Ġperformance Ġin Ġa Ġnumber Ġof ĠN LP Ġtasks âĢĶ including Ġnatural Ġlanguage Ġinference , Ġquestion Ġanswering , Ġand Ġcommons ense Ġreasoning Ġ( disc ussed Ġin ĠSection ĠSEC REF 2 )âĢĶ to Ġthe Ġbest Ġof Ġour Ġknowledge Ġthis Ġis Ġthe Ġfirst Ġattempt Ġat Ġapplying Ġthis Ġparadigm Ġto Ġconvers ational Ġforecasting . </s> To Ġtest Ġthe Ġeffectiveness Ġof Ġthis Ġnew Ġarchitecture Ġin Ġforecasting Ġderail ment Ġof Ġonline Ġconversations , Ġwe Ġdevelop Ġand Ġdistribute Ġtwo Ġnew Ġdatasets . ĠThe Ġfirst Ġtri ples Ġin Ġsize Ġthe Ġhighly Ġcurated Ġ` Con vers ations ĠGone ĠAw ry ' Ġdataset ĠB IB REF 9 , Ġwhere Ġcivil - starting ĠWikipedia ĠTalk ĠPage Ġconversations Ġare Ġcrowd - label ed Ġaccording Ġto Ġwhether Ġthey Ġeventually Ġlead Ġto Ġpersonal Ġattacks ; Ġthe Ġsecond Ġrelies Ġon Ġin - the - wild Ġmoderation Ġof Ġthe Ġpopular Ġsubreddit ĠChange My View , Ġwhere Ġthe Ġaim Ġis Ġto Ġforecast Ġwhether Ġa Ġdiscussion Ġwill Ġlater Ġbe Ġsubject Ġto Ġmoderator Ġaction Ġdue Ġto ĠâĢ ľ r ude Ġor Ġhostile âĢ Ŀ Ġbehavior . ĠIn Ġboth Ġdatasets , Ġour Ġmodel Ġoutper forms Ġexisting Ġfixed - window Ġapproaches , Ġas Ġwell Ġas Ġsimpler Ġsequential Ġbas elines Ġthat Ġcannot Ġaccount Ġfor Ġinter - comment Ġrelations . ĠFurthermore , Ġby Ġvirtue Ġof Ġits Ġonline Ġprocessing Ġof Ġthe Ġconversation , Ġour Ġsystem Ġcan Ġprovide Ġsubstantial Ġprior Ġnotice Ġof Ġupcoming Ġderail ment , Ġtriggering Ġon Ġaverage Ġ3 Ġcomments Ġ( or Ġ3 Ġhours ) Ġbefore Ġan Ġovertly Ġtoxic Ġcomment Ġis Ġposted . </s> To Ġsummarize , Ġin Ġthis Ġwork Ġwe : </s> introdu ce Ġthe Ġfirst Ġmodel Ġfor Ġforecasting Ġconvers ational Ġevents Ġthat Ġcan Ġcapture Ġthe Ġdynamics Ġof Ġa Ġconversation Ġas Ġit Ġdevelops ; </s> build Ġtwo Ġdiverse Ġdatasets Ġ( one Ġentirely Ġnew , Ġone Ġextending Ġprior Ġwork ) Ġfor Ġthe Ġtask Ġof Ġforecasting Ġderail ment Ġof Ġonline Ġconversations ; </s> comp are Ġthe Ġperformance Ġof Ġour Ġmodel Ġagainst Ġthe Ġcurrent Ġstate - of - the - art , Ġand Ġevaluate Ġits Ġability Ġto Ġprovide Ġearly Ġwarning Ġsigns . </s> Our Ġwork Ġis Ġmotivated Ġby Ġthe Ġgoal Ġof Ġassisting Ġhuman Ġmoderators Ġof Ġonline Ġcommunities Ġby Ġpreempt ively Ġsignaling Ġat - risk Ġconversations Ġthat Ġmight Ġdeserve Ġtheir Ġattention . ĠHowever , Ġwe Ġcaution Ġthat Ġany Ġautomated Ġsystems Ġmight Ġencode Ġor Ġeven Ġamplify Ġthe Ġbiases Ġexisting Ġin Ġthe Ġtraining Ġdata ĠB IB REF 26 , ĠB IB REF 27 , ĠB IB REF 28 , Ġso Ġa Ġpublic - facing Ġimplementation Ġwould Ġneed Ġto Ġbe Ġexhaust ively Ġscrutin ized Ġfor Ġsuch Ġbiases ĠB IB REF 29 . </s> Further ĠRelated ĠWork </s> Ant is ocial Ġbehavior . ĠAnt is ocial Ġbehavior Ġonline Ġcomes Ġin Ġmany Ġforms , Ġincluding Ġharassment ĠB IB REF 30 , Ġcyber bull ying ĠB IB REF 31 , Ġand Ġgeneral Ġaggression ĠB IB REF 32 . ĠPrior Ġwork Ġhas Ġsought Ġto Ġunderstand Ġdifferent Ġaspects Ġof Ġsuch Ġbehavior , Ġincluding Ġits Ġeffect Ġon Ġthe Ġcommunities Ġwhere Ġit Ġhappens ĠB IB REF 33 , ĠB IB REF 34 , Ġthe Ġactors Ġinvolved ĠB IB REF 35 , ĠB IB REF 36 , ĠB IB REF 37 , ĠB IB REF 38 Ġand Ġconnections Ġto Ġthe Ġoutside Ġworld ĠB IB REF 39 . </s> Post - h oc Ġclassification Ġof Ġconversations . ĠThere Ġis Ġa Ġrich Ġbody Ġof Ġprior Ġwork Ġon Ġclass ifying Ġthe Ġoutcome Ġof Ġa Ġconversation Ġafter Ġit Ġhas Ġconcluded , Ġor Ġclass ifying Ġconvers ational Ġevents Ġafter Ġthey Ġhappened . ĠMany Ġexamples Ġexist , Ġbut Ġsome Ġmore Ġclosely Ġrelated Ġto Ġour Ġpresent Ġwork Ġinclude Ġidentifying Ġthe Ġwinner Ġof Ġa Ġdebate ĠB IB REF 40 , ĠB IB REF 41 , ĠB IB REF 42 , Ġidentifying Ġsuccessful Ġnegotiations ĠB IB REF 21 , ĠB IB REF 43 , Ġas Ġwell Ġas Ġdetecting Ġwhether Ġdeception ĠB IB REF 44 , ĠB IB REF 45 , ĠB IB REF 46 Ġor Ġdisagreement ĠB IB REF 47 , ĠB IB REF 48 , ĠB IB REF 49 , ĠB IB REF 50 , ĠB IB REF 51 Ġhas Ġoccurred . </s> Our Ġgoal Ġis Ġdifferent Ġbecause Ġwe Ġwish Ġto Ġforecast Ġconvers ational Ġevents Ġbefore Ġthey Ġhappen Ġand Ġwhile Ġthe Ġconversation Ġis Ġstill Ġongoing Ġ( pot entially Ġallowing Ġfor Ġinterventions ). ĠNote Ġthat Ġsome Ġpost - h oc Ġtasks Ġcan Ġalso Ġbe Ġre - fram ed Ġas Ġforecasting Ġtasks Ġ( assuming Ġthe Ġexistence Ġof Ġnecessary Ġlabels ); Ġfor Ġinstance , Ġpredicting Ġwhether Ġan Ġongoing Ġconversation Ġwill Ġeventually Ġspark Ġdisagreement ĠB IB REF 18 , Ġrather Ġthan Ġdetecting Ġalready - existing Ġdisagreement . </s> Con vers ational Ġforecasting . ĠAs Ġdescribed Ġin ĠSection ĠSEC REF 1 , Ġprior Ġwork Ġon Ġforecasting Ġconvers ational Ġoutcomes Ġand Ġevents Ġhas Ġlargely Ġrelied Ġon Ġhand - crafted Ġfeatures Ġto Ġcapture Ġaspects Ġof Ġconvers ational Ġdynamics . ĠExample Ġfeature Ġsets Ġinclude Ġstatistical Ġmeasures Ġbased Ġon Ġsimilarity Ġbetween Ġutter ances ĠB IB REF 16 , Ġsentiment Ġimbalance ĠB IB REF 20 , Ġflow Ġof Ġideas ĠB IB REF 20 , Ġincrease Ġin Ġhostility ĠB IB REF 8 , Ġreply Ġrate ĠB IB REF 11 Ġand Ġgraph Ġrepresentations Ġof Ġconversations ĠB IB REF 52 , ĠB IB REF 17 . ĠBy Ġcontrast , Ġwe Ġaim Ġto Ġautomatically Ġlearn Ġneural Ġrepresentations Ġof Ġconvers ational Ġdynamics Ġthrough Ġpre - training . </s> Such Ġhand - crafted Ġfeatures Ġare Ġtypically Ġextracted Ġfrom Ġfixed - length Ġwindows Ġof Ġthe Ġconversation , Ġleaving Ġun add ressed Ġthe Ġproblem Ġof Ġunknown Ġhorizon . ĠWhile Ġsome Ġwork Ġhas Ġtrained Ġmultiple Ġmodels Ġfor Ġdifferent Ġwindow - length s ĠB IB REF 8 , ĠB IB REF 18 , Ġthey Ġconsider Ġthese Ġmodels Ġto Ġbe Ġindependent Ġand , Ġas Ġsuch , Ġdo Ġnot Ġaddress Ġthe Ġissue Ġof Ġaggreg ating Ġthem Ġinto Ġa Ġsingle Ġforecast Ġ( i . e ., Ġdeciding Ġat Ġwhat Ġpoint Ġto Ġmake Ġa Ġprediction ). ĠWe Ġimplement Ġa Ġsimple Ġsliding Ġwindows Ġsolution Ġas Ġa Ġbaseline Ġ( Section ĠSEC REF 5 ). </s> Pre - training Ġfor ĠN LP . ĠThe Ġuse Ġof Ġpre - training Ġfor Ġnatural Ġlanguage Ġtasks Ġhas Ġbeen Ġgrowing Ġin Ġpopularity Ġafter Ġrecent Ġbreakthrough s Ġdemonstrating Ġimproved Ġperformance Ġon Ġa Ġwide Ġarray Ġof Ġbenchmark Ġtasks ĠB IB REF 53 , ĠB IB REF 54 . ĠEx isting Ġwork Ġhas Ġgenerally Ġused Ġa Ġlanguage Ġmodeling Ġobjective Ġas Ġthe Ġpre - training Ġobjective ; Ġexamples Ġinclude Ġnext - word Ġprediction ĠB IB REF 55 , Ġsentence Ġaut oen c oding , ĠB IB REF 56 , Ġand Ġmachine Ġtranslation ĠB IB REF 57 . ĠB ERT ĠB IB REF 58 Ġintroduces Ġa Ġvariation Ġon Ġthis Ġin Ġwhich Ġthe Ġgoal Ġis Ġto Ġpredict Ġthe Ġnext Ġsentence Ġin Ġa Ġdocument Ġgiven Ġthe Ġcurrent Ġsentence . ĠOur Ġpre - training Ġobjective Ġis Ġsimilar Ġin Ġspirit , Ġbut Ġoperates Ġat Ġa Ġconversation Ġlevel , Ġrather Ġthan Ġa Ġdocument Ġlevel . ĠWe Ġhence Ġview Ġour Ġobjective Ġas Ġconvers ational Ġmodeling Ġrather Ġthan Ġ( only ) Ġlanguage Ġmodeling . ĠFurthermore , Ġwhile ĠB ERT 's Ġsentence Ġprediction Ġobjective Ġis Ġframed Ġas Ġa Ġmultiple - choice Ġtask , Ġour Ġobjective Ġis Ġframed Ġas Ġa Ġgener ative Ġtask . </s> Der ail ment ĠDat as ets </s> We Ġconsider Ġtwo Ġdatasets , Ġrepresenting Ġrelated Ġbut Ġslightly Ġdifferent Ġforecasting Ġtasks . ĠThe Ġfirst Ġdataset Ġis Ġan Ġexpanded Ġversion Ġof Ġthe Ġannot ated ĠWikipedia Ġconversations Ġdataset Ġfrom ĠB IB REF 9 . ĠThis Ġdataset Ġuses Ġcarefully - controlled Ġcrowds ourced Ġlabels , Ġstrictly Ġfiltered Ġto Ġensure Ġthe Ġconversations Ġare Ġcivil Ġup Ġto Ġthe Ġmoment Ġof Ġa Ġpersonal Ġattack . ĠThis Ġis Ġa Ġuseful Ġproperty Ġfor Ġthe Ġpurposes Ġof Ġmodel Ġanalysis , Ġand Ġhence Ġwe Ġfocus Ġon Ġthis Ġas Ġour Ġprimary Ġdataset . ĠHowever , Ġwe Ġare Ġconscious Ġof Ġthe Ġpossibility Ġthat Ġthese Ġstrict Ġlabels Ġmay Ġnot Ġfully Ġcapture Ġthe Ġkind Ġof Ġbehavior Ġthat Ġmoderators Ġcare Ġabout Ġin Ġpractice . ĠWe Ġtherefore Ġintroduce Ġa Ġsecondary Ġdataset , Ġconstructed Ġfrom Ġthe Ġsubreddit ĠChange My View Ġ( CM V ) Ġthat Ġdoes Ġnot Ġuse Ġpost - h oc Ġannotations . ĠInstead , Ġthe Ġprediction Ġtask Ġis Ġto Ġforecast Ġwhether Ġthe Ġconversation Ġwill Ġbe Ġsubject Ġto Ġmoderator Ġaction Ġin Ġthe Ġfuture . </s> Wikipedia Ġdata . ĠB IB REF 9 's Ġ` Con vers ations ĠGone ĠAw ry ' Ġdataset Ġconsists Ġof Ġ1 , 270 Ġconversations Ġthat Ġtook Ġplace Ġbetween ĠWikipedia Ġeditors Ġon Ġpublicly Ġaccessible Ġtalk Ġpages . ĠThe Ġconversations Ġare Ġsourced Ġfrom Ġthe ĠWiki Con v Ġdataset ĠB IB REF 59 Ġand Ġlabeled Ġby Ġcrowd workers Ġas Ġeither Ġcontaining Ġa Ġpersonal Ġattack Ġfrom Ġwithin Ġ( i . e ., Ġhostile Ġbehavior Ġby Ġone Ġuser Ġin Ġthe Ġconversation Ġdirected Ġtowards Ġanother ) Ġor Ġremaining Ġcivil Ġthroughout . </s> A Ġseries Ġof Ġcontrols Ġare Ġimplemented Ġto Ġprevent Ġmodels Ġfrom Ġpicking Ġup Ġon Ġtrivial Ġcorrelations . ĠTo Ġprevent Ġmodels Ġfrom Ġcapturing Ġtopic - specific Ġinformation Ġ( e . g ., Ġpolitical Ġconversations Ġare Ġmore Ġlikely Ġto Ġderail ), Ġeach Ġattack - containing Ġconversation Ġis Ġpaired Ġwith Ġa Ġclean Ġconversation Ġfrom Ġthe Ġsame Ġtalk Ġpage , Ġwhere Ġthe Ġtalk Ġpage Ġserves Ġas Ġa Ġproxy Ġfor Ġtopic . ĠTo Ġforce Ġmodels Ġto Ġactually Ġcapture Ġconvers ational Ġdynamics Ġrather Ġthan Ġdetecting Ġalready - existing Ġtoxicity , Ġhuman Ġannotations Ġare Ġused Ġto Ġensure Ġthat Ġall Ġcomments Ġpreceding Ġa Ġpersonal Ġattack Ġare Ġcivil . </s> To Ġthe Ġends Ġof Ġmore Ġeffective Ġmodel Ġtraining , Ġwe Ġelected Ġto Ġexpand Ġthe Ġ` Con vers ations ĠGone ĠAw ry ' Ġdataset , Ġusing Ġthe Ġoriginal Ġannotation Ġprocedure . ĠSince Ġwe Ġfound Ġthat Ġthe Ġoriginal Ġdata Ġskewed Ġtowards Ġshorter Ġconversations , Ġwe Ġfocused Ġthis Ġcrowds ourcing Ġrun Ġon Ġlonger Ġconversations : Ġones Ġwith Ġ4 Ġor Ġmore Ġcomments Ġpreceding Ġthe Ġattack . ĠThrough Ġthis Ġadditional Ġcrowds ourcing , Ġwe Ġexpand Ġthe Ġdataset Ġto Ġ4 , 188 Ġconversations , Ġwhich Ġwe Ġare Ġpublicly Ġreleasing Ġas Ġpart Ġof Ġthe ĠCornell ĠConvers ational ĠAnalysis ĠTool kit Ġ( Con vo Kit ). </s> We Ġperform Ġan Ġ80 - 20 - 20 Ġtrain / dev / test Ġsplit , Ġensuring Ġthat Ġpaired Ġconversations Ġend Ġup Ġin Ġthe Ġsame Ġsplit Ġin Ġorder Ġto Ġpreserve Ġthe Ġtopic Ġcontrol . ĠFinally , Ġwe Ġrandomly Ġsample Ġanother Ġ1 Ġmillion Ġconversations Ġfrom ĠWiki Con v Ġto Ġuse Ġfor Ġthe Ġun super vised Ġpre - training Ġof Ġthe Ġgener ative Ġcomponent . </s> Reddit ĠCM V Ġdata . ĠThe ĠCM V Ġdataset Ġis Ġconstructed Ġfrom Ġconversations Ġcollected Ġvia Ġthe ĠReddit ĠAPI . ĠIn Ġcontrast Ġto Ġthe ĠWikipedia - based Ġdataset , Ġwe Ġexplicitly Ġavoid Ġthe Ġuse Ġof Ġpost - h oc Ġannotation . ĠInstead , Ġwe Ġuse Ġas Ġour Ġlabel Ġwhether Ġa Ġconversation Ġeventually Ġhad Ġa Ġcomment Ġremoved Ġby Ġa Ġmoderator Ġfor Ġviolation Ġof ĠRule Ġ2 : ĠâĢ ľ Don 't Ġbe Ġrude Ġor Ġhostile Ġto Ġother Ġusers âĢ Ŀ . </s> Though Ġthe Ġlack Ġof Ġpost - h oc Ġannotation Ġlimits Ġthe Ġdegree Ġto Ġwhich Ġwe Ġcan Ġimpose Ġcontrols Ġon Ġthe Ġdata Ġ( e . g ., Ġsome Ġconversations Ġmay Ġcontain Ġtoxic Ġcomments Ġnot Ġflagged Ġby Ġthe Ġmoderators ) Ġwe Ġdo Ġreproduce Ġas Ġmany Ġof Ġthe ĠWikipedia Ġdata 's Ġcontrols Ġas Ġwe Ġcan . ĠName ly , Ġwe Ġreplicate Ġthe Ġtopic Ġcontrol Ġpairing Ġby Ġchoosing Ġpairs Ġof Ġpositive Ġand Ġnegative Ġexamples Ġthat Ġbelong Ġto Ġthe Ġsame Ġtop - level Ġpost , Ġfollowing ĠB IB REF 12 ; Ġand Ġenforce Ġthat Ġthe Ġremoved Ġcomment Ġwas Ġmade Ġby Ġa Ġuser Ġwho Ġwas Ġpreviously Ġinvolved Ġin Ġthe Ġconversation . ĠThis Ġprocess Ġresults Ġin Ġ6 , 8 42 Ġconversations , Ġto Ġwhich Ġwe Ġagain Ġapply Ġa Ġpair - pres erving Ġ80 - 20 - 20 Ġsplit . ĠFinally , Ġwe Ġgather Ġover Ġ600 , 000 Ġconversations Ġthat Ġdo Ġnot Ġinclude Ġany Ġremoved Ġcomment , Ġfor Ġun super vised Ġpre - training . </s> Con vers ational ĠFore casting ĠModel </s> We Ġnow Ġdescribe Ġour Ġgeneral Ġmodel Ġfor Ġforecasting Ġfuture Ġconvers ational Ġevents . ĠOur Ġmodel Ġintegrates Ġtwo Ġcomponents : Ġ( a ) Ġa Ġgener ative Ġdialog Ġmodel Ġthat Ġlearns Ġto Ġrepresent Ġconvers ational Ġdynamics Ġin Ġan Ġun super vised Ġfashion ; Ġand Ġ( b ) Ġa Ġsupervised Ġcomponent Ġthat Ġfine - t unes Ġthis Ġrepresentation Ġto Ġforecast Ġfuture Ġevents . ĠFigure ĠFIG REF 13 Ġprovides Ġan Ġoverview Ġof Ġthe Ġproposed Ġarchitecture , Ġhence forth ĠCRA FT Ġ( Con vers ational ĠRec urrent ĠArchitecture Ġfor ĠFore cas T ing ). </s> Termin ology . ĠFor Ġmodeling Ġpurposes , Ġwe Ġtreat Ġa Ġconversation Ġas Ġa Ġsequence Ġof Ġ$ N $ Ġcomments Ġ$ C Ġ= Ġ\ l brace Ġc _ 1 , \ d ots Ġ, c _ N \ r brace Ġ$ . ĠEach Ġcomment , Ġin Ġturn , Ġis Ġa Ġsequence Ġof Ġtokens , Ġwhere Ġthe Ġnumber Ġof Ġtokens Ġmay Ġvary Ġfrom Ġcomment Ġto Ġcomment . ĠFor Ġthe Ġ$ n $ - th Ġcomment Ġ($ 1 Ġ\ le Ġn Ġ\ le ĠN ) $, Ġwe Ġlet Ġ$ M _ n $ Ġdenote Ġthe Ġnumber Ġof Ġtokens . ĠThen , Ġa Ġcomment Ġ$ c _ n $ Ġcan Ġbe Ġrepresented Ġas Ġa Ġsequence Ġof Ġ$ M _ n $ Ġtokens : Ġ$ c _ n Ġ= Ġ\ l brace Ġw _ 1 , \ d ots Ġ, w _{ M _ n }\ r brace Ġ$ . </s> Gener ative Ġcomponent . ĠFor Ġthe Ġgener ative Ġcomponent Ġof Ġour Ġmodel , Ġwe Ġuse Ġa Ġhierarchical Ġrecurrent Ġenc oder - dec oder Ġ( HR ED ) Ġarchitecture ĠB IB REF 60 , Ġa Ġmodified Ġversion Ġof Ġthe Ġpopular Ġsequence - to - sequence Ġ( seq 2 seq ) Ġarchitecture ĠB IB REF 61 Ġdesigned Ġto Ġaccount Ġfor Ġdependencies Ġbetween Ġconsecutive Ġinputs . ĠB IB REF 23 Ġshowed Ġthat ĠHR ED Ġcan Ġsuccessfully Ġmodel Ġconvers ational Ġcontext Ġby Ġencoding Ġthe Ġtemporal Ġstructure Ġof Ġpreviously Ġseen Ġcomments , Ġmaking Ġit Ġan Ġideal Ġfit Ġfor Ġour Ġuse Ġcase . ĠHere , Ġwe Ġprovide Ġa Ġhigh - level Ġsummary Ġof Ġthe ĠHR ED Ġarchitecture , Ġdefer ring Ġdeeper Ġtechnical Ġdiscussion Ġto ĠB IB REF 60 Ġand ĠB IB REF 23 . </s> An ĠHR ED Ġdialog Ġmodel Ġconsists Ġof Ġthree Ġcomponents : Ġan Ġutter ance Ġenc oder , Ġa Ġcontext Ġenc oder , Ġand Ġa Ġdec oder . ĠThe Ġutter ance Ġenc oder Ġis Ġresponsible Ġfor Ġgenerating Ġsemantic Ġvector Ġrepresentations Ġof Ġcomments . ĠIt Ġconsists Ġof Ġa Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġthat Ġreads Ġa Ġcomment Ġtoken - by - token , Ġand Ġon Ġeach Ġtoken Ġ$ w _ m $ Ġupdates Ġa Ġhidden Ġstate Ġ$ h ^ {\ text { enc }} $ Ġbased Ġon Ġthe Ġcurrent Ġtoken Ġand Ġthe Ġprevious Ġhidden Ġstate : </s> where Ġ$ f ^ {\ text { R NN }} $ Ġis Ġa Ġnon linear Ġg ating Ġfunction Ġ( our Ġimplementation Ġuses ĠGR U ĠB IB REF 62 ). ĠThe Ġfinal Ġhidden Ġstate Ġ$ h ^ {\ text { enc }} _ M $ Ġcan Ġbe Ġviewed Ġas Ġa Ġvector Ġencoding Ġof Ġthe Ġentire Ġcomment . </s> Running Ġthe Ġenc oder Ġon Ġeach Ġcomment Ġ$ c _ n $ Ġresults Ġin Ġa Ġsequence Ġof Ġ$ N $ Ġvector Ġenc od ings . ĠA Ġsecond Ġenc oder , Ġthe Ġcontext Ġenc oder , Ġis Ġthen Ġrun Ġover Ġthis Ġsequence : </s> Each Ġhidden Ġstate Ġ$ h ^ {\ text { con }} _ n $ Ġcan Ġthen Ġbe Ġviewed Ġas Ġan Ġencoding Ġof Ġthe Ġfull Ġconvers ational Ġcontext Ġup Ġto Ġand Ġincluding Ġthe Ġ$ n $ - th Ġcomment . ĠTo Ġgenerate Ġa Ġresponse Ġto Ġcomment Ġ$ n $, Ġthe Ġcontext Ġencoding Ġ$ h ^ {\ text { con }} _ n $ Ġis Ġused Ġto Ġinitialize Ġthe Ġhidden Ġstate Ġ$ h ^ {\ text { dec }} _{ 0 } $ Ġof Ġa Ġdec oder ĠR NN . ĠThe Ġdec oder Ġproduces Ġa Ġresponse Ġtoken Ġby Ġtoken Ġusing Ġthe Ġfollowing Ġrec urrence : </s> where Ġ$ f ^ {\ text { out }} $ Ġis Ġsome Ġfunction Ġthat Ġoutputs Ġa Ġprobability Ġdistribution Ġover Ġwords ; Ġwe Ġimplement Ġthis Ġusing Ġa Ġsimple Ġfeed forward Ġlayer . ĠIn Ġour Ġimplementation , Ġwe Ġfurther Ġaugment Ġthe Ġdec oder Ġwith Ġattention ĠB IB REF 63 , ĠB IB REF 64 Ġover Ġcontext Ġenc oder Ġstates Ġto Ġhelp Ġcapture Ġlong - term Ġinter - comment Ġdependencies . ĠThis Ġgener ative Ġcomponent Ġcan Ġbe Ġpre - trained Ġusing Ġunl abel ed Ġconvers ational Ġdata . </s> Pred iction Ġcomponent . ĠGiven Ġa Ġpre - trained ĠHR ED Ġdialog Ġmodel , Ġwe Ġaim Ġto Ġextend Ġthe Ġmodel Ġto Ġpredict Ġfrom Ġthe Ġconvers ational Ġcontext Ġwhether Ġthe Ġto - be - fore cast ed Ġevent Ġwill Ġoccur . ĠOur Ġpredictor Ġconsists Ġof Ġa Ġmult il ayer Ġpercept ron Ġ( ML P ) Ġwith Ġ3 Ġfully - connected Ġlayers , Ġleak y ĠRe LU Ġactiv ations Ġbetween Ġlayers , Ġand Ġs igm oid Ġactivation Ġfor Ġoutput . ĠFor Ġeach Ġcomment Ġ$ c _ n $, Ġthe Ġpredictor Ġtakes Ġas Ġinput Ġthe Ġcontext Ġencoding Ġ$ h ^ {\ text { con }} _ n $ Ġand Ġforwards Ġit Ġthrough Ġthe ĠML P Ġlayers , Ġresulting Ġin Ġan Ġoutput Ġscore Ġthat Ġis Ġinterpreted Ġas Ġa Ġprobability Ġ$ p _{ \ text { event }} ( c _{ n + 1 }) $ Ġthat Ġthe Ġto - be - fore cast ed Ġevent Ġwill Ġhappen Ġ( e . g ., Ġthat Ġthe Ġconversation Ġwill Ġderail ). </s> Training Ġthe Ġpredictive Ġcomponent Ġstarts Ġby Ġinitial izing Ġthe Ġweights Ġof Ġthe Ġenc od ers Ġto Ġthe Ġvalues Ġlearned Ġin Ġpre - training . ĠThe Ġmain Ġtraining Ġloop Ġthen Ġworks Ġas Ġfollows : Ġfor Ġeach Ġpositive Ġsample âĢĶ i . e ., Ġa Ġconversation Ġcontaining Ġan Ġinstance Ġof Ġthe Ġto - be - fore cast ed Ġevent Ġ( e . g ., Ġderail ment ) Ġat Ġcomment Ġ$ c _ e $ âĢĶ we Ġfeed Ġthe Ġcontext Ġ$ c _ 1 , \ d ots Ġ, c _{ e - 1 } $ Ġthrough Ġthe Ġenc oder Ġand Ġclass ifier , Ġand Ġcompute Ġcross - ent ropy Ġloss Ġbetween Ġthe Ġclass ifier Ġoutput Ġand Ġexpected Ġoutput Ġof Ġ1 . ĠSimilarly , Ġfor Ġeach Ġnegative Ġsample âĢĶ i . e ., Ġa Ġconversation Ġwhere Ġnone Ġof Ġthe Ġcomments Ġexhibit Ġthe Ġto - be - fore cast ed Ġevent Ġand Ġthat Ġends Ġwith Ġ$ c _ N $ âĢĶ we Ġfeed Ġthe Ġcontext Ġ$ c _ 1 , \ d ots Ġ, c _{ N - 1 } $ Ġthrough Ġthe Ġmodel Ġand Ġcompute Ġloss Ġagainst Ġan Ġexpected Ġoutput Ġof Ġ0 . </s> Note Ġthat Ġthe Ġparameters Ġof Ġthe Ġgener ative Ġcomponent Ġare Ġnot Ġheld Ġfixed Ġduring Ġthis Ġprocess ; Ġinstead , Ġback prop ag ation Ġis Ġallowed Ġto Ġgo Ġall Ġthe Ġway Ġthrough Ġthe Ġenc oder Ġlayers . ĠThis Ġprocess , Ġknown Ġas Ġfine - tun ing , Ġresh apes Ġthe Ġrepresentation Ġlearned Ġduring Ġpre - training Ġto Ġbe Ġmore Ġdirectly Ġuseful Ġto Ġprediction ĠB IB REF 55 . </s> We Ġimplement Ġthe Ġmodel Ġand Ġtraining Ġcode Ġusing ĠPy Tor ch , Ġand Ġwe Ġare Ġpublicly Ġreleasing Ġour Ġimplementation Ġand Ġthe Ġtrained Ġmodels Ġtogether Ġwith Ġthe Ġdata Ġas Ġpart Ġof ĠCon vo Kit . </s> Fore casting ĠDer ail ment </s> We Ġevaluate Ġthe Ġperformance Ġof ĠCRA FT Ġin Ġthe Ġtask Ġof Ġforecasting Ġconvers ational Ġderail ment Ġin Ġboth Ġthe ĠWikipedia Ġand ĠCM V Ġscenarios . ĠTo Ġthis Ġend , Ġfor Ġeach Ġof Ġthese Ġdatasets Ġwe Ġpre - train Ġthe Ġgener ative Ġcomponent Ġon Ġthe Ġunl abel ed Ġportion Ġof Ġthe Ġdata Ġand Ġfine - t une Ġit Ġon Ġthe Ġlabeled Ġtraining Ġsplit Ġ( data Ġsize Ġdetailed Ġin ĠSection ĠSEC REF 3 ). </s> In Ġorder Ġto Ġevaluate Ġour Ġsequential Ġsystem Ġagainst Ġconvers ational - level Ġground Ġtruth , Ġwe Ġneed Ġto Ġaggregate Ġcomment Ġlevel Ġpredictions . ĠIf Ġany Ġcomment Ġin Ġthe Ġconversation Ġtriggers Ġa Ġpositive Ġprediction âĢĶ i . e ., Ġ$ p _{ \ text { event }} ( c _{ n + 1 }) $ Ġis Ġgreater Ġthan Ġa Ġthreshold Ġlearned Ġon Ġthe Ġdevelopment Ġsplit âĢĶ then Ġthe Ġrespective Ġconversation Ġis Ġpredicted Ġto Ġderail . ĠIf Ġthis Ġforecast Ġis Ġtriggered Ġin Ġa Ġconversation Ġthat Ġactually Ġder ails , Ġbut Ġbefore Ġthe Ġderail ment Ġactually Ġhappens , Ġthen Ġthe Ġconversation Ġis Ġcounted Ġas Ġa Ġtrue Ġpositive ; Ġotherwise Ġit Ġis Ġa Ġfalse Ġpositive . ĠIf Ġno Ġpositive Ġpredictions Ġare Ġtriggered Ġfor Ġa Ġconversation , Ġbut Ġit Ġactually Ġder ails Ġthen Ġit Ġcounts Ġas Ġa Ġfalse Ġnegative ; Ġif Ġit Ġdoes Ġnot Ġderail Ġthen Ġit Ġis Ġa Ġtrue Ġnegative . </s> Fixed - length Ġwindow Ġbas elines . ĠWe Ġfirst Ġseek Ġto Ġcompare ĠCRA FT Ġto Ġexisting , Ġfixed - length Ġwindow Ġapproaches Ġto Ġforecasting . ĠTo Ġthis Ġend , Ġwe Ġimplement Ġtwo Ġsuch Ġbas elines : ĠAw ry , Ġwhich Ġis Ġthe Ġstate - of - the - art Ġmethod Ġproposed Ġin ĠB IB REF 9 Ġbased Ġon Ġpragmatic Ġfeatures Ġin Ġthe Ġfirst Ġcomment - reply Ġpair , Ġand ĠBo W , Ġa Ġsimple Ġbag - of - words Ġbaseline Ġthat Ġmakes Ġa Ġprediction Ġusing ĠTF - ID F Ġweighted Ġbag - of - words Ġfeatures Ġextracted Ġfrom Ġthe Ġfirst Ġcomment - reply Ġpair . </s> Online Ġforecasting Ġbas elines . ĠNext , Ġwe Ġconsider Ġsimpler Ġapproaches Ġfor Ġmaking Ġforecasts Ġas Ġthe Ġconversations Ġhappen Ġ( i . e ., Ġin Ġan Ġonline Ġfashion ). ĠFirst , Ġwe Ġpropose ĠCum ulative ĠBo W , Ġa Ġmodel Ġthat Ġrecomp utes Ġbag - of - words Ġfeatures Ġon Ġall Ġcomments Ġseen Ġthus Ġfar Ġevery Ġtime Ġa Ġnew Ġcomment Ġarrives . ĠWhile Ġthis Ġapproach Ġdoes Ġexhibit Ġthe Ġdesired Ġbehavior Ġof Ġproducing Ġupdated Ġpredictions Ġfor Ġeach Ġnew Ġcomment , Ġit Ġfails Ġto Ġaccount Ġfor Ġrelationships Ġbetween Ġcomments . </s> This Ġsimple Ġcumulative Ġapproach Ġcannot Ġbe Ġdirectly Ġextended Ġto Ġmodels Ġwhose Ġfeatures Ġare Ġstrictly Ġbased Ġon Ġa Ġfixed Ġnumber Ġof Ġcomments , Ġlike ĠAw ry . ĠAn Ġalternative Ġis Ġto Ġuse Ġa Ġsliding Ġwindow : Ġfor Ġa Ġfeature Ġset Ġbased Ġon Ġa Ġwindow Ġof Ġ$ W $ Ġcomments , Ġupon Ġeach Ġnew Ġcomment Ġwe Ġcan Ġextract Ġfeatures Ġfrom Ġa Ġwindow Ġcontaining Ġthat Ġcomment Ġand Ġthe Ġ$ W - 1 $ Ġcomments Ġpreceding Ġit . ĠWe Ġapply Ġthis Ġto Ġthe ĠAw ry Ġmethod Ġand Ġcall Ġthis Ġmodel ĠSl iding ĠAw ry . ĠFor Ġboth Ġthese Ġbas elines , Ġwe Ġaggregate Ġcomment - level Ġpredictions Ġin Ġthe Ġsame Ġway Ġas Ġin Ġour Ġmain Ġmodel . </s> C RAFT Ġabl ations . ĠFinally , Ġwe Ġconsider Ġtwo Ġmodified Ġversions Ġof Ġthe ĠCRA FT Ġmodel Ġin Ġorder Ġto Ġevaluate Ġthe Ġimpact Ġof Ġtwo Ġof Ġits Ġkey Ġcomponents : Ġ( 1 ) Ġthe Ġpre - training Ġstep , Ġand Ġ( 2 ) Ġits Ġability Ġto Ġcapture Ġinter - comment Ġdependencies Ġthrough Ġits Ġhierarchical Ġmemory . </s> To Ġevaluate Ġthe Ġimpact Ġof Ġpre - training , Ġwe Ġtrain Ġthe Ġprediction Ġcomponent Ġof ĠCRA FT Ġon Ġonly Ġthe Ġlabeled Ġtraining Ġdata , Ġwithout Ġfirst Ġpre - training Ġthe Ġenc oder Ġlayers Ġwith Ġthe Ġunl abel ed Ġdata . ĠWe Ġfind Ġthat Ġgiven Ġthe Ġrelatively Ġsmall Ġsize Ġof Ġlabeled Ġdata , Ġthis Ġbaseline Ġfails Ġto Ġsuccessfully Ġlearn , Ġand Ġends Ġup Ġperforming Ġat Ġthe Ġlevel Ġof Ġrandom Ġguessing . ĠThis Ġresult Ġunderscores Ġthe Ġneed Ġfor Ġthe Ġpre - training Ġstep Ġthat Ġcan Ġmake Ġuse Ġof Ġunl abel ed Ġdata . </s> To Ġevaluate Ġthe Ġimpact Ġof Ġthe Ġhierarchical Ġmemory , Ġwe Ġimplement Ġa Ġsimplified Ġversion Ġof ĠCRA FT Ġwhere Ġthe Ġmemory Ġsize Ġof Ġthe Ġcontext Ġenc oder Ġis Ġzero Ġ( C RAFT Ġ$ -$ ĠCE ), Ġthus Ġeffectively Ġacting Ġas Ġif Ġthe Ġpre - training Ġcomponent Ġis Ġa Ġvanilla Ġseq 2 seq Ġmodel . ĠIn Ġother Ġwords , Ġthis Ġmodel Ġcannot Ġcapture Ġinter - comment Ġdependencies , Ġand Ġinstead Ġat Ġeach Ġstep Ġmakes Ġa Ġprediction Ġbased Ġonly Ġon Ġthe Ġutter ance Ġencoding Ġof Ġthe Ġlatest Ġcomment . </s> Results . ĠTable ĠT AB REF 17 Ġcompares ĠCRA FT Ġto Ġthe Ġbas elines Ġon Ġthe Ġtest Ġsplits Ġ( random Ġbaseline Ġis Ġ50 %) Ġand Ġillustrates Ġseveral Ġkey Ġfindings . ĠFirst , Ġwe Ġfind Ġthat Ġunsurprisingly , Ġaccounting Ġfor Ġfull Ġconvers ational Ġcontext Ġis Ġindeed Ġhelpful , Ġwith Ġeven Ġthe Ġsimple Ġonline Ġbas elines Ġoutper forming Ġthe Ġfixed - window Ġbas elines . ĠOn Ġboth Ġdatasets , ĠCRA FT Ġoutper forms Ġall Ġbas elines Ġ( including Ġthe Ġother Ġonline Ġmodels ) Ġin Ġterms Ġof Ġaccuracy Ġand ĠF 1 . ĠFurthermore , Ġalthough Ġit Ġloses Ġon Ġprecision Ġ( to ĠCRA FT Ġ$ -$ ĠCE ) Ġand Ġrecall Ġ( to ĠCum ulative ĠBo W ) Ġindividually Ġon Ġthe ĠWikipedia Ġdata , ĠCRA FT Ġhas Ġthe Ġsuperior Ġbalance Ġbetween Ġthe Ġtwo , Ġhaving Ġboth Ġa Ġvisibly Ġhigher Ġprecision - rec all Ġcurve Ġand Ġlarger Ġarea Ġunder Ġthe Ġcurve Ġ( AU PR ) Ġthan Ġthe Ġbas elines Ġ( Figure ĠFIG REF 20 ). ĠThis Ġlatter Ġproperty Ġis Ġparticularly Ġuseful Ġin Ġa Ġpractical Ġsetting , Ġas Ġit Ġallows Ġmoderators Ġto Ġtune Ġmodel Ġperformance Ġto Ġsome Ġdesired Ġprecision Ġwithout Ġhaving Ġto Ġsacrifice Ġas Ġmuch Ġin Ġthe Ġway Ġof Ġrecall Ġ( or Ġvice Ġversa ) Ġcompared Ġto Ġthe Ġbas elines Ġand Ġpre - existing Ġsolutions . </s> Analysis </s> We Ġnow Ġexamine Ġthe Ġbehavior Ġof ĠCRA FT Ġin Ġgreater Ġdetail , Ġto Ġbetter Ġunderstand Ġits Ġbenefits Ġand Ġlimitations . ĠWe Ġspecifically Ġaddress Ġthe Ġfollowing Ġquestions : Ġ( 1 ) ĠHow Ġmuch Ġearly Ġwarning Ġdoes Ġthe Ġthe Ġmodel Ġprovide ? Ġ( 2 ) ĠDoes Ġthe Ġmodel Ġactually Ġlearn Ġan Ġorder - sensitive Ġrepresentation Ġof Ġconvers ational Ġcontext ? </s> Early Ġwarning , Ġbut Ġhow Ġearly ? ĠThe Ġrecent Ġinterest Ġin Ġforecasting Ġantis ocial Ġbehavior Ġhas Ġbeen Ġdriven Ġby Ġa Ġdesire Ġto Ġprovide Ġpre - empt ive , Ġaction able Ġwarning Ġto Ġmoderators . ĠBut Ġdoes Ġour Ġmodel Ġtrigger Ġearly Ġenough Ġfor Ġany Ġsuch Ġpractical Ġgoals ? </s> For Ġeach Ġpersonal Ġattack Ġcorrectly Ġforecast ed Ġby Ġour Ġmodel , Ġwe Ġcount Ġthe Ġnumber Ġof Ġcomments Ġelapsed Ġbetween Ġthe Ġtime Ġthe Ġmodel Ġis Ġfirst Ġtriggered Ġand Ġthe Ġattack . ĠFigure ĠFIG REF 22 Ġshows Ġthe Ġdistribution Ġof Ġthese Ġcounts : Ġon Ġaverage , Ġthe Ġmodel Ġwarns Ġof Ġan Ġattack Ġ3 Ġcomments Ġbefore Ġit Ġactually Ġhappens Ġ( 4 Ġcomments Ġfor ĠCM V ). ĠTo Ġfurther Ġevaluate Ġhow Ġmuch Ġtime Ġthis Ġearly Ġwarning Ġwould Ġgive Ġto Ġthe Ġmoderator , Ġwe Ġalso Ġconsider Ġthe Ġdifference Ġin Ġtim est amps Ġbetween Ġthe Ġcomment Ġwhere Ġthe Ġmodel Ġfirst Ġtriggers Ġand Ġthe Ġcomment Ġcontaining Ġthe Ġactual Ġattack . ĠOver Ġ50 % Ġof Ġconversations Ġget Ġat Ġleast Ġ3 Ġhours Ġof Ġadvance Ġwarning Ġ( 2 Ġhours Ġfor ĠCM V ). ĠMoreover , Ġ39 % Ġof Ġconversations Ġget Ġat Ġleast Ġ12 Ġhours Ġof Ġearly Ġwarning Ġbefore Ġthey Ġderail . </s> Does Ġorder Ġmatter ? ĠOne Ġmotivation Ġbehind Ġthe Ġdesign Ġof Ġour Ġmodel Ġwas Ġthe Ġintuition Ġthat Ġcomments Ġin Ġa Ġconversation Ġare Ġnot Ġindependent Ġevents ; Ġrather , Ġthe Ġorder Ġin Ġwhich Ġthey Ġappear Ġmatters Ġ( e . g ., Ġa Ġblunt Ġcomment Ġfollowed Ġby Ġa Ġpolite Ġone Ġfeels Ġintuitive ly Ġdifferent Ġfrom Ġa Ġpolite Ġcomment Ġfollowed Ġby Ġa Ġblunt Ġone ). ĠBy Ġdesign , ĠCRA FT Ġhas Ġthe Ġcapacity Ġto Ġlearn Ġan Ġorder - sensitive Ġrepresentation Ġof Ġconvers ational Ġcontext , Ġbut Ġhow Ġcan Ġwe Ġknow Ġthat Ġthis Ġcapacity Ġis Ġactually Ġused ? ĠIt Ġis Ġconceivable Ġthat Ġthe Ġmodel Ġis Ġsimply Ġcomputing Ġan Ġorder - ins ensitive ĠâĢ ľ bag - of - features âĢ Ŀ . ĠNeural Ġnetwork Ġmodels Ġare Ġnotorious Ġfor Ġtheir Ġlack Ġof Ġtransparency , Ġpre cluding Ġan Ġanalysis Ġof Ġhow Ġexactly ĠCRA FT Ġmodels Ġconvers ational Ġcontext . ĠNevertheless , Ġthrough Ġtwo Ġsimple Ġexplor atory Ġexperiments , Ġwe Ġseek Ġto Ġshow Ġthat Ġit Ġdoes Ġnot Ġcompletely Ġignore Ġcomment Ġorder . </s> The Ġfirst Ġexperiment Ġfor Ġtesting Ġwhether Ġthe Ġmodel Ġaccounts Ġfor Ġcomment Ġorder Ġis Ġa Ġprefix - sh uff ling Ġexperiment , Ġvisual ized Ġin ĠFigure ĠFIG REF 23 . ĠFor Ġeach Ġconversation Ġthat Ġthe Ġmodel Ġpredicts Ġwill Ġderail , Ġlet Ġ$ t $ Ġdenote Ġthe Ġindex Ġof Ġthe Ġtriggering Ġcomment , Ġi . e ., Ġthe Ġindex Ġwhere Ġthe Ġmodel Ġfirst Ġmade Ġa Ġderail ment Ġforecast . ĠWe Ġthen Ġconstruct Ġsynthetic Ġconversations Ġby Ġtaking Ġthe Ġfirst Ġ$ t - 1 $ Ġcomments Ġ( hen ce forth Ġreferred Ġto Ġas Ġthe Ġprefix ) Ġand Ġrandom izing Ġtheir Ġorder . ĠFinally , Ġwe Ġcount Ġhow Ġoften Ġthe Ġmodel Ġno Ġlonger Ġpredicts Ġderail ment Ġat Ġindex Ġ$ t $ Ġin Ġthe Ġsynthetic Ġconversations . ĠIf Ġthe Ġmodel Ġwere Ġignoring Ġcomment Ġorder , Ġits Ġprediction Ġshould Ġremain Ġunchanged Ġ( as Ġit Ġremains Ġfor Ġthe ĠCum ulative ĠBo W Ġbaseline ), Ġsince Ġthe Ġactual Ġcontent Ġof Ġthe Ġfirst Ġ$ t $ Ġcomments Ġhas Ġnot Ġchanged Ġ( and ĠCRA FT Ġinference Ġis Ġdeter ministic ). ĠWe Ġinstead Ġfind Ġthat Ġin Ġroughly Ġone Ġfifth Ġof Ġcases Ġ( 12 % Ġfor ĠCM V ) Ġthe Ġmodel Ġchanges Ġits Ġprediction Ġon Ġthe Ġsynthetic Ġconversations . ĠThis Ġsuggests Ġthat ĠCRA FT Ġlearns Ġan Ġorder - sensitive Ġrepresentation Ġof Ġcontext , Ġnot Ġa Ġmere ĠâĢ ľ bag - of - features âĢ Ŀ . </s> To Ġmore Ġconcrete ly Ġquantify Ġhow Ġmuch Ġthis Ġorder - sensitive Ġcontext Ġmodeling Ġhelps Ġwith Ġprediction , Ġwe Ġcan Ġactively Ġprevent Ġthe Ġmodel Ġfrom Ġlearning Ġand Ġexploiting Ġany Ġorder - related Ġdynamics . ĠWe Ġachieve Ġthis Ġthrough Ġanother Ġtype Ġof Ġshuff ling Ġexperiment , Ġwhere Ġwe Ġgo Ġback Ġeven Ġfurther Ġand Ġshuffle Ġthe Ġcomment Ġorder Ġin Ġthe Ġconversations Ġused Ġfor Ġpre - training , Ġfine - tun ing Ġand Ġtesting . ĠThis Ġprocedure Ġpreserves Ġthe Ġmodel 's Ġability Ġto Ġcapture Ġsignals Ġpresent Ġwithin Ġthe Ġindividual Ġcomments Ġprocessed Ġso Ġfar , Ġas Ġthe Ġutter ance Ġenc oder Ġis Ġunaffected , Ġbut Ġinhibits Ġit Ġfrom Ġcapturing Ġany Ġmeaningful Ġorder - sensitive Ġdynamics . ĠWe Ġfind Ġthat Ġthis Ġhurts Ġthe Ġmodel 's Ġperformance Ġ( 65 % Ġaccuracy Ġfor ĠWikipedia , Ġ59 . 5 % Ġfor ĠCM V ), Ġlowering Ġit Ġto Ġa Ġlevel Ġsimilar Ġto Ġthat Ġof Ġthe Ġversion Ġwhere Ġwe Ġcompletely Ġdisable Ġthe Ġcontext Ġenc oder . </s> T aken Ġtogether , Ġthese Ġexperiments Ġprovide Ġevidence Ġthat ĠCRA FT Ġuses Ġits Ġcapacity Ġto Ġmodel Ġconvers ational Ġcontext Ġin Ġan Ġorder - sensitive Ġfashion , Ġand Ġthat Ġit Ġmakes Ġeffective Ġuse Ġof Ġthe Ġdynamics Ġwithin . ĠAn Ġimportant Ġavenue Ġfor Ġfuture Ġwork Ġwould Ġbe Ġdeveloping Ġmore Ġtransparent Ġmodels Ġthat Ġcan Ġshed Ġlight Ġon Ġexactly Ġwhat Ġkinds Ġof Ġorder - related Ġfeatures Ġare Ġbeing Ġextracted Ġand Ġhow Ġthey Ġare Ġused Ġin Ġprediction . </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġwork , Ġwe Ġintroduced Ġa Ġmodel Ġfor Ġforecasting Ġconvers ational Ġevents Ġthat Ġprocesses Ġcomments Ġas Ġthey Ġhappen Ġand Ġtakes Ġthe Ġfull Ġconvers ational Ġcontext Ġinto Ġaccount Ġto Ġmake Ġan Ġupdated Ġprediction Ġat Ġeach Ġstep . ĠThis Ġmodel Ġfills Ġa Ġvoid Ġin Ġthe Ġexisting Ġliterature Ġon Ġconvers ational Ġforecasting , Ġsimultaneously Ġaddressing Ġthe Ġdual Ġchallenges Ġof Ġcapturing Ġinter - comment Ġdynamics Ġand Ġdealing Ġwith Ġan Ġunknown Ġhorizon . ĠWe Ġfind Ġthat Ġour Ġmodel Ġachieves Ġstate - of - the - art Ġperformance Ġon Ġthe Ġtask Ġof Ġforecasting Ġderail ment Ġin Ġtwo Ġdifferent Ġdatasets Ġthat Ġwe Ġrelease Ġpublicly . ĠWe Ġfurther Ġshow Ġthat Ġthe Ġresulting Ġsystem Ġcan Ġprovide Ġsubstantial Ġprior Ġnotice Ġof Ġderail ment , Ġopening Ġup Ġthe Ġpotential Ġfor Ġpreempt ive Ġinterventions Ġby Ġhuman Ġmoderators ĠB IB REF 65 . </s> While Ġwe Ġhave Ġfocused Ġspecifically Ġon Ġthe Ġtask Ġof Ġforecasting Ġderail ment , Ġwe Ġview Ġthis Ġwork Ġas Ġa Ġstep Ġtowards Ġa Ġmore Ġgeneral Ġmodel Ġfor Ġreal - time Ġforecasting Ġof Ġother Ġtypes Ġof Ġemer gent Ġproperties Ġof Ġconversations . ĠFollow - up Ġwork Ġcould Ġadapt Ġthe ĠCRA FT Ġarchitecture Ġto Ġaddress Ġother Ġforecasting Ġtasks Ġmentioned Ġin ĠSection ĠSEC REF 2 âĢĶ including Ġthose Ġfor Ġwhich Ġthe Ġoutcome Ġis Ġextr aneous Ġto Ġthe Ġconversation . ĠWe Ġexpect Ġdifferent Ġtasks Ġto Ġbe Ġinformed Ġby Ġdifferent Ġtypes Ġof Ġinter - comment Ġdynamics , Ġand Ġfurther Ġarchitecture Ġextensions Ġcould Ġadd Ġadditional Ġsupervised Ġfine - tun ing Ġin Ġorder Ġto Ġdirect Ġit Ġto Ġfocus Ġon Ġspecific Ġdynamics Ġthat Ġmight Ġbe Ġrelevant Ġto Ġthe Ġtask Ġ( e . g ., Ġexchange Ġof Ġideas Ġbetween Ġinter loc ut ors Ġor Ġst onew alling ). </s> With Ġrespect Ġto Ġforecasting Ġderail ment , Ġthere Ġremain Ġopen Ġquestions Ġregarding Ġwhat Ġhuman Ġmoderators Ġactually Ġdesire Ġfrom Ġan Ġearly - warning Ġsystem , Ġwhich Ġwould Ġaffect Ġthe Ġdesign Ġof Ġa Ġpractical Ġsystem Ġbased Ġon Ġthis Ġwork . ĠFor Ġinstance , Ġhow Ġearly Ġdoes Ġa Ġwarning Ġneed Ġto Ġbe Ġin Ġorder Ġfor Ġmoderators Ġto Ġfind Ġit Ġuseful ? ĠWhat Ġis Ġthe Ġoptimal Ġbalance Ġbetween Ġprecision , Ġrecall , Ġand Ġfalse Ġpositive Ġrate Ġat Ġwhich Ġsuch Ġa Ġsystem Ġis Ġtruly Ġimproving Ġmoderator Ġproductivity Ġrather Ġthan Ġwasting Ġtheir Ġtime Ġthrough Ġfalse Ġpositives ? ĠWhat Ġare Ġthe Ġethical Ġimplications Ġof Ġsuch Ġa Ġsystem ? ĠFollow - up Ġwork Ġcould Ġrun Ġa Ġuser Ġstudy Ġof Ġa Ġprototype Ġsystem Ġwith Ġactual Ġmoderators Ġto Ġaddress Ġthese Ġquestions . </s> A Ġpractical Ġlimitation Ġof Ġthe Ġcurrent Ġanalysis Ġis Ġthat Ġit Ġrelies Ġon Ġbalanced Ġdatasets , Ġwhile Ġderail ment Ġis Ġa Ġrelatively Ġrare Ġevent Ġfor Ġwhich Ġa Ġmore Ġrestrictive Ġtrigger Ġthreshold Ġwould Ġbe Ġappropriate . ĠWhile Ġour Ġanalysis Ġof Ġthe Ġprecision - rec all Ġcurve Ġsuggests Ġthe Ġsystem Ġis Ġrobust Ġacross Ġmultiple Ġthresholds Ġ($ AU PR = 0 . 7 $ ), Ġadditional Ġwork Ġis Ġneeded Ġto Ġestablish Ġwhether Ġthe Ġrecall Ġtrade off Ġwould Ġbe Ġacceptable Ġin Ġpractice . </s> Finally , Ġone Ġmajor Ġlimitation Ġof Ġthe Ġpresent Ġwork Ġis Ġthat Ġit Ġassigns Ġa Ġsingle Ġlabel Ġto Ġeach Ġconversation : Ġdoes Ġit Ġderail Ġor Ġnot ? ĠIn Ġreality , Ġderail ment Ġneed Ġnot Ġspell Ġthe Ġend Ġof Ġa Ġconversation ; Ġit Ġis Ġpossible Ġthat Ġa Ġconversation Ġcould Ġget Ġback Ġon Ġtrack , Ġsuffer Ġa Ġrepeat Ġoccurrence Ġof Ġantis ocial Ġbehavior , Ġor Ġany Ġnumber Ġof Ġother Ġtraject ories . ĠIt Ġwould Ġbe Ġexciting Ġto Ġconsider Ġfiner - gr ained Ġforecasting Ġof Ġconvers ational Ġtraject ories , Ġaccounting Ġfor Ġthe Ġnatural âĢĶ and Ġsometimes Ġchaotic âĢĶ eb b - and - flow Ġof Ġhuman Ġinteractions . </s> Acknowled gements . ĠWe Ġthank ĠCaleb ĠCh iam , ĠLi ye ĠFu , ĠL illian ĠLee , ĠAlexand ru ĠNic ul esc u - M iz il , ĠAndrew ĠWang Ġand ĠJust ine ĠZhang Ġfor Ġinsightful Ġconversations Ġ( with Ġunknown Ġhorizon ), ĠAd ity a ĠJ ha Ġfor Ġhis Ġgreat Ġhelp Ġwith Ġimplementing Ġand Ġrunning Ġthe Ġcrowd - s ourcing Ġtasks , ĠThomas ĠDavidson Ġand ĠClaire ĠLiang Ġfor Ġexplor atory Ġdata Ġannotation , Ġas Ġwell Ġas Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġhelpful Ġcomments . ĠThis Ġwork Ġis Ġsupported Ġin Ġpart Ġby Ġthe ĠNS F ĠCARE ER Ġaward ĠI IS - 17 506 15 Ġand Ġby Ġthe ĠNS F ĠGrant ĠS ES - 174 14 41 .