context inference
EXTRACTIVE
In order to understand if context inference is useful to intervention prediction, we ablate the attention components and experiment with the vanilla hierarchical LSTM model. Row 3 of Table TABREF17 shows the macro averaged result from this experiment. The UPA and PPA attention models better the vanilla hLSTM by 5% and 2% on average in INLINEFORM0 respectively. Recall that the vanilla hLSTM already has access to a context consisting of all posts (from INLINEFORM1 through INLINEFORM2 ). In contrast, the UPA and PPA models selectively infers a context for INLINEFORM3 and INLINEFORM4 posts, respectively, and use it to predict intervention. The improved performance of our attention models that actively select their optimal context, over a model with the complete thread as context, hLSTM, shows that the context inference improves intervention prediction over using the default full context.
1905.10851
Introduction </s> Mass ive ĠOpen ĠOnline ĠCour ses Ġ( MO OC s ) Ġhave Ġstri ved Ġto Ġbridge Ġthe Ġsocial Ġgap Ġin Ġhigher Ġeducation Ġby Ġbringing Ġquality Ġeducation Ġfrom Ġrep uted Ġuniversities Ġto Ġstudents Ġat Ġlarge . ĠSuch Ġmassive Ġscaling Ġthrough Ġonline Ġclassrooms , Ġhowever , Ġdisrupt Ġco - l ocated , Ġsynchron ous Ġtwo - way Ġcommunication Ġbetween Ġthe Ġstudents Ġand Ġthe Ġinstructor . </s> MO OC Ġplatforms Ġprovide Ġdiscussion Ġforums Ġfor Ġstudents Ġto Ġtalk Ġto Ġtheir Ġclassmates Ġabout Ġthe Ġlectures , Ġhom eworks , Ġquizz es Ġand Ġprovide Ġa Ġvenue Ġto Ġsocial ise . ĠInstruct ors Ġ( defined Ġhere Ġas Ġthe Ġcourse Ġinstructors , Ġtheir Ġteaching Ġassistants Ġand Ġthe ĠMO OC Ġplatform 's Ġtechnical Ġstaff ) Ġmonitor Ġthe Ġdiscussion Ġforum Ġto Ġpost Ġ( reply Ġto Ġtheir Ġmessage ) Ġin Ġdiscussion Ġthreads Ġamong Ġstudents . ĠWe Ġrefer Ġto Ġthis Ġposting Ġas Ġintervention , Ġfollowing Ġprior Ġwork ĠB IB REF 0 Ġ. ĠHowever , Ġdue Ġto Ġlarge Ġstudent Ġenrol ment , Ġthe Ġstudent âĢĵ in struct or Ġratio Ġin ĠMO OC s Ġis Ġvery Ġhigh ĠTherefore , Ġinstructors Ġare Ġnot Ġable Ġto Ġmonitor Ġand Ġparticipate Ġin Ġall Ġstudent Ġdiscussions . ĠTo Ġaddress Ġthis Ġproblem , Ġa Ġnumber Ġof Ġworks Ġhave Ġproposed Ġsystems Ġe . g ., ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġto Ġaid Ġinstructors Ġto Ġselectively Ġintervene Ġon Ġstudent Ġdiscussions Ġwhere Ġthey Ġare Ġneeded Ġthe Ġmost . </s> In Ġthis Ġpaper , Ġwe Ġimprove Ġthe Ġstate - of - the - art Ġfor Ġinstructor Ġintervention Ġin ĠMO OC Ġforums . ĠWe Ġpropose Ġthe Ġfirst Ġneural Ġmodels Ġfor Ġthis Ġprediction Ġproblem . ĠWe Ġshow Ġthat Ġmodelling Ġthe Ġthread Ġstructure Ġand Ġthe Ġsequence Ġof Ġposts Ġexplicitly Ġimproves Ġperformance . ĠInstruct ors Ġin Ġdifferent ĠMO OC s Ġfrom Ġdifferent Ġsubject Ġareas Ġintervene Ġdifferently . ĠFor Ġexample , Ġon Ġa ĠScience , ĠTechnology , ĠEngineering Ġand ĠMathematics Ġ( STEM ) ĠMO OC , Ġinstructors Ġmay Ġoften Ġintervene Ġearly Ġas Ġpossible Ġto Ġresolve Ġmisunderstanding Ġof Ġthe Ġsubject Ġmaterial Ġand Ġprevent Ġconfusion . ĠHowever , Ġin Ġa ĠHuman ities ĠMO OC , Ġinstructors Ġallow Ġfor Ġthe Ġstudents Ġto Ġexplore Ġopen - ended Ġdiscussions Ġand Ġdebate Ġamong Ġthemselves . ĠSuch Ġinstructors Ġmay Ġprefer Ġto Ġintervene Ġlater Ġin Ġthe Ġdiscussion Ġto Ġencourage Ġfurther Ġdiscussion Ġor Ġresolve Ġconflicts Ġamong Ġstudents . ĠWe Ġtherefore Ġpropose Ġattention Ġmodels Ġto Ġinfer Ġthe Ġlatent Ġcontext , Ġi . e ., Ġthe Ġseries Ġof Ġposts Ġthat Ġtrigger Ġan Ġintervention . ĠEarlier Ġstudies Ġon ĠMO OC Ġforum Ġintervention Ġeither Ġmodel Ġthe Ġentire Ġcontext Ġor Ġrequire Ġthe Ġcontext Ġsize Ġto Ġbe Ġspecified Ġexplicitly . </s> Problem ĠStatement </s> A Ġthread ĠIN LINE FORM 0 Ġconsists Ġof Ġa Ġseries Ġof Ġposts ĠIN LINE FORM 1 Ġthrough ĠIN LINE FORM 2 Ġwhere ĠIN LINE FORM 3 Ġis Ġan Ġinstructor 's Ġpost Ġwhen ĠIN LINE FORM 4 Ġis Ġintervened , Ġif Ġapplicable . ĠIN LINE FORM 5 Ġis Ġconsidered Ġintervened Ġif Ġan Ġinstructor Ġhad Ġposted Ġat Ġleast Ġonce . ĠThe Ġproblem Ġof Ġpredicting Ġinstructor Ġintervention Ġis Ġcast Ġas Ġa Ġbinary Ġclassification Ġproblem . ĠInter ven ed Ġthreads Ġare Ġpredicted Ġas Ġ1 Ġgiven Ġwhile Ġnon - inter ven ed Ġthreads Ġare Ġpredicted Ġas Ġ0 Ġgiven Ġposts ĠIN LINE FORM 6 Ġthrough ĠIN LINE FORM 7 Ġ. </s> The Ġprimary Ġproblem Ġleads Ġto Ġa Ġsecondary Ġproblem Ġof Ġinfer ring Ġthe Ġappropriate Ġamount Ġof Ġcontext Ġto Ġintervene . ĠWe Ġdefine Ġa Ġcontext ĠIN LINE FORM 0 Ġof Ġa Ġpost ĠIN LINE FORM 1 Ġas Ġa Ġseries Ġof Ġlinear Ġcontiguous Ġposts ĠIN LINE FORM 2 Ġthrough ĠIN LINE FORM 3 Ġwhere ĠIN LINE FORM 4 Ġ. ĠThe Ġproblem Ġof Ġinfer ring Ġcontext Ġis Ġto Ġidentify Ġcontext ĠIN LINE FORM 5 Ġfrom Ġa Ġset Ġof Ġcandidate Ġcontexts ĠIN LINE FORM 6 Ġ. </s> Mod elling ĠContext Ġin ĠForums </s> Context Ġhas Ġbeen Ġused Ġand Ġmod elled Ġin Ġvarious Ġways Ġfor Ġdifferent Ġproblems Ġin Ġdiscussion Ġforums . ĠIn Ġa Ġwork Ġon Ġa Ġclosely Ġrelated Ġproblem Ġof Ġforum Ġthread Ġretrieval ĠB IB REF 2 Ġmodels Ġcontext Ġusing Ġinter - post Ġdiscourse Ġe . g ., ĠQuestion - Answer . ĠB IB REF 3 Ġmodels Ġthe Ġstructural Ġdependencies Ġand Ġrelationships Ġbetween Ġforum Ġposts Ġusing Ġa Ġconditional Ġrandom Ġfield Ġin Ġtheir Ġproblem Ġto Ġinfer Ġthe Ġreply Ġstructure . ĠUnlike ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġcan Ġbe Ġused Ġto Ġmodel Ġany Ġstructural Ġdependency Ġand Ġis , Ġtherefore , Ġmore Ġgeneral . ĠIn Ġthis Ġpaper , Ġwe Ġseek Ġto Ġinfer Ġgeneral Ġdependencies Ġbetween Ġa Ġreply Ġand Ġits Ġprevious Ġcontext Ġwhereas ĠB IB REF 3 Ġinference Ġis Ġlimited Ġto Ġpairs Ġof Ġposts . ĠMore Ġrecently ĠB IB REF 4 Ġproposed Ġa Ġcontext Ġbased Ġmodel Ġwhich Ġfactor ises Ġattention Ġover Ġthreads Ġof Ġdifferent Ġlengths . ĠDifferent ly , Ġwe Ġdo Ġnot Ġmodel Ġlength Ġbut Ġthe Ġcontext Ġbefore Ġa Ġpost . ĠHowever , Ġour Ġattention Ġmodels Ġcater Ġto Ġthreads Ġof Ġall Ġlengths . </s> BI B REF 5 Ġproposed Ġgraph Ġstructured ĠL ST M Ġto Ġmodel Ġthe Ġexplicit Ġreply Ġstructure Ġin ĠReddit Ġforums . ĠOur Ġwork Ġdoes Ġnot Ġassume Ġaccess Ġto Ġsuch Ġa Ġreply Ġstructure Ġbecause Ġ1 ) ĠCour ser a Ġforums Ġdo Ġnot Ġprovide Ġone Ġand Ġ2 ) Ġforum Ġparticipants Ġoften Ġerr Ġby Ġposting Ġtheir Ġreply Ġto Ġa Ġdifferent Ġpost Ġthan Ġthat Ġthey Ġintended . ĠAt Ġthe Ġother Ġend Ġof Ġthe Ġspectrum Ġare Ġdocument Ġclassification Ġmodels Ġthat Ġdo Ġnot Ġassume Ġstructure Ġin Ġthe Ġdocument Ġlayout Ġbut Ġtry Ġto Ġinfer Ġinherent Ġstructure Ġin Ġthe Ġnatural Ġlanguage , Ġviz , Ġwords , Ġsentences , Ġparagraphs Ġand Ġdocuments . ĠHier arch ical Ġattention ĠB IB REF 6 Ġis Ġa Ġwell Ġknow Ġrecent Ġwork Ġthat Ġclass ifies Ġdocuments Ġusing Ġa Ġmulti - level ĠL ST Ms Ġwith Ġattention Ġmechanism Ġto Ġselect Ġimportant Ġunits Ġat Ġeach Ġhierarchical Ġlevel . ĠDifferent ly , Ġwe Ġpropose Ġa Ġhierarchical Ġmodel Ġthat Ġenc odes Ġlayout Ġhierarchy Ġbetween Ġa Ġpost Ġand Ġa Ġthread Ġbut Ġalso Ġinf ers Ġreply Ġstructure Ġusing Ġa Ġattention Ġmechanism Ġsince Ġthe Ġlayout Ġdoes Ġnot Ġreliably Ġencode Ġit . </s> Instruct or ĠIntervention Ġin ĠMO OC Ġforums </s> The Ġproblem Ġof Ġpredicting Ġinstructor Ġintervention Ġin ĠMO OC s Ġwas Ġproposed Ġby ĠB IB REF 0 Ġ. ĠLater ĠB IB REF 7 Ġevaluated Ġbaseline Ġmodels Ġby ĠB IB REF 0 Ġover Ġa Ġlarger Ġcorpus Ġand Ġfound Ġthe Ġresults Ġto Ġvary Ġwidely Ġacross ĠMO OC s . ĠSince Ġthen Ġsubsequent Ġworks Ġhave Ġused Ġsimilar Ġdiverse Ġevaluations Ġon Ġthe Ġsame Ġprediction Ġproblem ĠB IB REF 1 Ġ, ĠB IB REF 8 Ġ. ĠB IB REF 1 Ġproposed Ġmodels Ġwith Ġdiscourse Ġfeatures Ġto Ġenable Ġbetter Ġprediction Ġover Ġunseen ĠMO OC s . ĠB IB REF 8 Ġrecently Ġshowed Ġinterventions Ġon ĠCour ser a Ġforums Ġto Ġbe Ġbiased Ġby Ġthe Ġposition Ġat Ġwhich Ġa Ġthread Ġappears Ġto Ġan Ġinstructor Ġviewing Ġthe Ġforum Ġinterface Ġand Ġproposed Ġmethods Ġfor Ġdeb i ased Ġprediction . </s> While Ġall Ġworks Ġsince ĠB IB REF 0 Ġaddress Ġkey Ġlimitations Ġin Ġthis Ġline Ġof Ġresearch , Ġthey Ġhave Ġnot Ġinvestigated Ġthe Ġrole Ġof Ġstructure Ġand Ġsequence Ġin Ġthe Ġthreaded Ġdiscussion Ġin Ġpredicting Ġinstructor Ġinterventions . ĠB IB REF 0 Ġproposed Ġprob abil istic Ġgraphical Ġmodels Ġto Ġmodel Ġstructure Ġand Ġsequence . ĠThey Ġinferred Ġvocabulary Ġdependent Ġlatent Ġpost Ġcategories Ġto Ġmodel Ġthe Ġthread Ġsequence Ġand Ġinfer Ġstates Ġthat Ġtriggered Ġintervention . ĠTheir Ġmodel , Ġhowever , Ġrequires Ġa Ġhyper param eter Ġfor Ġthe Ġnumber Ġof Ġlatent Ġstates . ĠIt Ġis Ġlikely Ġthat Ġtheir Ġempir ically Ġreported Ġsetting Ġwill Ġnot Ġgeneral ise Ġdue Ġto Ġtheir Ġweak Ġevaluation ĠB IB REF 7 Ġ. ĠIn Ġthis Ġpaper , Ġwe Ġpropose Ġmodels Ġto Ġinfer Ġthe Ġcontext Ġthat Ġtriggers Ġinstructor Ġintervention Ġthat Ġdoes Ġnot Ġrequire Ġcontext Ġlengths Ġto Ġbe Ġset Ġap rior i . ĠAll Ġour Ġproposed Ġmodels Ġgeneral ise Ġover Ġmodelling Ġassumptions Ġmade Ġby ĠB IB REF 0 Ġ. </s> For Ġthe Ġpurpose Ġof Ġcomparison Ġagainst Ġa Ġstate - of - the - art Ġand Ġcompeting Ġbas elines Ġwe Ġchoose ĠB IB REF 7 Ġsince ĠB IB REF 0 Ġ' s Ġsystem Ġand Ġdata Ġare Ġnot Ġavailable Ġfor Ġreplication . </s> Data Ġand ĠPre processing </s> We Ġevaluate Ġour Ġproposed Ġmodels Ġover Ġa Ġcorpus Ġof Ġ12 ĠMO OC Ġiterations Ġ( offer ings ) Ġon ĠCour ser a . org ĠIn Ġpartnership Ġwith ĠCour ser a Ġand Ġin Ġline Ġwith Ġits ĠTerms Ġof ĠService , Ġwe Ġobtained Ġthe Ġdata Ġfor Ġuse Ġin Ġour Ġacademic Ġresearch . ĠFollowing Ġprior Ġwork ĠB IB REF 7 Ġwe Ġevaluate Ġover Ġa Ġdiverse Ġdataset Ġto Ġrepresent ĠMO OC s Ġof Ġvarying Ġsizes , Ġinstructor Ġstyles , Ġinstructor Ġteam Ġsizes Ġand Ġnumber Ġof Ġthreads Ġintervened . ĠWe Ġonly Ġinclude Ġthreads Ġfrom Ġsub - forums Ġon ĠLect ure , ĠHom ework , ĠQu iz Ġand ĠExam . ĠWe Ġalso Ġnormal ise Ġand Ġlabel Ġsub - forums Ġwith Ġother Ġnon - standard Ġnames Ġ( e . g ., ĠAss ign ments Ġinstead Ġof ĠHom ework ) Ġinto Ġof Ġthe Ġfour Ġsaid Ġsub - forums . ĠThread s Ġon Ġgeneral Ġdiscussion , Ġmeet Ġand Ġgreet Ġand Ġother Ġcustom Ġsub - forums Ġfor Ġsocial Ġch itch at Ġare Ġomitted Ġas Ġour Ġfocus Ġis Ġto Ġaid Ġinstructors Ġon Ġintervening Ġon Ġdiscussion Ġon Ġthe Ġsubject Ġmatter . ĠWe Ġalso Ġexclude Ġannouncement Ġthreads Ġand Ġother Ġthreads Ġstarted Ġby Ġinstructors Ġsince Ġthey Ġare Ġnot Ġinterventions . ĠWe Ġpre process Ġeach Ġthread Ġby Ġreplacing ĠURLs , Ġequations Ġand Ġother Ġmathematical Ġform ul ae Ġand Ġreferences Ġto Ġtim est amps Ġin Ġlecture Ġvideos Ġby Ġtokens ĠIN LINE FORM 0 ĠURL ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 ĠM ATH ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 ĠTIM ERE F ĠIN LINE FORM 5 Ġrespectively . ĠWe Ġalso Ġtrunc ate Ġintervened Ġthreads Ġto Ġonly Ġinclude Ġposts Ġbefore Ġthe Ġfirst Ġinstructor Ġpost Ġsince Ġthe Ġinstructor 's Ġand Ġsubsequent Ġposts Ġwill Ġbias Ġthe Ġprediction Ġdue Ġto Ġthe Ġinstructor 's Ġpost . </s> Model </s> The Ġkey Ġinnovation Ġof Ġour Ġwork Ġis Ġto Ġdecom pose Ġthe Ġintervention Ġprediction Ġproblem Ġinto Ġa Ġtwo - stage Ġmodel Ġthat Ġfirst Ġexplicitly Ġtries Ġto Ġdiscover Ġthe Ġproper Ġcontext Ġto Ġwhich Ġa Ġpotential Ġintervention Ġcould Ġbe Ġrep lying Ġto , Ġand Ġthen , Ġpredict Ġthe Ġintervention Ġstatus . ĠThis Ġmodel Ġimplicitly Ġassess es Ġthe Ġimportance Ġ( or Ġurgency ) Ġof Ġthe Ġexisting Ġthread 's Ġcontext Ġto Ġdecide Ġwhether Ġan Ġintervention Ġis Ġnecessary . ĠFor Ġexample Ġin ĠFigure ĠSEC REF 1 Ġ, Ġprior Ġto Ġthe Ġinstructor 's Ġintervention , Ġthe Ġultimate Ġpost Ġ( Post Ġ# 6 ) Ġby ĠStudent Ġ2 Ġalready Ġacknowledged Ġthe ĠOP 's Ġgratitude Ġfor Ġhis Ġanswer . ĠIn Ġthis Ġregard , Ġthe Ġinstructor Ġmay Ġhave Ġdecided Ġto Ġuse Ġthis Ġpoint Ġto Ġsummarize Ġthe Ġentire Ġthread Ġto Ġconsolidate Ġall Ġthe Ġpertinent Ġpositions . ĠHere , Ġwe Ġmight Ġassume Ġthat Ġthe Ġinstructor 's Ġreply Ġtakes Ġthe Ġentire Ġthread Ġ( Posts Ġ# 1 âĢĵ 6 ) Ġas Ġthe Ġcontext Ġfor Ġher Ġreply . </s> This Ġsub problem Ġof Ġinfer ring Ġthe Ġcontext Ġscope Ġis Ġwhere Ġour Ġinnovation Ġcenters Ġon . ĠTo Ġbe Ġclear , Ġin Ġorder Ġto Ġmake Ġthe Ġprediction Ġthat Ġa Ġinstruction Ġintervention Ġis Ġnow Ġnecessary Ġon Ġa Ġthread , Ġthe Ġinstructor 's Ġreply Ġis Ġnot Ġyet Ġavailable ĠâĢĶ Ġthe Ġmodel Ġpredicts Ġwhether Ġa Ġreply Ġis Ġnecessary ĠâĢĶ Ġso Ġin Ġthe Ġexample , Ġonly ĠPosts Ġ# 1 âĢĵ 6 Ġare Ġavailable Ġin Ġthe Ġproblem Ġsetting . ĠTo Ġinfer Ġthe Ġcontext , Ġwe Ġhave Ġto Ġdecide Ġwhich Ġsubsequ ence Ġof Ġposts Ġare Ġthe Ġmost Ġplausible Ġmotivation Ġfor Ġan Ġintervention . </s> Recent Ġwork Ġin Ġdeep Ġneural Ġmodeling Ġhas Ġused Ġan Ġattention Ġmechanism Ġas Ġa Ġfocusing Ġquery Ġto Ġhighlight Ġspecific Ġitems Ġwithin Ġthe Ġinput Ġhistory Ġthat Ġsignificantly Ġinfluence Ġthe Ġcurrent Ġdecision Ġpoint . ĠOur Ġwork Ġemploys Ġthis Ġmechanism ĠâĢĵ Ġbut Ġwith Ġa Ġtwist : Ġdue Ġto Ġthe Ġfact Ġthat Ġthe Ġactual Ġinstructor Ġintervention Ġis Ġnot Ġ( yet ) Ġavailable Ġat Ġthe Ġdecision Ġtiming , Ġwe Ġcannot Ġuse Ġany Ġactual Ġintervention Ġto Ġdecide Ġthe Ġcontext . ĠTo Ġemploy Ġattention , Ġwe Ġmust Ġthen Ġemploy Ġa Ġsurrogate Ġtext Ġas Ġthe Ġquery Ġto Ġtrain Ġour Ġprediction Ġmodel . ĠOur Ġmodel Ġvariants Ġmodel Ġassess Ġthe Ġsuit ability Ġof Ġsuch Ġsurrogate Ġtexts Ġfor Ġthe Ġattention Ġmechanism Ġbasis . </s> Cong ru ent Ġwith Ġthe Ġrepresentation Ġof Ġthe Ġinput Ġforums , Ġin Ġall Ġour Ġproposed Ġmodels , Ġwe Ġencode Ġthe Ġdiscussion Ġthread Ġhierarch ically . ĠWe Ġfirst Ġbuild Ġrepresentations Ġfor Ġeach Ġpost Ġby Ġpassing Ġpre - trained Ġword Ġvector Ġrepresentations Ġfrom ĠGl o Ve ĠB IB REF 9 Ġfor Ġeach Ġword Ġthrough Ġan ĠL ST M ĠB IB REF 10 Ġ, ĠIN LINE FORM 0 Ġ. ĠWe Ġuse Ġthe Ġlast Ġlayer Ġoutput Ġof Ġthe ĠL ST M Ġas Ġa Ġrepresentation Ġof Ġthe Ġpost . ĠWe Ġrefer Ġthis Ġas Ġthe Ġpost Ġvector ĠIN LINE FORM 1 Ġ. </s> Then Ġeach Ġpost ĠIN LINE FORM 0 Ġis Ġpassed Ġthrough Ġanother ĠL ST M , ĠIN LINE FORM 1 Ġ, Ġwhose Ġlast Ġlayer Ġoutput Ġforms Ġthe Ġencoding Ġof Ġthe Ġentire Ġthread . ĠHidden Ġunit Ġoutputs Ġof ĠIN LINE FORM 2 Ġrepresent Ġthe Ġcontexts ĠIN LINE FORM 3 Ġ; Ġthat Ġis , Ġsnapshots Ġof Ġthe Ġthreads Ġafter Ġeach Ġpost , Ġas Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ. </s> The ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġtogether Ġconstitute Ġthe Ġhierarchical ĠL ST M Ġ( h L ST M ) Ġmodel . ĠThis Ġgeneral Ġh L ST M Ġmodel Ġserves Ġas Ġthe Ġbasis Ġfor Ġour Ġmodel Ġexploration Ġin Ġthe Ġrest Ġof Ġthis Ġsection . </s> Context ual ĠAttention ĠModels </s> When Ġthey Ġintervene , Ġinstructors Ġeither Ġpay Ġattention Ġto Ġa Ġspecific Ġpost Ġor Ġa Ġseries Ġof Ġposts , Ġwhich Ġtrigger Ġtheir Ġreply . ĠHowever , Ġinstructors Ġrarely Ġexplicitly Ġindicate Ġto Ġwhich Ġpost ( s ) Ġtheir Ġintervention Ġis Ġin Ġrelation Ġto . ĠThis Ġis Ġthe Ġcase Ġin Ġour Ġcorpus , Ġparty Ġdue Ġto ĠCour ser a 's Ġuser Ġinterface Ġwhich Ġonly Ġallows Ġfor Ġsingle Ġlevel Ġcomments Ġ( see ĠFigure ĠFIG REF 2 Ġ). ĠBased Ġsolely Ġon Ġthe Ġbinary , Ġthread - level Ġintervention Ġsignal , Ġour Ġsecondary Ġobjective Ġseeks Ġto Ġinfer Ġthe Ġappropriate Ġcontext ĠâĢĵ Ġrepresented Ġby Ġa Ġsequence Ġof Ġposts ĠâĢĵ Ġas Ġthe Ġbasis Ġfor Ġthe Ġintervention . </s> We Ġonly Ġconsider Ġlinear Ġcontiguous Ġseries Ġof Ġposts Ġstarting Ġwith Ġthe Ġthread 's Ġoriginal Ġpost Ġto Ġconstitute Ġto Ġa Ġcontext ; Ġe . g ., ĠIN LINE FORM 0 Ġ. ĠThis Ġis Ġa Ġreasonable Ġas ĠMO OC Ġforum Ġposts Ġalways Ġreply Ġto Ġthe Ġoriginal Ġpost Ġor Ġto Ġa Ġsubsequent Ġpost , Ġwhich Ġin Ġturn Ġreplies Ġto Ġthe Ġoriginal Ġpost . ĠThis Ġis Ġin Ġcontrast Ġto Ġforums Ġsuch Ġas ĠReddit Ġthat Ġhave Ġa Ġtree Ġor Ġgraph - like Ġstructure Ġthat Ġrequire Ġforum Ġstructure Ġto Ġbe Ġmod elled Ġexplicitly , Ġsuch Ġas Ġin ĠB IB REF 5 Ġ. </s> We Ġpropose Ġthree Ġneural Ġattention ĠB IB REF 11 Ġvariants Ġbased Ġon Ġhow Ġan Ġinstructor Ġmight Ġattend Ġand Ġreply Ġto Ġa Ġcontext Ġin Ġa Ġthread : Ġthe Ġultimate , Ġpen ultimate Ġand Ġany Ġpost Ġattention Ġmodels . ĠWe Ġreview Ġeach Ġof Ġthese Ġin Ġturn . </s> Ultimate ĠPost ĠAttention Ġ( U PA ) ĠModel . ĠIn Ġthis Ġmodel Ġwe Ġattend Ġto Ġthe Ġcontext Ġrepresented Ġby Ġhidden Ġstate Ġof Ġthe ĠIN LINE FORM 0 Ġ. ĠWe Ġuse Ġthe Ġpost Ġprior Ġto Ġthe Ġinstructor 's Ġreply Ġas Ġa Ġquery Ġover Ġthe Ġcontexts ĠIN LINE FORM 1 Ġto Ġcompute Ġattention Ġweights ĠIN LINE FORM 2 Ġ, Ġwhich Ġare Ġthen Ġused Ġto Ġcompute Ġthe Ġattended Ġcontext Ġrepresentation ĠIN LINE FORM 3 Ġ( rec all Ġagain Ġthat Ġthe Ġintervention Ġtext Ġitself Ġis Ġnot Ġavailable Ġfor Ġthis Ġpurpose ). ĠThis Ġattention Ġformulation Ġmakes Ġan Ġequival ence Ġbetween Ġthe Ġfinal ĠIN LINE FORM 4 Ġpost Ġand Ġthe Ġprospective Ġintervention , Ġusing ĠPost ĠIN LINE FORM 5 Ġas Ġthe Ġquery Ġfor Ġfinding Ġthe Ġappropriate Ġcontext ĠIN LINE FORM 6 Ġ, Ġinclusive Ġof Ġitself ĠIN LINE FORM 7 Ġ. ĠSaid Ġin Ġanother Ġway , ĠU PA Ġuses Ġthe Ġmost Ġrecent Ġcontent Ġin Ġthe Ġthread Ġas Ġthe Ġattention al Ġquery Ġfor Ġcontext . </s> For Ġexample , Ġif Ġpost ĠIN LINE FORM 0 Ġis Ġthe Ġinstructor 's Ġreply , Ġpost ĠIN LINE FORM 1 Ġwill Ġquery Ġover Ġthe Ġcontexts ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠThe Ġmodel Ġschematic Ġis Ġshown Ġin ĠFigure ĠFIG REF 12 Ġ. </s> The Ġattended Ġcontext Ġrepresentations Ġare Ġcomputed Ġas : ĠDIS PLAY FORM 0 </s> The ĠIN LINE FORM 0 Ġrepresentation Ġis Ġthen Ġpassed Ġthrough Ġa Ġfully Ġconnected Ġsoft max Ġlayer Ġto Ġyield Ġthe Ġbinary Ġprediction . </s> Pen ultimate ĠPost ĠAttention Ġ( P PA ) ĠModel . ĠWhile Ġthe ĠU PA Ġmodel Ġuses Ġthe Ġmost Ġrecent Ġtext Ġand Ġmakes Ġthe Ġultimate Ġpost Ġitself Ġavailable Ġas Ġpotential Ġcontext , Ġour Ġthe Ġultimate Ġpost Ġmay Ġbe Ġbetter Ġmodeled Ġas Ġhaving Ġany Ġof Ġits Ġprior Ġposts Ġas Ġpotential Ġcontext . ĠPen ultimate ĠPost ĠAttention Ġ( P PA ) Ġvariant Ġdoes Ġthis . ĠThe Ġschematic Ġand Ġthe Ġequations Ġfor Ġthe ĠP PA Ġmodel Ġare Ġobtained Ġby Ġsum ming Ġover Ġcontexts ĠIN LINE FORM 0 Ġin ĠEqu ation ĠEQ REF 10 Ġand ĠFigure ĠFIG REF 12 Ġ. ĠWhile Ġwe Ġcould Ġproperly Ġmodel Ġsuch Ġa Ġcontext Ġinference Ġdecision Ġwith Ġany Ġpost ĠIN LINE FORM 1 Ġand Ġprospective Ġcontexts ĠIN LINE FORM 2 Ġ( where ĠIN LINE FORM 3 Ġis Ġa Ġrandom Ġpost ), Ġit Ġmakes Ġsense Ġto Ġuse Ġthe Ġpen ultimate Ġpost , Ġas Ġwe Ġcan Ġmake Ġthe Ġmost Ġinformation Ġavailable Ġto Ġthe Ġmodel Ġfor Ġthe Ġcontext Ġinference . </s> The Ġattended Ġcontext Ġrepresentations Ġare Ġcomputed Ġas : ĠDIS PLAY FORM 0 </s> Any ĠPost ĠAttention Ġ( AP A ) ĠModel . ĠAP A Ġfurther Ġrelax es Ġboth ĠU PA Ġand ĠP PA , Ġallowing ĠAP A Ġto Ġgeneral ize Ġand Ġhypothes ize Ġthat Ġthe Ġprospective Ġinstructor Ġintervention Ġis Ġbased Ġon Ġthe Ġcontext Ġthat Ġany Ġprevious Ġpost ĠIN LINE FORM 0 Ġreplied Ġto . ĠIn Ġthis Ġmodel , Ġeach Ġpost ĠIN LINE FORM 1 Ġis Ġset Ġas Ġa Ġquery Ġto Ġattend Ġto Ġits Ġprevious Ġcontext ĠIN LINE FORM 2 Ġ. ĠFor Ġexample , ĠIN LINE FORM 3 Ġwill Ġattend Ġto ĠIN LINE FORM 4 Ġ. ĠDifferent Ġfrom Ġstandard Ġattention Ġmechanisms , ĠAP A Ġattention Ġweights ĠIN LINE FORM 5 Ġare Ġobtained Ġby Ġnormal ising Ġinteraction Ġmatrix Ġover Ġthe Ġdifferent Ġqueries . </s> In ĠAP A , Ġthe Ġattention Ġcontext ĠIN LINE FORM 0 Ġis Ġcomputed Ġvia : ĠDIS PLAY FORM 0 </s> E val uation </s> The Ġbaseline Ġand Ġthe Ġmodels Ġare Ġevaluated Ġon Ġa Ġcorpus Ġof Ġ12 ĠMO OC Ġdiscussion Ġforums . ĠWe Ġtrain Ġon Ġ80 % Ġof Ġthe Ġtraining Ġdata Ġand Ġreport Ġevaluation Ġresults Ġon Ġthe Ġheld - out Ġ20 % Ġof Ġtest Ġdata . ĠWe Ġreport ĠIN LINE FORM 0 Ġscores Ġon Ġthe Ġpositive Ġclass Ġ( inter ventions ), Ġin Ġline Ġwith Ġprior Ġwork . ĠWe Ġalso Ġargue Ġthat Ġrecall Ġof Ġthe Ġpositive Ġclass Ġis Ġmore Ġimportant Ġthan Ġprecision , Ġsince Ġit Ġis Ġcost lier Ġfor Ġinstructors Ġto Ġmiss Ġintervening Ġon Ġa Ġthread Ġthan Ġspending Ġirrelevant Ġtime Ġintervening Ġon Ġa Ġless Ġcritical Ġthreads Ġdue Ġto Ġfalse Ġpositives . </s> Model Ġhyper p am eter Ġsettings . ĠAll Ġproposed Ġand Ġbaseline Ġneural Ġmodels Ġare Ġtrained Ġusing ĠAdam Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 001 . ĠWe Ġused Ġcross - ent ropy Ġas Ġloss Ġfunction . ĠImport antly Ġwe Ġupdated Ġthe Ġmodel Ġparameters Ġduring Ġtraining Ġafter Ġeach Ġinstance Ġas Ġin Ġvanilla Ġsto ch astic Ġgradient Ġdescent ; Ġthis Ġsetting Ġwas Ġpractical Ġsince Ġdata Ġon Ġmost Ġcourses Ġhad Ġonly Ġa Ġfew Ġhundred Ġinstances Ġenabling Ġconvergence Ġwithin Ġa Ġreasonable Ġtraining Ġtime Ġof Ġa Ġfew Ġhours Ġ( see ĠTable ĠT AB REF 15 Ġ, Ġcolumn Ġ2 ). ĠModels Ġwere Ġtrained Ġfor Ġa Ġsingle Ġepoch Ġas Ġmost Ġof Ġour Ġcourses Ġwith Ġa Ġfew Ġhundred Ġthread Ġconver ged Ġafter Ġa Ġsingle Ġepoch . ĠWe Ġused Ġ300 - dimensional ĠGl o Ve Ġvectors Ġand Ġpermitted Ġthe Ġembed d ings Ġto Ġbe Ġupdated Ġduring Ġthe Ġmodel 's Ġend - to - end Ġtraining . ĠThe Ġhidden Ġdimension Ġsize Ġof Ġboth ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġset Ġto Ġ128 Ġfor Ġall Ġthe Ġmodels . </s> Bas elines . ĠWe Ġcompare Ġour Ġmodels Ġagainst Ġa Ġneural Ġbaseline Ġmodels , Ġhierarchical ĠL ST M Ġ( h L ST M ), Ġwith Ġthe Ġattention Ġab lated Ġbut Ġwith Ġaccess Ġto Ġthe Ġcomplete Ġcontext , Ġand Ġa Ġstrong , Ġopen - s ourced Ġfeature - rich Ġbaseline ĠB IB REF 7 Ġ. ĠWe Ġchoose ĠB IB REF 7 Ġover Ġother Ġprior Ġworks Ġsuch Ġas ĠB IB REF 0 Ġsince Ġwe Ġdo Ġnot Ġhave Ġaccess Ġto Ġthe Ġdataset Ġor Ġthe Ġsystem Ġused Ġin Ġtheir Ġpapers Ġfor Ġreplication . ĠB IB REF 7 Ġis Ġa Ġlog istic Ġregression Ġclass ifier Ġwith Ġfeatures Ġinclusive Ġof Ġbag - of - words Ġrepresentation Ġof Ġthe Ġun ig rams Ġand Ġthread Ġlength , Ġnormal ised Ġcounts Ġof Ġagreements Ġto Ġprevious Ġposts , Ġcounts Ġof Ġnon - lex ical Ġreference Ġitems Ġsuch Ġas ĠURLs , Ġand Ġthe ĠCour ser a Ġforum Ġtype Ġin Ġwhich Ġa Ġthread Ġappeared . ĠWe Ġalso Ġreport Ġaggreg ated Ġresults Ġfrom Ġa Ġh L ST M Ġmodel Ġwith Ġaccess Ġonly Ġto Ġthe Ġlast Ġpost Ġas Ġcontext Ġfor Ġcomparison . ĠTable ĠT AB REF 17 Ġcompares Ġthe Ġperformance Ġof Ġthese Ġbas elines Ġagainst Ġour Ġproposed Ġmethods . </s> Results </s> Table ĠT AB REF 15 Ġshows Ġperformance Ġof Ġall Ġour Ġproposed Ġmodels Ġand Ġthe Ġneural Ġbaseline Ġover Ġour Ġ12 ĠMO OC Ġdataset . ĠOur Ġmodels Ġof ĠU PA , ĠP PA Ġindividually Ġbetter Ġthe Ġbaseline Ġby Ġ5 Ġand Ġ2 % Ġon ĠIN LINE FORM 0 Ġand Ġ3 Ġand Ġ6 % Ġon Ġrecall Ġrespectively . ĠU PA Ġperforms Ġthe Ġbest Ġin Ġterms Ġof ĠIN LINE FORM 1 Ġon Ġaverage Ġwhile ĠP PA Ġperforms Ġthe Ġbest Ġin Ġterms Ġof Ġrecall Ġon Ġaverage . ĠAt Ġthe Ġindividual Ġcourse Ġlevel , Ġhowever , Ġthe Ġresults Ġare Ġmixed . ĠU PA Ġperforms Ġthe Ġbest Ġon ĠIN LINE FORM 2 Ġon Ġ5 Ġout Ġof Ġ12 Ġcourses , ĠP PA Ġon Ġ3 Ġout Ġ12 Ġcourses , ĠAP A Ġ1 Ġout Ġof Ġ12 Ġcourses Ġand Ġthe Ġbaseline Ġh L ST M Ġon Ġ1 . ĠP PA Ġperforms Ġthe Ġbest Ġon Ġrecall Ġon Ġ7 Ġout Ġof Ġthe Ġ12 Ġcourses . ĠWe Ġalso Ġnote Ġthat Ġcourse Ġlevel Ġperformance Ġdifferences Ġcorrelate Ġwith Ġthe Ġcourse Ġsize Ġand Ġintervention Ġratio Ġ( here after , Ġi . rat io ), Ġwhich Ġis Ġthe Ġratio Ġof Ġintervened Ġto Ġnon - inter ven ed Ġthreads . ĠU PA Ġperforms Ġbetter Ġthan ĠP PA Ġand ĠAP A Ġon Ġlow Ġintervention Ġcourses Ġ( i . rat io ĠIN LINE FORM 3 Ġ0 . 25 ) Ġmainly Ġbecause ĠP PA Ġand ĠAP A 's Ġperformance Ġdrops Ġsteep ly Ġwhen Ġi . rat io Ġdrops Ġ( see Ġcol . 2 Ġparent hesis Ġand ĠIN LINE FORM 4 Ġof ĠP PA Ġand ĠAP A ). ĠWhile Ġall Ġthe Ġproposed Ġmodels Ġbeat Ġthe Ġbaseline Ġon Ġevery Ġcourse Ġexcept Ġcase based - 2 . ĠOn Ġmedical ne uro - 2 Ġand Ġcomp ilers - 4 Ġwhich Ġhave Ġthe Ġlowest Ġi . rat io Ġamong Ġthe Ġ12 Ġcourses Ġnone Ġof Ġthe Ġneural Ġmodels Ġbetter Ġthe Ġreported Ġbaseline ĠB IB REF 7 Ġ( course Ġlevel Ġnot Ġscores Ġnot Ġshown Ġin Ġthis Ġpaper ). ĠThe Ġeffect Ġis Ġpronounced Ġin Ġcomp ilers - 4 Ġcourse Ġwhere Ġnone Ġof Ġthe Ġneural Ġmodels Ġwere Ġable Ġto Ġpredict Ġany Ġintervened Ġthreads . ĠThis Ġis Ġdue Ġto Ġthe Ġinherent Ġweakness Ġof Ġstandard Ġneural Ġmodels , Ġwhich Ġare Ġunable Ġto Ġlearn Ġfeatures Ġwell Ġenough Ġwhen Ġfaced Ġwith Ġsparse Ġdata . </s> The Ġbest Ġperformance Ġof ĠU PA Ġindicates Ġthat Ġthe Ġreply Ġcontext Ġof Ġthe Ġinstructor 's Ġpost ĠIN LINE FORM 0 Ġcorrelates Ġstrongly Ġwith Ġthat Ġof Ġthe Ġprevious Ġpost ĠIN LINE FORM 1 Ġ. ĠThis Ġis Ġnot Ġsurprising Ġsince Ġnormal Ġconversations Ġare Ġtypically Ġstructured Ġthat Ġway . </s> Discussion </s> In Ġorder Ġto Ġfurther Ġunderstand Ġthe Ġmodels ' Ġability Ġto Ġinfer Ġthe Ġcontext Ġand Ġits Ġeffect Ġon Ġintervention Ġprediction , Ġwe Ġfurther Ġinvestigate Ġthe Ġfollowing Ġresearch Ġquestions . </s> R Q 1 . ĠDoes Ġcontext Ġinference Ġhelp Ġintervention Ġprediction ? </s> In Ġorder Ġto Ġunderstand Ġif Ġcontext Ġinference Ġis Ġuseful Ġto Ġintervention Ġprediction , Ġwe Ġab late Ġthe Ġattention Ġcomponents Ġand Ġexperiment Ġwith Ġthe Ġvanilla Ġhierarchical ĠL ST M Ġmodel . ĠRow Ġ3 Ġof ĠTable ĠT AB REF 17 Ġshows Ġthe Ġmacro Ġaveraged Ġresult Ġfrom Ġthis Ġexperiment . ĠThe ĠU PA Ġand ĠP PA Ġattention Ġmodels Ġbetter Ġthe Ġvanilla Ġh L ST M Ġby Ġ5 % Ġand Ġ2 % Ġon Ġaverage Ġin ĠIN LINE FORM 0 Ġrespectively . ĠRecall Ġthat Ġthe Ġvanilla Ġh L ST M Ġalready Ġhas Ġaccess Ġto Ġa Ġcontext Ġconsisting Ġof Ġall Ġposts Ġ( from ĠIN LINE FORM 1 Ġthrough ĠIN LINE FORM 2 Ġ). ĠIn Ġcontrast , Ġthe ĠU PA Ġand ĠP PA Ġmodels Ġselectively Ġinf ers Ġa Ġcontext Ġfor ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġposts , Ġrespectively , Ġand Ġuse Ġit Ġto Ġpredict Ġintervention . ĠThe Ġimproved Ġperformance Ġof Ġour Ġattention Ġmodels Ġthat Ġactively Ġselect Ġtheir Ġoptimal Ġcontext , Ġover Ġa Ġmodel Ġwith Ġthe Ġcomplete Ġthread Ġas Ġcontext , Ġh L ST M , Ġshows Ġthat Ġthe Ġcontext Ġinference Ġimproves Ġintervention Ġprediction Ġover Ġusing Ġthe Ġdefault Ġfull Ġcontext . </s> R Q 2 . ĠHow Ġwell Ġdo Ġthe Ġmodels Ġperform Ġacross Ġthreads Ġof Ġdifferent Ġlengths ? ĠTo Ġunderstand Ġthe Ġmodels ' Ġprediction Ġperformance Ġacross Ġthreads Ġof Ġdifferent Ġlengths , Ġwe Ġbin Ġthreads Ġby Ġlength Ġand Ġstudy Ġthe Ġmodels ' Ġrecall . ĠWe Ġchoose Ġthree Ġcourses , Ġml - 5 , Ġr pro g - 3 Ġand Ġcalc - 1 , Ġfrom Ġour Ġcorpus Ġof Ġ12 Ġwith Ġthe Ġhighest Ġnumber Ġof Ġpositive Ġinstances Ġ( ĠIN LINE FORM 0 Ġ100 Ġthreads ). ĠWe Ġlimit Ġour Ġanalysis Ġto Ġthese Ġsince Ġbin ning Ġrenders Ġcourses Ġwith Ġfewer Ġpositive Ġinstances Ġsparse . ĠFigure ĠFIG REF 18 Ġshows Ġperformance Ġacross Ġthread Ġlengths Ġfrom Ġ1 Ġthrough Ġ7 Ġposts Ġand ĠIN LINE FORM 1 Ġposts . ĠClearly , Ġthe ĠU PA Ġmodel Ġperforms Ġmuch Ġbetter Ġon Ġshorter Ġthreads Ġthan Ġon Ġlonger Ġthreads Ġwhile ĠP PA Ġand ĠAP A Ġworks Ġbetter Ġon Ġlonger Ġthreads . ĠAlthough , ĠU PA Ġis Ġthe Ġbest Ġperforming Ġmodel Ġin Ġterms Ġof Ġoverall ĠIN LINE FORM 2 Ġits Ġperformance Ġdrops Ġsteep ly Ġon Ġthreads Ġof Ġlength ĠIN LINE FORM 3 Ġ. ĠU PA 's Ġoverall Ġbest Ġperformance Ġis Ġbecause Ġmost Ġof Ġthe Ġinterventions Ġin Ġthe Ġcorpus Ġhappen Ġafter Ġone Ġpost . ĠTo Ġhighlight Ġthe Ġperformance Ġof ĠAP A Ġwe Ġshow Ġan Ġexample Ġfrom Ġsm ac - 1 Ġin ĠFigure ĠFIG REF 22 Ġwith Ġnine Ġposts Ġwhich Ġwas Ġpredicted Ġcorrectly Ġas Ġintervened Ġby ĠAP A Ġbut Ġnot Ġby Ġother Ġmodels . ĠThread s Ġshows Ġstudents Ġconfused Ġover Ġa Ġmissing Ġfigure Ġin Ġa Ġhomework . ĠThe Ġinstructor Ġfinally Ġshows Ġup , Ġthough Ġlate , Ġto Ġresolve Ġthe Ġconfusion . </s> R Q 3 . ĠDo Ġmodels Ġtrained Ġwith Ġdifferent Ġcontext Ġlengths Ġperform Ġbetter Ġthan Ġwhen Ġtrained Ġon Ġa Ġsingle Ġcontext Ġlength ? </s> We Ġfind Ġthat Ġcontext Ġlength Ġhas Ġa Ġregular ising Ġeffect Ġon Ġthe Ġmodel 's Ġperformance Ġat Ġtest Ġtime . ĠThis Ġis Ġnot Ġsurprising Ġsince Ġmodels Ġtrained Ġwith Ġthreads Ġof Ġsingle Ġcontext Ġlength Ġwill Ġnot Ġgeneral ise Ġto Ġinfer Ġdifferent Ġcontext Ġlengths . ĠRow Ġ4 Ġof ĠTable ĠT AB REF 17 Ġshows Ġa Ġsteep Ġperformance Ġdrop Ġin Ġtraining Ġby Ġclass ifier Ġwith Ġall Ġthreads Ġtrunc ated Ġto Ġa Ġcontext Ġof Ġjust Ġone Ġpost , ĠIN LINE FORM 0 Ġ, Ġthe Ġpost Ġimmediately Ġpreceding Ġthe Ġintervened Ġpost . ĠWe Ġalso Ġconducted Ġan Ġexperiment Ġwith Ġa Ġmulti - object ive Ġloss Ġfunction Ġwith Ġan Ġadditive Ġcross - ent ropy Ġterm Ġwhere Ġeach Ġterm Ġcomp utes Ġloss Ġfrom Ġa Ġmodel Ġwith Ġcontext Ġlimited Ġto Ġa Ġlength Ġof Ġ3 . ĠWe Ġchose Ġ3 Ġsince Ġintervened Ġthreads Ġin Ġall Ġthe Ġcourses Ġhad Ġa Ġmedian Ġlength Ġbetween Ġ3 Ġand Ġ4 . ĠWe Ġachieved Ġan ĠIN LINE FORM 1 Ġof Ġ0 . 45 Ġwith Ġa Ġprecision Ġof Ġ0 . 47 Ġand Ġrecall Ġof Ġ0 . 43 . ĠThis Ġachieves Ġa Ġperformance Ġcomparable Ġto Ġthat Ġof Ġthe ĠB IB REF 7 Ġwith Ġcontext Ġlength Ġset Ġto Ġonly Ġto Ġ3 . ĠThis Ġapproach Ġof Ġusing Ġinfinitely Ġmany Ġloss Ġterms Ġfor Ġeach Ġcontext Ġlength Ġfrom Ġ1 Ġthrough Ġthe Ġmaximum Ġthread Ġlength Ġin Ġa Ġcourse Ġis Ġnaive Ġand Ġnot Ġpractical . ĠWe Ġonly Ġuse Ġthis Ġmodel Ġto Ġshow Ġthe Ġimportance Ġof Ġtraining Ġthe Ġmodel Ġwith Ġloss Ġfrom Ġthreads Ġof Ġdifferent Ġlengths Ġto Ġprevent Ġmodels Ġover fitting Ġto Ġthreads Ġof Ġspecific Ġcontext Ġlengths . </s> Conclusion </s> We Ġpredict Ġinstructor Ġintervention Ġon Ġstudent Ġdiscussions Ġby Ġfirst Ġinfer ring Ġthe Ġoptimal Ġsize Ġof Ġthe Ġcontext Ġneeded Ġto Ġdecide Ġon Ġthe Ġintervention Ġdecision Ġfor Ġthe Ġintervened Ġpost . ĠWe Ġfirst Ġshow Ġthat Ġa Ġstructured Ġrepresentation Ġof Ġthe Ġcomplete Ġthread Ġas Ġthe Ġcontext Ġis Ġbetter Ġthan Ġa Ġbag - of - words , Ġfeature - rich Ġrepresentation . ĠWe Ġthen Ġpropose Ġattention - based Ġmodels Ġto Ġinfer Ġand Ġselect Ġa Ġcontext ĠâĢĵ Ġdefined Ġas Ġa Ġcontiguous Ġsubsequ ence Ġof Ġstudent Ġposts ĠâĢĵ Ġto Ġimprove Ġover Ġa Ġmodel Ġthat Ġalways Ġtakes Ġthe Ġcomplete Ġthread Ġas Ġa Ġcontext Ġto Ġprediction Ġintervention . ĠOur ĠAny ĠPost ĠAttention Ġ( AP A ) Ġmodel Ġenables Ġinstructors Ġto Ġtune Ġthe Ġmodel Ġto Ġpredict Ġintervention Ġearly Ġor Ġlate . ĠWe Ġposit Ġour ĠAP A Ġmodel Ġwill Ġenable ĠMO OC Ġinstructors Ġemploying Ġvarying Ġped ag ogical Ġstyles Ġto Ġuse Ġthe Ġmodel Ġequally Ġwell . ĠWe Ġintro spect Ġthe Ġattention Ġmodels ' Ġperformance Ġacross Ġthreads Ġof Ġvarying Ġlengths Ġand Ġshow Ġthat ĠAP A Ġpredicts Ġintervention Ġon Ġlonger Ġthreads , Ġwhich Ġpossesses Ġmore Ġcandidate Ġcontexts , Ġbetter . </s> We Ġnote Ġthat Ġthe Ġrecall Ġof Ġthe Ġpredictive Ġmodels Ġfor Ġlonger Ġthreads Ġ( that Ġis , Ġthreads Ġof Ġlength Ġgreater Ġ2 ) Ġcan Ġstill Ġbe Ġimproved . ĠModels Ġperform Ġdifferently Ġbetween Ġshorter Ġand Ġlonger Ġlength . ĠAn Ġensemble Ġmodel Ġor Ġa Ġmulti - object ive Ġloss Ġfunction Ġis Ġthus Ġplanned Ġin Ġour Ġfuture Ġwork Ġto Ġbetter Ġprediction Ġon Ġsuch Ġlonger Ġthreads .
Introduction Massive Open Online Courses (MOOCs) have strived to bridge the social gap in higher education by bringing quality education from reputed universities to students at large. Such massive scaling through online classrooms, however, disrupt co-located, synchronous two-way communication between the students and the instructor. MOOC platforms provide discussion forums for students to talk to their classmates about the lectures, homeworks, quizzes and provide a venue to socialise. Instructors (defined here as the course instructors, their teaching assistants and the MOOC platform's technical staff) monitor the discussion forum to post (reply to their message) in discussion threads among students. We refer to this posting as intervention, following prior work BIBREF0 . However, due to large student enrolment, the student–instructor ratio in MOOCs is very high Therefore, instructors are not able to monitor and participate in all student discussions. To address this problem, a number of works have proposed systems e.g., BIBREF0 , BIBREF1 to aid instructors to selectively intervene on student discussions where they are needed the most. In this paper, we improve the state-of-the-art for instructor intervention in MOOC forums. We propose the first neural models for this prediction problem. We show that modelling the thread structure and the sequence of posts explicitly improves performance. Instructors in different MOOCs from different subject areas intervene differently. For example, on a Science, Technology, Engineering and Mathematics (STEM) MOOC, instructors may often intervene early as possible to resolve misunderstanding of the subject material and prevent confusion. However, in a Humanities MOOC, instructors allow for the students to explore open-ended discussions and debate among themselves. Such instructors may prefer to intervene later in the discussion to encourage further discussion or resolve conflicts among students. We therefore propose attention models to infer the latent context, i.e., the series of posts that trigger an intervention. Earlier studies on MOOC forum intervention either model the entire context or require the context size to be specified explicitly. Problem Statement A thread INLINEFORM0 consists of a series of posts INLINEFORM1 through INLINEFORM2 where INLINEFORM3 is an instructor's post when INLINEFORM4 is intervened, if applicable. INLINEFORM5 is considered intervened if an instructor had posted at least once. The problem of predicting instructor intervention is cast as a binary classification problem. Intervened threads are predicted as 1 given while non-intervened threads are predicted as 0 given posts INLINEFORM6 through INLINEFORM7 . The primary problem leads to a secondary problem of inferring the appropriate amount of context to intervene. We define a context INLINEFORM0 of a post INLINEFORM1 as a series of linear contiguous posts INLINEFORM2 through INLINEFORM3 where INLINEFORM4 . The problem of inferring context is to identify context INLINEFORM5 from a set of candidate contexts INLINEFORM6 . Modelling Context in Forums Context has been used and modelled in various ways for different problems in discussion forums. In a work on a closely related problem of forum thread retrieval BIBREF2 models context using inter-post discourse e.g., Question-Answer. BIBREF3 models the structural dependencies and relationships between forum posts using a conditional random field in their problem to infer the reply structure. Unlike BIBREF2 , BIBREF3 can be used to model any structural dependency and is, therefore, more general. In this paper, we seek to infer general dependencies between a reply and its previous context whereas BIBREF3 inference is limited to pairs of posts. More recently BIBREF4 proposed a context based model which factorises attention over threads of different lengths. Differently, we do not model length but the context before a post. However, our attention models cater to threads of all lengths. BIBREF5 proposed graph structured LSTM to model the explicit reply structure in Reddit forums. Our work does not assume access to such a reply structure because 1) Coursera forums do not provide one and 2) forum participants often err by posting their reply to a different post than that they intended. At the other end of the spectrum are document classification models that do not assume structure in the document layout but try to infer inherent structure in the natural language, viz, words, sentences, paragraphs and documents. Hierarchical attention BIBREF6 is a well know recent work that classifies documents using a multi-level LSTMs with attention mechanism to select important units at each hierarchical level. Differently, we propose a hierarchical model that encodes layout hierarchy between a post and a thread but also infers reply structure using a attention mechanism since the layout does not reliably encode it. Instructor Intervention in MOOC forums The problem of predicting instructor intervention in MOOCs was proposed by BIBREF0 . Later BIBREF7 evaluated baseline models by BIBREF0 over a larger corpus and found the results to vary widely across MOOCs. Since then subsequent works have used similar diverse evaluations on the same prediction problem BIBREF1 , BIBREF8 . BIBREF1 proposed models with discourse features to enable better prediction over unseen MOOCs. BIBREF8 recently showed interventions on Coursera forums to be biased by the position at which a thread appears to an instructor viewing the forum interface and proposed methods for debiased prediction. While all works since BIBREF0 address key limitations in this line of research, they have not investigated the role of structure and sequence in the threaded discussion in predicting instructor interventions. BIBREF0 proposed probabilistic graphical models to model structure and sequence. They inferred vocabulary dependent latent post categories to model the thread sequence and infer states that triggered intervention. Their model, however, requires a hyperparameter for the number of latent states. It is likely that their empirically reported setting will not generalise due to their weak evaluation BIBREF7 . In this paper, we propose models to infer the context that triggers instructor intervention that does not require context lengths to be set apriori. All our proposed models generalise over modelling assumptions made by BIBREF0 . For the purpose of comparison against a state-of-the-art and competing baselines we choose BIBREF7 since BIBREF0 's system and data are not available for replication. Data and Preprocessing We evaluate our proposed models over a corpus of 12 MOOC iterations (offerings) on Coursera.org In partnership with Coursera and in line with its Terms of Service, we obtained the data for use in our academic research. Following prior work BIBREF7 we evaluate over a diverse dataset to represent MOOCs of varying sizes, instructor styles, instructor team sizes and number of threads intervened. We only include threads from sub-forums on Lecture, Homework, Quiz and Exam. We also normalise and label sub-forums with other non-standard names (e.g., Assignments instead of Homework) into of the four said sub-forums. Threads on general discussion, meet and greet and other custom sub-forums for social chitchat are omitted as our focus is to aid instructors on intervening on discussion on the subject matter. We also exclude announcement threads and other threads started by instructors since they are not interventions. We preprocess each thread by replacing URLs, equations and other mathematical formulae and references to timestamps in lecture videos by tokens INLINEFORM0 URL INLINEFORM1 , INLINEFORM2 MATH INLINEFORM3 , INLINEFORM4 TIMEREF INLINEFORM5 respectively. We also truncate intervened threads to only include posts before the first instructor post since the instructor's and subsequent posts will bias the prediction due to the instructor's post. Model The key innovation of our work is to decompose the intervention prediction problem into a two-stage model that first explicitly tries to discover the proper context to which a potential intervention could be replying to, and then, predict the intervention status. This model implicitly assesses the importance (or urgency) of the existing thread's context to decide whether an intervention is necessary. For example in Figure SECREF1 , prior to the instructor's intervention, the ultimate post (Post #6) by Student 2 already acknowledged the OP's gratitude for his answer. In this regard, the instructor may have decided to use this point to summarize the entire thread to consolidate all the pertinent positions. Here, we might assume that the instructor's reply takes the entire thread (Posts #1–6) as the context for her reply. This subproblem of inferring the context scope is where our innovation centers on. To be clear, in order to make the prediction that a instruction intervention is now necessary on a thread, the instructor's reply is not yet available — the model predicts whether a reply is necessary — so in the example, only Posts #1–6 are available in the problem setting. To infer the context, we have to decide which subsequence of posts are the most plausible motivation for an intervention. Recent work in deep neural modeling has used an attention mechanism as a focusing query to highlight specific items within the input history that significantly influence the current decision point. Our work employs this mechanism – but with a twist: due to the fact that the actual instructor intervention is not (yet) available at the decision timing, we cannot use any actual intervention to decide the context. To employ attention, we must then employ a surrogate text as the query to train our prediction model. Our model variants model assess the suitability of such surrogate texts for the attention mechanism basis. Congruent with the representation of the input forums, in all our proposed models, we encode the discussion thread hierarchically. We first build representations for each post by passing pre-trained word vector representations from GloVe BIBREF9 for each word through an LSTM BIBREF10 , INLINEFORM0 . We use the last layer output of the LSTM as a representation of the post. We refer this as the post vector INLINEFORM1 . Then each post INLINEFORM0 is passed through another LSTM, INLINEFORM1 , whose last layer output forms the encoding of the entire thread. Hidden unit outputs of INLINEFORM2 represent the contexts INLINEFORM3 ; that is, snapshots of the threads after each post, as shown in Figure FIGREF1 . The INLINEFORM0 and INLINEFORM1 together constitute the hierarchical LSTM (hLSTM) model. This general hLSTM model serves as the basis for our model exploration in the rest of this section. Contextual Attention Models When they intervene, instructors either pay attention to a specific post or a series of posts, which trigger their reply. However, instructors rarely explicitly indicate to which post(s) their intervention is in relation to. This is the case in our corpus, party due to Coursera's user interface which only allows for single level comments (see Figure FIGREF2 ). Based solely on the binary, thread-level intervention signal, our secondary objective seeks to infer the appropriate context – represented by a sequence of posts – as the basis for the intervention. We only consider linear contiguous series of posts starting with the thread's original post to constitute to a context; e.g., INLINEFORM0 . This is a reasonable as MOOC forum posts always reply to the original post or to a subsequent post, which in turn replies to the original post. This is in contrast to forums such as Reddit that have a tree or graph-like structure that require forum structure to be modelled explicitly, such as in BIBREF5 . We propose three neural attention BIBREF11 variants based on how an instructor might attend and reply to a context in a thread: the ultimate, penultimate and any post attention models. We review each of these in turn. Ultimate Post Attention (UPA) Model. In this model we attend to the context represented by hidden state of the INLINEFORM0 . We use the post prior to the instructor's reply as a query over the contexts INLINEFORM1 to compute attention weights INLINEFORM2 , which are then used to compute the attended context representation INLINEFORM3 (recall again that the intervention text itself is not available for this purpose). This attention formulation makes an equivalence between the final INLINEFORM4 post and the prospective intervention, using Post INLINEFORM5 as the query for finding the appropriate context INLINEFORM6 , inclusive of itself INLINEFORM7 . Said in another way, UPA uses the most recent content in the thread as the attentional query for context. For example, if post INLINEFORM0 is the instructor's reply, post INLINEFORM1 will query over the contexts INLINEFORM2 and INLINEFORM3 . The model schematic is shown in Figure FIGREF12 . The attended context representations are computed as: DISPLAYFORM0 The INLINEFORM0 representation is then passed through a fully connected softmax layer to yield the binary prediction. Penultimate Post Attention (PPA) Model. While the UPA model uses the most recent text and makes the ultimate post itself available as potential context, our the ultimate post may be better modeled as having any of its prior posts as potential context. Penultimate Post Attention (PPA) variant does this. The schematic and the equations for the PPA model are obtained by summing over contexts INLINEFORM0 in Equation EQREF10 and Figure FIGREF12 . While we could properly model such a context inference decision with any post INLINEFORM1 and prospective contexts INLINEFORM2 (where INLINEFORM3 is a random post), it makes sense to use the penultimate post, as we can make the most information available to the model for the context inference. The attended context representations are computed as: DISPLAYFORM0 Any Post Attention (APA) Model. APA further relaxes both UPA and PPA, allowing APA to generalize and hypothesize that the prospective instructor intervention is based on the context that any previous post INLINEFORM0 replied to. In this model, each post INLINEFORM1 is set as a query to attend to its previous context INLINEFORM2 . For example, INLINEFORM3 will attend to INLINEFORM4 . Different from standard attention mechanisms, APA attention weights INLINEFORM5 are obtained by normalising interaction matrix over the different queries. In APA, the attention context INLINEFORM0 is computed via: DISPLAYFORM0 Evaluation The baseline and the models are evaluated on a corpus of 12 MOOC discussion forums. We train on 80% of the training data and report evaluation results on the held-out 20% of test data. We report INLINEFORM0 scores on the positive class (interventions), in line with prior work. We also argue that recall of the positive class is more important than precision, since it is costlier for instructors to miss intervening on a thread than spending irrelevant time intervening on a less critical threads due to false positives. Model hyperpameter settings. All proposed and baseline neural models are trained using Adam optimizer with a learning rate of 0.001. We used cross-entropy as loss function. Importantly we updated the model parameters during training after each instance as in vanilla stochastic gradient descent; this setting was practical since data on most courses had only a few hundred instances enabling convergence within a reasonable training time of a few hours (see Table TABREF15 , column 2). Models were trained for a single epoch as most of our courses with a few hundred thread converged after a single epoch. We used 300-dimensional GloVe vectors and permitted the embeddings to be updated during the model's end-to-end training. The hidden dimension size of both INLINEFORM0 and INLINEFORM1 are set to 128 for all the models. Baselines. We compare our models against a neural baseline models, hierarchical LSTM (hLSTM), with the attention ablated but with access to the complete context, and a strong, open-sourced feature-rich baseline BIBREF7 . We choose BIBREF7 over other prior works such as BIBREF0 since we do not have access to the dataset or the system used in their papers for replication. BIBREF7 is a logistic regression classifier with features inclusive of bag-of-words representation of the unigrams and thread length, normalised counts of agreements to previous posts, counts of non-lexical reference items such as URLs, and the Coursera forum type in which a thread appeared. We also report aggregated results from a hLSTM model with access only to the last post as context for comparison. Table TABREF17 compares the performance of these baselines against our proposed methods. Results Table TABREF15 shows performance of all our proposed models and the neural baseline over our 12 MOOC dataset. Our models of UPA, PPA individually better the baseline by 5 and 2% on INLINEFORM0 and 3 and 6% on recall respectively. UPA performs the best in terms of INLINEFORM1 on average while PPA performs the best in terms of recall on average. At the individual course level, however, the results are mixed. UPA performs the best on INLINEFORM2 on 5 out of 12 courses, PPA on 3 out 12 courses, APA 1 out of 12 courses and the baseline hLSTM on 1. PPA performs the best on recall on 7 out of the 12 courses. We also note that course level performance differences correlate with the course size and intervention ratio (hereafter, i.ratio), which is the ratio of intervened to non-intervened threads. UPA performs better than PPA and APA on low intervention courses (i.ratio INLINEFORM3 0.25) mainly because PPA and APA's performance drops steeply when i.ratio drops (see col.2 parenthesis and INLINEFORM4 of PPA and APA). While all the proposed models beat the baseline on every course except casebased-2. On medicalneuro-2 and compilers-4 which have the lowest i.ratio among the 12 courses none of the neural models better the reported baseline BIBREF7 (course level not scores not shown in this paper). The effect is pronounced in compilers-4 course where none of the neural models were able to predict any intervened threads. This is due to the inherent weakness of standard neural models, which are unable to learn features well enough when faced with sparse data. The best performance of UPA indicates that the reply context of the instructor's post INLINEFORM0 correlates strongly with that of the previous post INLINEFORM1 . This is not surprising since normal conversations are typically structured that way. Discussion In order to further understand the models' ability to infer the context and its effect on intervention prediction, we further investigate the following research questions. RQ1. Does context inference help intervention prediction? In order to understand if context inference is useful to intervention prediction, we ablate the attention components and experiment with the vanilla hierarchical LSTM model. Row 3 of Table TABREF17 shows the macro averaged result from this experiment. The UPA and PPA attention models better the vanilla hLSTM by 5% and 2% on average in INLINEFORM0 respectively. Recall that the vanilla hLSTM already has access to a context consisting of all posts (from INLINEFORM1 through INLINEFORM2 ). In contrast, the UPA and PPA models selectively infers a context for INLINEFORM3 and INLINEFORM4 posts, respectively, and use it to predict intervention. The improved performance of our attention models that actively select their optimal context, over a model with the complete thread as context, hLSTM, shows that the context inference improves intervention prediction over using the default full context. RQ2. How well do the models perform across threads of different lengths? To understand the models' prediction performance across threads of different lengths, we bin threads by length and study the models' recall. We choose three courses, ml-5, rprog-3 and calc-1, from our corpus of 12 with the highest number of positive instances ( INLINEFORM0 100 threads). We limit our analysis to these since binning renders courses with fewer positive instances sparse. Figure FIGREF18 shows performance across thread lengths from 1 through 7 posts and INLINEFORM1 posts. Clearly, the UPA model performs much better on shorter threads than on longer threads while PPA and APA works better on longer threads. Although, UPA is the best performing model in terms of overall INLINEFORM2 its performance drops steeply on threads of length INLINEFORM3 . UPA's overall best performance is because most of the interventions in the corpus happen after one post. To highlight the performance of APA we show an example from smac-1 in Figure FIGREF22 with nine posts which was predicted correctly as intervened by APA but not by other models. Threads shows students confused over a missing figure in a homework. The instructor finally shows up, though late, to resolve the confusion. RQ3. Do models trained with different context lengths perform better than when trained on a single context length? We find that context length has a regularising effect on the model's performance at test time. This is not surprising since models trained with threads of single context length will not generalise to infer different context lengths. Row 4 of Table TABREF17 shows a steep performance drop in training by classifier with all threads truncated to a context of just one post, INLINEFORM0 , the post immediately preceding the intervened post. We also conducted an experiment with a multi-objective loss function with an additive cross-entropy term where each term computes loss from a model with context limited to a length of 3. We chose 3 since intervened threads in all the courses had a median length between 3 and 4. We achieved an INLINEFORM1 of 0.45 with a precision of 0.47 and recall of 0.43. This achieves a performance comparable to that of the BIBREF7 with context length set to only to 3. This approach of using infinitely many loss terms for each context length from 1 through the maximum thread length in a course is naive and not practical. We only use this model to show the importance of training the model with loss from threads of different lengths to prevent models overfitting to threads of specific context lengths. Conclusion We predict instructor intervention on student discussions by first inferring the optimal size of the context needed to decide on the intervention decision for the intervened post. We first show that a structured representation of the complete thread as the context is better than a bag-of-words, feature-rich representation. We then propose attention-based models to infer and select a context – defined as a contiguous subsequence of student posts – to improve over a model that always takes the complete thread as a context to prediction intervention. Our Any Post Attention (APA) model enables instructors to tune the model to predict intervention early or late. We posit our APA model will enable MOOC instructors employing varying pedagogical styles to use the model equally well. We introspect the attention models' performance across threads of varying lengths and show that APA predicts intervention on longer threads, which possesses more candidate contexts, better. We note that the recall of the predictive models for longer threads (that is, threads of length greater 2) can still be improved. Models perform differently between shorter and longer length. An ensemble model or a multi-objective loss function is thus planned in our future work to better prediction on such longer threads.
What aspects of discussion are relevant to instructor intervention, according to the attention mechanism?
49eb52b3ec0647e165a5e41488088c80a20cc78f
What Ġaspects Ġof Ġdiscussion Ġare Ġrelevant Ġto Ġinstructor Ġintervention , Ġaccording Ġto Ġthe Ġattention Ġmechanism ?
context Ġinference
<s> What Ġaspects Ġof Ġdiscussion Ġare Ġrelevant Ġto Ġinstructor Ġintervention , Ġaccording Ġto Ġthe Ġattention Ġmechanism ? </s> Introduction </s> Mass ive ĠOpen ĠOnline ĠCour ses Ġ( MO OC s ) Ġhave Ġstri ved Ġto Ġbridge Ġthe Ġsocial Ġgap Ġin Ġhigher Ġeducation Ġby Ġbringing Ġquality Ġeducation Ġfrom Ġrep uted Ġuniversities Ġto Ġstudents Ġat Ġlarge . ĠSuch Ġmassive Ġscaling Ġthrough Ġonline Ġclassrooms , Ġhowever , Ġdisrupt Ġco - l ocated , Ġsynchron ous Ġtwo - way Ġcommunication Ġbetween Ġthe Ġstudents Ġand Ġthe Ġinstructor . </s> MO OC Ġplatforms Ġprovide Ġdiscussion Ġforums Ġfor Ġstudents Ġto Ġtalk Ġto Ġtheir Ġclassmates Ġabout Ġthe Ġlectures , Ġhom eworks , Ġquizz es Ġand Ġprovide Ġa Ġvenue Ġto Ġsocial ise . ĠInstruct ors Ġ( defined Ġhere Ġas Ġthe Ġcourse Ġinstructors , Ġtheir Ġteaching Ġassistants Ġand Ġthe ĠMO OC Ġplatform 's Ġtechnical Ġstaff ) Ġmonitor Ġthe Ġdiscussion Ġforum Ġto Ġpost Ġ( reply Ġto Ġtheir Ġmessage ) Ġin Ġdiscussion Ġthreads Ġamong Ġstudents . ĠWe Ġrefer Ġto Ġthis Ġposting Ġas Ġintervention , Ġfollowing Ġprior Ġwork ĠB IB REF 0 Ġ. ĠHowever , Ġdue Ġto Ġlarge Ġstudent Ġenrol ment , Ġthe Ġstudent âĢĵ in struct or Ġratio Ġin ĠMO OC s Ġis Ġvery Ġhigh ĠTherefore , Ġinstructors Ġare Ġnot Ġable Ġto Ġmonitor Ġand Ġparticipate Ġin Ġall Ġstudent Ġdiscussions . ĠTo Ġaddress Ġthis Ġproblem , Ġa Ġnumber Ġof Ġworks Ġhave Ġproposed Ġsystems Ġe . g ., ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġto Ġaid Ġinstructors Ġto Ġselectively Ġintervene Ġon Ġstudent Ġdiscussions Ġwhere Ġthey Ġare Ġneeded Ġthe Ġmost . </s> In Ġthis Ġpaper , Ġwe Ġimprove Ġthe Ġstate - of - the - art Ġfor Ġinstructor Ġintervention Ġin ĠMO OC Ġforums . ĠWe Ġpropose Ġthe Ġfirst Ġneural Ġmodels Ġfor Ġthis Ġprediction Ġproblem . ĠWe Ġshow Ġthat Ġmodelling Ġthe Ġthread Ġstructure Ġand Ġthe Ġsequence Ġof Ġposts Ġexplicitly Ġimproves Ġperformance . ĠInstruct ors Ġin Ġdifferent ĠMO OC s Ġfrom Ġdifferent Ġsubject Ġareas Ġintervene Ġdifferently . ĠFor Ġexample , Ġon Ġa ĠScience , ĠTechnology , ĠEngineering Ġand ĠMathematics Ġ( STEM ) ĠMO OC , Ġinstructors Ġmay Ġoften Ġintervene Ġearly Ġas Ġpossible Ġto Ġresolve Ġmisunderstanding Ġof Ġthe Ġsubject Ġmaterial Ġand Ġprevent Ġconfusion . ĠHowever , Ġin Ġa ĠHuman ities ĠMO OC , Ġinstructors Ġallow Ġfor Ġthe Ġstudents Ġto Ġexplore Ġopen - ended Ġdiscussions Ġand Ġdebate Ġamong Ġthemselves . ĠSuch Ġinstructors Ġmay Ġprefer Ġto Ġintervene Ġlater Ġin Ġthe Ġdiscussion Ġto Ġencourage Ġfurther Ġdiscussion Ġor Ġresolve Ġconflicts Ġamong Ġstudents . ĠWe Ġtherefore Ġpropose Ġattention Ġmodels Ġto Ġinfer Ġthe Ġlatent Ġcontext , Ġi . e ., Ġthe Ġseries Ġof Ġposts Ġthat Ġtrigger Ġan Ġintervention . ĠEarlier Ġstudies Ġon ĠMO OC Ġforum Ġintervention Ġeither Ġmodel Ġthe Ġentire Ġcontext Ġor Ġrequire Ġthe Ġcontext Ġsize Ġto Ġbe Ġspecified Ġexplicitly . </s> Problem ĠStatement </s> A Ġthread ĠIN LINE FORM 0 Ġconsists Ġof Ġa Ġseries Ġof Ġposts ĠIN LINE FORM 1 Ġthrough ĠIN LINE FORM 2 Ġwhere ĠIN LINE FORM 3 Ġis Ġan Ġinstructor 's Ġpost Ġwhen ĠIN LINE FORM 4 Ġis Ġintervened , Ġif Ġapplicable . ĠIN LINE FORM 5 Ġis Ġconsidered Ġintervened Ġif Ġan Ġinstructor Ġhad Ġposted Ġat Ġleast Ġonce . ĠThe Ġproblem Ġof Ġpredicting Ġinstructor Ġintervention Ġis Ġcast Ġas Ġa Ġbinary Ġclassification Ġproblem . ĠInter ven ed Ġthreads Ġare Ġpredicted Ġas Ġ1 Ġgiven Ġwhile Ġnon - inter ven ed Ġthreads Ġare Ġpredicted Ġas Ġ0 Ġgiven Ġposts ĠIN LINE FORM 6 Ġthrough ĠIN LINE FORM 7 Ġ. </s> The Ġprimary Ġproblem Ġleads Ġto Ġa Ġsecondary Ġproblem Ġof Ġinfer ring Ġthe Ġappropriate Ġamount Ġof Ġcontext Ġto Ġintervene . ĠWe Ġdefine Ġa Ġcontext ĠIN LINE FORM 0 Ġof Ġa Ġpost ĠIN LINE FORM 1 Ġas Ġa Ġseries Ġof Ġlinear Ġcontiguous Ġposts ĠIN LINE FORM 2 Ġthrough ĠIN LINE FORM 3 Ġwhere ĠIN LINE FORM 4 Ġ. ĠThe Ġproblem Ġof Ġinfer ring Ġcontext Ġis Ġto Ġidentify Ġcontext ĠIN LINE FORM 5 Ġfrom Ġa Ġset Ġof Ġcandidate Ġcontexts ĠIN LINE FORM 6 Ġ. </s> Mod elling ĠContext Ġin ĠForums </s> Context Ġhas Ġbeen Ġused Ġand Ġmod elled Ġin Ġvarious Ġways Ġfor Ġdifferent Ġproblems Ġin Ġdiscussion Ġforums . ĠIn Ġa Ġwork Ġon Ġa Ġclosely Ġrelated Ġproblem Ġof Ġforum Ġthread Ġretrieval ĠB IB REF 2 Ġmodels Ġcontext Ġusing Ġinter - post Ġdiscourse Ġe . g ., ĠQuestion - Answer . ĠB IB REF 3 Ġmodels Ġthe Ġstructural Ġdependencies Ġand Ġrelationships Ġbetween Ġforum Ġposts Ġusing Ġa Ġconditional Ġrandom Ġfield Ġin Ġtheir Ġproblem Ġto Ġinfer Ġthe Ġreply Ġstructure . ĠUnlike ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġcan Ġbe Ġused Ġto Ġmodel Ġany Ġstructural Ġdependency Ġand Ġis , Ġtherefore , Ġmore Ġgeneral . ĠIn Ġthis Ġpaper , Ġwe Ġseek Ġto Ġinfer Ġgeneral Ġdependencies Ġbetween Ġa Ġreply Ġand Ġits Ġprevious Ġcontext Ġwhereas ĠB IB REF 3 Ġinference Ġis Ġlimited Ġto Ġpairs Ġof Ġposts . ĠMore Ġrecently ĠB IB REF 4 Ġproposed Ġa Ġcontext Ġbased Ġmodel Ġwhich Ġfactor ises Ġattention Ġover Ġthreads Ġof Ġdifferent Ġlengths . ĠDifferent ly , Ġwe Ġdo Ġnot Ġmodel Ġlength Ġbut Ġthe Ġcontext Ġbefore Ġa Ġpost . ĠHowever , Ġour Ġattention Ġmodels Ġcater Ġto Ġthreads Ġof Ġall Ġlengths . </s> BI B REF 5 Ġproposed Ġgraph Ġstructured ĠL ST M Ġto Ġmodel Ġthe Ġexplicit Ġreply Ġstructure Ġin ĠReddit Ġforums . ĠOur Ġwork Ġdoes Ġnot Ġassume Ġaccess Ġto Ġsuch Ġa Ġreply Ġstructure Ġbecause Ġ1 ) ĠCour ser a Ġforums Ġdo Ġnot Ġprovide Ġone Ġand Ġ2 ) Ġforum Ġparticipants Ġoften Ġerr Ġby Ġposting Ġtheir Ġreply Ġto Ġa Ġdifferent Ġpost Ġthan Ġthat Ġthey Ġintended . ĠAt Ġthe Ġother Ġend Ġof Ġthe Ġspectrum Ġare Ġdocument Ġclassification Ġmodels Ġthat Ġdo Ġnot Ġassume Ġstructure Ġin Ġthe Ġdocument Ġlayout Ġbut Ġtry Ġto Ġinfer Ġinherent Ġstructure Ġin Ġthe Ġnatural Ġlanguage , Ġviz , Ġwords , Ġsentences , Ġparagraphs Ġand Ġdocuments . ĠHier arch ical Ġattention ĠB IB REF 6 Ġis Ġa Ġwell Ġknow Ġrecent Ġwork Ġthat Ġclass ifies Ġdocuments Ġusing Ġa Ġmulti - level ĠL ST Ms Ġwith Ġattention Ġmechanism Ġto Ġselect Ġimportant Ġunits Ġat Ġeach Ġhierarchical Ġlevel . ĠDifferent ly , Ġwe Ġpropose Ġa Ġhierarchical Ġmodel Ġthat Ġenc odes Ġlayout Ġhierarchy Ġbetween Ġa Ġpost Ġand Ġa Ġthread Ġbut Ġalso Ġinf ers Ġreply Ġstructure Ġusing Ġa Ġattention Ġmechanism Ġsince Ġthe Ġlayout Ġdoes Ġnot Ġreliably Ġencode Ġit . </s> Instruct or ĠIntervention Ġin ĠMO OC Ġforums </s> The Ġproblem Ġof Ġpredicting Ġinstructor Ġintervention Ġin ĠMO OC s Ġwas Ġproposed Ġby ĠB IB REF 0 Ġ. ĠLater ĠB IB REF 7 Ġevaluated Ġbaseline Ġmodels Ġby ĠB IB REF 0 Ġover Ġa Ġlarger Ġcorpus Ġand Ġfound Ġthe Ġresults Ġto Ġvary Ġwidely Ġacross ĠMO OC s . ĠSince Ġthen Ġsubsequent Ġworks Ġhave Ġused Ġsimilar Ġdiverse Ġevaluations Ġon Ġthe Ġsame Ġprediction Ġproblem ĠB IB REF 1 Ġ, ĠB IB REF 8 Ġ. ĠB IB REF 1 Ġproposed Ġmodels Ġwith Ġdiscourse Ġfeatures Ġto Ġenable Ġbetter Ġprediction Ġover Ġunseen ĠMO OC s . ĠB IB REF 8 Ġrecently Ġshowed Ġinterventions Ġon ĠCour ser a Ġforums Ġto Ġbe Ġbiased Ġby Ġthe Ġposition Ġat Ġwhich Ġa Ġthread Ġappears Ġto Ġan Ġinstructor Ġviewing Ġthe Ġforum Ġinterface Ġand Ġproposed Ġmethods Ġfor Ġdeb i ased Ġprediction . </s> While Ġall Ġworks Ġsince ĠB IB REF 0 Ġaddress Ġkey Ġlimitations Ġin Ġthis Ġline Ġof Ġresearch , Ġthey Ġhave Ġnot Ġinvestigated Ġthe Ġrole Ġof Ġstructure Ġand Ġsequence Ġin Ġthe Ġthreaded Ġdiscussion Ġin Ġpredicting Ġinstructor Ġinterventions . ĠB IB REF 0 Ġproposed Ġprob abil istic Ġgraphical Ġmodels Ġto Ġmodel Ġstructure Ġand Ġsequence . ĠThey Ġinferred Ġvocabulary Ġdependent Ġlatent Ġpost Ġcategories Ġto Ġmodel Ġthe Ġthread Ġsequence Ġand Ġinfer Ġstates Ġthat Ġtriggered Ġintervention . ĠTheir Ġmodel , Ġhowever , Ġrequires Ġa Ġhyper param eter Ġfor Ġthe Ġnumber Ġof Ġlatent Ġstates . ĠIt Ġis Ġlikely Ġthat Ġtheir Ġempir ically Ġreported Ġsetting Ġwill Ġnot Ġgeneral ise Ġdue Ġto Ġtheir Ġweak Ġevaluation ĠB IB REF 7 Ġ. ĠIn Ġthis Ġpaper , Ġwe Ġpropose Ġmodels Ġto Ġinfer Ġthe Ġcontext Ġthat Ġtriggers Ġinstructor Ġintervention Ġthat Ġdoes Ġnot Ġrequire Ġcontext Ġlengths Ġto Ġbe Ġset Ġap rior i . ĠAll Ġour Ġproposed Ġmodels Ġgeneral ise Ġover Ġmodelling Ġassumptions Ġmade Ġby ĠB IB REF 0 Ġ. </s> For Ġthe Ġpurpose Ġof Ġcomparison Ġagainst Ġa Ġstate - of - the - art Ġand Ġcompeting Ġbas elines Ġwe Ġchoose ĠB IB REF 7 Ġsince ĠB IB REF 0 Ġ' s Ġsystem Ġand Ġdata Ġare Ġnot Ġavailable Ġfor Ġreplication . </s> Data Ġand ĠPre processing </s> We Ġevaluate Ġour Ġproposed Ġmodels Ġover Ġa Ġcorpus Ġof Ġ12 ĠMO OC Ġiterations Ġ( offer ings ) Ġon ĠCour ser a . org ĠIn Ġpartnership Ġwith ĠCour ser a Ġand Ġin Ġline Ġwith Ġits ĠTerms Ġof ĠService , Ġwe Ġobtained Ġthe Ġdata Ġfor Ġuse Ġin Ġour Ġacademic Ġresearch . ĠFollowing Ġprior Ġwork ĠB IB REF 7 Ġwe Ġevaluate Ġover Ġa Ġdiverse Ġdataset Ġto Ġrepresent ĠMO OC s Ġof Ġvarying Ġsizes , Ġinstructor Ġstyles , Ġinstructor Ġteam Ġsizes Ġand Ġnumber Ġof Ġthreads Ġintervened . ĠWe Ġonly Ġinclude Ġthreads Ġfrom Ġsub - forums Ġon ĠLect ure , ĠHom ework , ĠQu iz Ġand ĠExam . ĠWe Ġalso Ġnormal ise Ġand Ġlabel Ġsub - forums Ġwith Ġother Ġnon - standard Ġnames Ġ( e . g ., ĠAss ign ments Ġinstead Ġof ĠHom ework ) Ġinto Ġof Ġthe Ġfour Ġsaid Ġsub - forums . ĠThread s Ġon Ġgeneral Ġdiscussion , Ġmeet Ġand Ġgreet Ġand Ġother Ġcustom Ġsub - forums Ġfor Ġsocial Ġch itch at Ġare Ġomitted Ġas Ġour Ġfocus Ġis Ġto Ġaid Ġinstructors Ġon Ġintervening Ġon Ġdiscussion Ġon Ġthe Ġsubject Ġmatter . ĠWe Ġalso Ġexclude Ġannouncement Ġthreads Ġand Ġother Ġthreads Ġstarted Ġby Ġinstructors Ġsince Ġthey Ġare Ġnot Ġinterventions . ĠWe Ġpre process Ġeach Ġthread Ġby Ġreplacing ĠURLs , Ġequations Ġand Ġother Ġmathematical Ġform ul ae Ġand Ġreferences Ġto Ġtim est amps Ġin Ġlecture Ġvideos Ġby Ġtokens ĠIN LINE FORM 0 ĠURL ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 ĠM ATH ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 ĠTIM ERE F ĠIN LINE FORM 5 Ġrespectively . ĠWe Ġalso Ġtrunc ate Ġintervened Ġthreads Ġto Ġonly Ġinclude Ġposts Ġbefore Ġthe Ġfirst Ġinstructor Ġpost Ġsince Ġthe Ġinstructor 's Ġand Ġsubsequent Ġposts Ġwill Ġbias Ġthe Ġprediction Ġdue Ġto Ġthe Ġinstructor 's Ġpost . </s> Model </s> The Ġkey Ġinnovation Ġof Ġour Ġwork Ġis Ġto Ġdecom pose Ġthe Ġintervention Ġprediction Ġproblem Ġinto Ġa Ġtwo - stage Ġmodel Ġthat Ġfirst Ġexplicitly Ġtries Ġto Ġdiscover Ġthe Ġproper Ġcontext Ġto Ġwhich Ġa Ġpotential Ġintervention Ġcould Ġbe Ġrep lying Ġto , Ġand Ġthen , Ġpredict Ġthe Ġintervention Ġstatus . ĠThis Ġmodel Ġimplicitly Ġassess es Ġthe Ġimportance Ġ( or Ġurgency ) Ġof Ġthe Ġexisting Ġthread 's Ġcontext Ġto Ġdecide Ġwhether Ġan Ġintervention Ġis Ġnecessary . ĠFor Ġexample Ġin ĠFigure ĠSEC REF 1 Ġ, Ġprior Ġto Ġthe Ġinstructor 's Ġintervention , Ġthe Ġultimate Ġpost Ġ( Post Ġ# 6 ) Ġby ĠStudent Ġ2 Ġalready Ġacknowledged Ġthe ĠOP 's Ġgratitude Ġfor Ġhis Ġanswer . ĠIn Ġthis Ġregard , Ġthe Ġinstructor Ġmay Ġhave Ġdecided Ġto Ġuse Ġthis Ġpoint Ġto Ġsummarize Ġthe Ġentire Ġthread Ġto Ġconsolidate Ġall Ġthe Ġpertinent Ġpositions . ĠHere , Ġwe Ġmight Ġassume Ġthat Ġthe Ġinstructor 's Ġreply Ġtakes Ġthe Ġentire Ġthread Ġ( Posts Ġ# 1 âĢĵ 6 ) Ġas Ġthe Ġcontext Ġfor Ġher Ġreply . </s> This Ġsub problem Ġof Ġinfer ring Ġthe Ġcontext Ġscope Ġis Ġwhere Ġour Ġinnovation Ġcenters Ġon . ĠTo Ġbe Ġclear , Ġin Ġorder Ġto Ġmake Ġthe Ġprediction Ġthat Ġa Ġinstruction Ġintervention Ġis Ġnow Ġnecessary Ġon Ġa Ġthread , Ġthe Ġinstructor 's Ġreply Ġis Ġnot Ġyet Ġavailable ĠâĢĶ Ġthe Ġmodel Ġpredicts Ġwhether Ġa Ġreply Ġis Ġnecessary ĠâĢĶ Ġso Ġin Ġthe Ġexample , Ġonly ĠPosts Ġ# 1 âĢĵ 6 Ġare Ġavailable Ġin Ġthe Ġproblem Ġsetting . ĠTo Ġinfer Ġthe Ġcontext , Ġwe Ġhave Ġto Ġdecide Ġwhich Ġsubsequ ence Ġof Ġposts Ġare Ġthe Ġmost Ġplausible Ġmotivation Ġfor Ġan Ġintervention . </s> Recent Ġwork Ġin Ġdeep Ġneural Ġmodeling Ġhas Ġused Ġan Ġattention Ġmechanism Ġas Ġa Ġfocusing Ġquery Ġto Ġhighlight Ġspecific Ġitems Ġwithin Ġthe Ġinput Ġhistory Ġthat Ġsignificantly Ġinfluence Ġthe Ġcurrent Ġdecision Ġpoint . ĠOur Ġwork Ġemploys Ġthis Ġmechanism ĠâĢĵ Ġbut Ġwith Ġa Ġtwist : Ġdue Ġto Ġthe Ġfact Ġthat Ġthe Ġactual Ġinstructor Ġintervention Ġis Ġnot Ġ( yet ) Ġavailable Ġat Ġthe Ġdecision Ġtiming , Ġwe Ġcannot Ġuse Ġany Ġactual Ġintervention Ġto Ġdecide Ġthe Ġcontext . ĠTo Ġemploy Ġattention , Ġwe Ġmust Ġthen Ġemploy Ġa Ġsurrogate Ġtext Ġas Ġthe Ġquery Ġto Ġtrain Ġour Ġprediction Ġmodel . ĠOur Ġmodel Ġvariants Ġmodel Ġassess Ġthe Ġsuit ability Ġof Ġsuch Ġsurrogate Ġtexts Ġfor Ġthe Ġattention Ġmechanism Ġbasis . </s> Cong ru ent Ġwith Ġthe Ġrepresentation Ġof Ġthe Ġinput Ġforums , Ġin Ġall Ġour Ġproposed Ġmodels , Ġwe Ġencode Ġthe Ġdiscussion Ġthread Ġhierarch ically . ĠWe Ġfirst Ġbuild Ġrepresentations Ġfor Ġeach Ġpost Ġby Ġpassing Ġpre - trained Ġword Ġvector Ġrepresentations Ġfrom ĠGl o Ve ĠB IB REF 9 Ġfor Ġeach Ġword Ġthrough Ġan ĠL ST M ĠB IB REF 10 Ġ, ĠIN LINE FORM 0 Ġ. ĠWe Ġuse Ġthe Ġlast Ġlayer Ġoutput Ġof Ġthe ĠL ST M Ġas Ġa Ġrepresentation Ġof Ġthe Ġpost . ĠWe Ġrefer Ġthis Ġas Ġthe Ġpost Ġvector ĠIN LINE FORM 1 Ġ. </s> Then Ġeach Ġpost ĠIN LINE FORM 0 Ġis Ġpassed Ġthrough Ġanother ĠL ST M , ĠIN LINE FORM 1 Ġ, Ġwhose Ġlast Ġlayer Ġoutput Ġforms Ġthe Ġencoding Ġof Ġthe Ġentire Ġthread . ĠHidden Ġunit Ġoutputs Ġof ĠIN LINE FORM 2 Ġrepresent Ġthe Ġcontexts ĠIN LINE FORM 3 Ġ; Ġthat Ġis , Ġsnapshots Ġof Ġthe Ġthreads Ġafter Ġeach Ġpost , Ġas Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ. </s> The ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġtogether Ġconstitute Ġthe Ġhierarchical ĠL ST M Ġ( h L ST M ) Ġmodel . ĠThis Ġgeneral Ġh L ST M Ġmodel Ġserves Ġas Ġthe Ġbasis Ġfor Ġour Ġmodel Ġexploration Ġin Ġthe Ġrest Ġof Ġthis Ġsection . </s> Context ual ĠAttention ĠModels </s> When Ġthey Ġintervene , Ġinstructors Ġeither Ġpay Ġattention Ġto Ġa Ġspecific Ġpost Ġor Ġa Ġseries Ġof Ġposts , Ġwhich Ġtrigger Ġtheir Ġreply . ĠHowever , Ġinstructors Ġrarely Ġexplicitly Ġindicate Ġto Ġwhich Ġpost ( s ) Ġtheir Ġintervention Ġis Ġin Ġrelation Ġto . ĠThis Ġis Ġthe Ġcase Ġin Ġour Ġcorpus , Ġparty Ġdue Ġto ĠCour ser a 's Ġuser Ġinterface Ġwhich Ġonly Ġallows Ġfor Ġsingle Ġlevel Ġcomments Ġ( see ĠFigure ĠFIG REF 2 Ġ). ĠBased Ġsolely Ġon Ġthe Ġbinary , Ġthread - level Ġintervention Ġsignal , Ġour Ġsecondary Ġobjective Ġseeks Ġto Ġinfer Ġthe Ġappropriate Ġcontext ĠâĢĵ Ġrepresented Ġby Ġa Ġsequence Ġof Ġposts ĠâĢĵ Ġas Ġthe Ġbasis Ġfor Ġthe Ġintervention . </s> We Ġonly Ġconsider Ġlinear Ġcontiguous Ġseries Ġof Ġposts Ġstarting Ġwith Ġthe Ġthread 's Ġoriginal Ġpost Ġto Ġconstitute Ġto Ġa Ġcontext ; Ġe . g ., ĠIN LINE FORM 0 Ġ. ĠThis Ġis Ġa Ġreasonable Ġas ĠMO OC Ġforum Ġposts Ġalways Ġreply Ġto Ġthe Ġoriginal Ġpost Ġor Ġto Ġa Ġsubsequent Ġpost , Ġwhich Ġin Ġturn Ġreplies Ġto Ġthe Ġoriginal Ġpost . ĠThis Ġis Ġin Ġcontrast Ġto Ġforums Ġsuch Ġas ĠReddit Ġthat Ġhave Ġa Ġtree Ġor Ġgraph - like Ġstructure Ġthat Ġrequire Ġforum Ġstructure Ġto Ġbe Ġmod elled Ġexplicitly , Ġsuch Ġas Ġin ĠB IB REF 5 Ġ. </s> We Ġpropose Ġthree Ġneural Ġattention ĠB IB REF 11 Ġvariants Ġbased Ġon Ġhow Ġan Ġinstructor Ġmight Ġattend Ġand Ġreply Ġto Ġa Ġcontext Ġin Ġa Ġthread : Ġthe Ġultimate , Ġpen ultimate Ġand Ġany Ġpost Ġattention Ġmodels . ĠWe Ġreview Ġeach Ġof Ġthese Ġin Ġturn . </s> Ultimate ĠPost ĠAttention Ġ( U PA ) ĠModel . ĠIn Ġthis Ġmodel Ġwe Ġattend Ġto Ġthe Ġcontext Ġrepresented Ġby Ġhidden Ġstate Ġof Ġthe ĠIN LINE FORM 0 Ġ. ĠWe Ġuse Ġthe Ġpost Ġprior Ġto Ġthe Ġinstructor 's Ġreply Ġas Ġa Ġquery Ġover Ġthe Ġcontexts ĠIN LINE FORM 1 Ġto Ġcompute Ġattention Ġweights ĠIN LINE FORM 2 Ġ, Ġwhich Ġare Ġthen Ġused Ġto Ġcompute Ġthe Ġattended Ġcontext Ġrepresentation ĠIN LINE FORM 3 Ġ( rec all Ġagain Ġthat Ġthe Ġintervention Ġtext Ġitself Ġis Ġnot Ġavailable Ġfor Ġthis Ġpurpose ). ĠThis Ġattention Ġformulation Ġmakes Ġan Ġequival ence Ġbetween Ġthe Ġfinal ĠIN LINE FORM 4 Ġpost Ġand Ġthe Ġprospective Ġintervention , Ġusing ĠPost ĠIN LINE FORM 5 Ġas Ġthe Ġquery Ġfor Ġfinding Ġthe Ġappropriate Ġcontext ĠIN LINE FORM 6 Ġ, Ġinclusive Ġof Ġitself ĠIN LINE FORM 7 Ġ. ĠSaid Ġin Ġanother Ġway , ĠU PA Ġuses Ġthe Ġmost Ġrecent Ġcontent Ġin Ġthe Ġthread Ġas Ġthe Ġattention al Ġquery Ġfor Ġcontext . </s> For Ġexample , Ġif Ġpost ĠIN LINE FORM 0 Ġis Ġthe Ġinstructor 's Ġreply , Ġpost ĠIN LINE FORM 1 Ġwill Ġquery Ġover Ġthe Ġcontexts ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠThe Ġmodel Ġschematic Ġis Ġshown Ġin ĠFigure ĠFIG REF 12 Ġ. </s> The Ġattended Ġcontext Ġrepresentations Ġare Ġcomputed Ġas : ĠDIS PLAY FORM 0 </s> The ĠIN LINE FORM 0 Ġrepresentation Ġis Ġthen Ġpassed Ġthrough Ġa Ġfully Ġconnected Ġsoft max Ġlayer Ġto Ġyield Ġthe Ġbinary Ġprediction . </s> Pen ultimate ĠPost ĠAttention Ġ( P PA ) ĠModel . ĠWhile Ġthe ĠU PA Ġmodel Ġuses Ġthe Ġmost Ġrecent Ġtext Ġand Ġmakes Ġthe Ġultimate Ġpost Ġitself Ġavailable Ġas Ġpotential Ġcontext , Ġour Ġthe Ġultimate Ġpost Ġmay Ġbe Ġbetter Ġmodeled Ġas Ġhaving Ġany Ġof Ġits Ġprior Ġposts Ġas Ġpotential Ġcontext . ĠPen ultimate ĠPost ĠAttention Ġ( P PA ) Ġvariant Ġdoes Ġthis . ĠThe Ġschematic Ġand Ġthe Ġequations Ġfor Ġthe ĠP PA Ġmodel Ġare Ġobtained Ġby Ġsum ming Ġover Ġcontexts ĠIN LINE FORM 0 Ġin ĠEqu ation ĠEQ REF 10 Ġand ĠFigure ĠFIG REF 12 Ġ. ĠWhile Ġwe Ġcould Ġproperly Ġmodel Ġsuch Ġa Ġcontext Ġinference Ġdecision Ġwith Ġany Ġpost ĠIN LINE FORM 1 Ġand Ġprospective Ġcontexts ĠIN LINE FORM 2 Ġ( where ĠIN LINE FORM 3 Ġis Ġa Ġrandom Ġpost ), Ġit Ġmakes Ġsense Ġto Ġuse Ġthe Ġpen ultimate Ġpost , Ġas Ġwe Ġcan Ġmake Ġthe Ġmost Ġinformation Ġavailable Ġto Ġthe Ġmodel Ġfor Ġthe Ġcontext Ġinference . </s> The Ġattended Ġcontext Ġrepresentations Ġare Ġcomputed Ġas : ĠDIS PLAY FORM 0 </s> Any ĠPost ĠAttention Ġ( AP A ) ĠModel . ĠAP A Ġfurther Ġrelax es Ġboth ĠU PA Ġand ĠP PA , Ġallowing ĠAP A Ġto Ġgeneral ize Ġand Ġhypothes ize Ġthat Ġthe Ġprospective Ġinstructor Ġintervention Ġis Ġbased Ġon Ġthe Ġcontext Ġthat Ġany Ġprevious Ġpost ĠIN LINE FORM 0 Ġreplied Ġto . ĠIn Ġthis Ġmodel , Ġeach Ġpost ĠIN LINE FORM 1 Ġis Ġset Ġas Ġa Ġquery Ġto Ġattend Ġto Ġits Ġprevious Ġcontext ĠIN LINE FORM 2 Ġ. ĠFor Ġexample , ĠIN LINE FORM 3 Ġwill Ġattend Ġto ĠIN LINE FORM 4 Ġ. ĠDifferent Ġfrom Ġstandard Ġattention Ġmechanisms , ĠAP A Ġattention Ġweights ĠIN LINE FORM 5 Ġare Ġobtained Ġby Ġnormal ising Ġinteraction Ġmatrix Ġover Ġthe Ġdifferent Ġqueries . </s> In ĠAP A , Ġthe Ġattention Ġcontext ĠIN LINE FORM 0 Ġis Ġcomputed Ġvia : ĠDIS PLAY FORM 0 </s> E val uation </s> The Ġbaseline Ġand Ġthe Ġmodels Ġare Ġevaluated Ġon Ġa Ġcorpus Ġof Ġ12 ĠMO OC Ġdiscussion Ġforums . ĠWe Ġtrain Ġon Ġ80 % Ġof Ġthe Ġtraining Ġdata Ġand Ġreport Ġevaluation Ġresults Ġon Ġthe Ġheld - out Ġ20 % Ġof Ġtest Ġdata . ĠWe Ġreport ĠIN LINE FORM 0 Ġscores Ġon Ġthe Ġpositive Ġclass Ġ( inter ventions ), Ġin Ġline Ġwith Ġprior Ġwork . ĠWe Ġalso Ġargue Ġthat Ġrecall Ġof Ġthe Ġpositive Ġclass Ġis Ġmore Ġimportant Ġthan Ġprecision , Ġsince Ġit Ġis Ġcost lier Ġfor Ġinstructors Ġto Ġmiss Ġintervening Ġon Ġa Ġthread Ġthan Ġspending Ġirrelevant Ġtime Ġintervening Ġon Ġa Ġless Ġcritical Ġthreads Ġdue Ġto Ġfalse Ġpositives . </s> Model Ġhyper p am eter Ġsettings . ĠAll Ġproposed Ġand Ġbaseline Ġneural Ġmodels Ġare Ġtrained Ġusing ĠAdam Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 001 . ĠWe Ġused Ġcross - ent ropy Ġas Ġloss Ġfunction . ĠImport antly Ġwe Ġupdated Ġthe Ġmodel Ġparameters Ġduring Ġtraining Ġafter Ġeach Ġinstance Ġas Ġin Ġvanilla Ġsto ch astic Ġgradient Ġdescent ; Ġthis Ġsetting Ġwas Ġpractical Ġsince Ġdata Ġon Ġmost Ġcourses Ġhad Ġonly Ġa Ġfew Ġhundred Ġinstances Ġenabling Ġconvergence Ġwithin Ġa Ġreasonable Ġtraining Ġtime Ġof Ġa Ġfew Ġhours Ġ( see ĠTable ĠT AB REF 15 Ġ, Ġcolumn Ġ2 ). ĠModels Ġwere Ġtrained Ġfor Ġa Ġsingle Ġepoch Ġas Ġmost Ġof Ġour Ġcourses Ġwith Ġa Ġfew Ġhundred Ġthread Ġconver ged Ġafter Ġa Ġsingle Ġepoch . ĠWe Ġused Ġ300 - dimensional ĠGl o Ve Ġvectors Ġand Ġpermitted Ġthe Ġembed d ings Ġto Ġbe Ġupdated Ġduring Ġthe Ġmodel 's Ġend - to - end Ġtraining . ĠThe Ġhidden Ġdimension Ġsize Ġof Ġboth ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġset Ġto Ġ128 Ġfor Ġall Ġthe Ġmodels . </s> Bas elines . ĠWe Ġcompare Ġour Ġmodels Ġagainst Ġa Ġneural Ġbaseline Ġmodels , Ġhierarchical ĠL ST M Ġ( h L ST M ), Ġwith Ġthe Ġattention Ġab lated Ġbut Ġwith Ġaccess Ġto Ġthe Ġcomplete Ġcontext , Ġand Ġa Ġstrong , Ġopen - s ourced Ġfeature - rich Ġbaseline ĠB IB REF 7 Ġ. ĠWe Ġchoose ĠB IB REF 7 Ġover Ġother Ġprior Ġworks Ġsuch Ġas ĠB IB REF 0 Ġsince Ġwe Ġdo Ġnot Ġhave Ġaccess Ġto Ġthe Ġdataset Ġor Ġthe Ġsystem Ġused Ġin Ġtheir Ġpapers Ġfor Ġreplication . ĠB IB REF 7 Ġis Ġa Ġlog istic Ġregression Ġclass ifier Ġwith Ġfeatures Ġinclusive Ġof Ġbag - of - words Ġrepresentation Ġof Ġthe Ġun ig rams Ġand Ġthread Ġlength , Ġnormal ised Ġcounts Ġof Ġagreements Ġto Ġprevious Ġposts , Ġcounts Ġof Ġnon - lex ical Ġreference Ġitems Ġsuch Ġas ĠURLs , Ġand Ġthe ĠCour ser a Ġforum Ġtype Ġin Ġwhich Ġa Ġthread Ġappeared . ĠWe Ġalso Ġreport Ġaggreg ated Ġresults Ġfrom Ġa Ġh L ST M Ġmodel Ġwith Ġaccess Ġonly Ġto Ġthe Ġlast Ġpost Ġas Ġcontext Ġfor Ġcomparison . ĠTable ĠT AB REF 17 Ġcompares Ġthe Ġperformance Ġof Ġthese Ġbas elines Ġagainst Ġour Ġproposed Ġmethods . </s> Results </s> Table ĠT AB REF 15 Ġshows Ġperformance Ġof Ġall Ġour Ġproposed Ġmodels Ġand Ġthe Ġneural Ġbaseline Ġover Ġour Ġ12 ĠMO OC Ġdataset . ĠOur Ġmodels Ġof ĠU PA , ĠP PA Ġindividually Ġbetter Ġthe Ġbaseline Ġby Ġ5 Ġand Ġ2 % Ġon ĠIN LINE FORM 0 Ġand Ġ3 Ġand Ġ6 % Ġon Ġrecall Ġrespectively . ĠU PA Ġperforms Ġthe Ġbest Ġin Ġterms Ġof ĠIN LINE FORM 1 Ġon Ġaverage Ġwhile ĠP PA Ġperforms Ġthe Ġbest Ġin Ġterms Ġof Ġrecall Ġon Ġaverage . ĠAt Ġthe Ġindividual Ġcourse Ġlevel , Ġhowever , Ġthe Ġresults Ġare Ġmixed . ĠU PA Ġperforms Ġthe Ġbest Ġon ĠIN LINE FORM 2 Ġon Ġ5 Ġout Ġof Ġ12 Ġcourses , ĠP PA Ġon Ġ3 Ġout Ġ12 Ġcourses , ĠAP A Ġ1 Ġout Ġof Ġ12 Ġcourses Ġand Ġthe Ġbaseline Ġh L ST M Ġon Ġ1 . ĠP PA Ġperforms Ġthe Ġbest Ġon Ġrecall Ġon Ġ7 Ġout Ġof Ġthe Ġ12 Ġcourses . ĠWe Ġalso Ġnote Ġthat Ġcourse Ġlevel Ġperformance Ġdifferences Ġcorrelate Ġwith Ġthe Ġcourse Ġsize Ġand Ġintervention Ġratio Ġ( here after , Ġi . rat io ), Ġwhich Ġis Ġthe Ġratio Ġof Ġintervened Ġto Ġnon - inter ven ed Ġthreads . ĠU PA Ġperforms Ġbetter Ġthan ĠP PA Ġand ĠAP A Ġon Ġlow Ġintervention Ġcourses Ġ( i . rat io ĠIN LINE FORM 3 Ġ0 . 25 ) Ġmainly Ġbecause ĠP PA Ġand ĠAP A 's Ġperformance Ġdrops Ġsteep ly Ġwhen Ġi . rat io Ġdrops Ġ( see Ġcol . 2 Ġparent hesis Ġand ĠIN LINE FORM 4 Ġof ĠP PA Ġand ĠAP A ). ĠWhile Ġall Ġthe Ġproposed Ġmodels Ġbeat Ġthe Ġbaseline Ġon Ġevery Ġcourse Ġexcept Ġcase based - 2 . ĠOn Ġmedical ne uro - 2 Ġand Ġcomp ilers - 4 Ġwhich Ġhave Ġthe Ġlowest Ġi . rat io Ġamong Ġthe Ġ12 Ġcourses Ġnone Ġof Ġthe Ġneural Ġmodels Ġbetter Ġthe Ġreported Ġbaseline ĠB IB REF 7 Ġ( course Ġlevel Ġnot Ġscores Ġnot Ġshown Ġin Ġthis Ġpaper ). ĠThe Ġeffect Ġis Ġpronounced Ġin Ġcomp ilers - 4 Ġcourse Ġwhere Ġnone Ġof Ġthe Ġneural Ġmodels Ġwere Ġable Ġto Ġpredict Ġany Ġintervened Ġthreads . ĠThis Ġis Ġdue Ġto Ġthe Ġinherent Ġweakness Ġof Ġstandard Ġneural Ġmodels , Ġwhich Ġare Ġunable Ġto Ġlearn Ġfeatures Ġwell Ġenough Ġwhen Ġfaced Ġwith Ġsparse Ġdata . </s> The Ġbest Ġperformance Ġof ĠU PA Ġindicates Ġthat Ġthe Ġreply Ġcontext Ġof Ġthe Ġinstructor 's Ġpost ĠIN LINE FORM 0 Ġcorrelates Ġstrongly Ġwith Ġthat Ġof Ġthe Ġprevious Ġpost ĠIN LINE FORM 1 Ġ. ĠThis Ġis Ġnot Ġsurprising Ġsince Ġnormal Ġconversations Ġare Ġtypically Ġstructured Ġthat Ġway . </s> Discussion </s> In Ġorder Ġto Ġfurther Ġunderstand Ġthe Ġmodels ' Ġability Ġto Ġinfer Ġthe Ġcontext Ġand Ġits Ġeffect Ġon Ġintervention Ġprediction , Ġwe Ġfurther Ġinvestigate Ġthe Ġfollowing Ġresearch Ġquestions . </s> R Q 1 . ĠDoes Ġcontext Ġinference Ġhelp Ġintervention Ġprediction ? </s> In Ġorder Ġto Ġunderstand Ġif Ġcontext Ġinference Ġis Ġuseful Ġto Ġintervention Ġprediction , Ġwe Ġab late Ġthe Ġattention Ġcomponents Ġand Ġexperiment Ġwith Ġthe Ġvanilla Ġhierarchical ĠL ST M Ġmodel . ĠRow Ġ3 Ġof ĠTable ĠT AB REF 17 Ġshows Ġthe Ġmacro Ġaveraged Ġresult Ġfrom Ġthis Ġexperiment . ĠThe ĠU PA Ġand ĠP PA Ġattention Ġmodels Ġbetter Ġthe Ġvanilla Ġh L ST M Ġby Ġ5 % Ġand Ġ2 % Ġon Ġaverage Ġin ĠIN LINE FORM 0 Ġrespectively . ĠRecall Ġthat Ġthe Ġvanilla Ġh L ST M Ġalready Ġhas Ġaccess Ġto Ġa Ġcontext Ġconsisting Ġof Ġall Ġposts Ġ( from ĠIN LINE FORM 1 Ġthrough ĠIN LINE FORM 2 Ġ). ĠIn Ġcontrast , Ġthe ĠU PA Ġand ĠP PA Ġmodels Ġselectively Ġinf ers Ġa Ġcontext Ġfor ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġposts , Ġrespectively , Ġand Ġuse Ġit Ġto Ġpredict Ġintervention . ĠThe Ġimproved Ġperformance Ġof Ġour Ġattention Ġmodels Ġthat Ġactively Ġselect Ġtheir Ġoptimal Ġcontext , Ġover Ġa Ġmodel Ġwith Ġthe Ġcomplete Ġthread Ġas Ġcontext , Ġh L ST M , Ġshows Ġthat Ġthe Ġcontext Ġinference Ġimproves Ġintervention Ġprediction Ġover Ġusing Ġthe Ġdefault Ġfull Ġcontext . </s> R Q 2 . ĠHow Ġwell Ġdo Ġthe Ġmodels Ġperform Ġacross Ġthreads Ġof Ġdifferent Ġlengths ? ĠTo Ġunderstand Ġthe Ġmodels ' Ġprediction Ġperformance Ġacross Ġthreads Ġof Ġdifferent Ġlengths , Ġwe Ġbin Ġthreads Ġby Ġlength Ġand Ġstudy Ġthe Ġmodels ' Ġrecall . ĠWe Ġchoose Ġthree Ġcourses , Ġml - 5 , Ġr pro g - 3 Ġand Ġcalc - 1 , Ġfrom Ġour Ġcorpus Ġof Ġ12 Ġwith Ġthe Ġhighest Ġnumber Ġof Ġpositive Ġinstances Ġ( ĠIN LINE FORM 0 Ġ100 Ġthreads ). ĠWe Ġlimit Ġour Ġanalysis Ġto Ġthese Ġsince Ġbin ning Ġrenders Ġcourses Ġwith Ġfewer Ġpositive Ġinstances Ġsparse . ĠFigure ĠFIG REF 18 Ġshows Ġperformance Ġacross Ġthread Ġlengths Ġfrom Ġ1 Ġthrough Ġ7 Ġposts Ġand ĠIN LINE FORM 1 Ġposts . ĠClearly , Ġthe ĠU PA Ġmodel Ġperforms Ġmuch Ġbetter Ġon Ġshorter Ġthreads Ġthan Ġon Ġlonger Ġthreads Ġwhile ĠP PA Ġand ĠAP A Ġworks Ġbetter Ġon Ġlonger Ġthreads . ĠAlthough , ĠU PA Ġis Ġthe Ġbest Ġperforming Ġmodel Ġin Ġterms Ġof Ġoverall ĠIN LINE FORM 2 Ġits Ġperformance Ġdrops Ġsteep ly Ġon Ġthreads Ġof Ġlength ĠIN LINE FORM 3 Ġ. ĠU PA 's Ġoverall Ġbest Ġperformance Ġis Ġbecause Ġmost Ġof Ġthe Ġinterventions Ġin Ġthe Ġcorpus Ġhappen Ġafter Ġone Ġpost . ĠTo Ġhighlight Ġthe Ġperformance Ġof ĠAP A Ġwe Ġshow Ġan Ġexample Ġfrom Ġsm ac - 1 Ġin ĠFigure ĠFIG REF 22 Ġwith Ġnine Ġposts Ġwhich Ġwas Ġpredicted Ġcorrectly Ġas Ġintervened Ġby ĠAP A Ġbut Ġnot Ġby Ġother Ġmodels . ĠThread s Ġshows Ġstudents Ġconfused Ġover Ġa Ġmissing Ġfigure Ġin Ġa Ġhomework . ĠThe Ġinstructor Ġfinally Ġshows Ġup , Ġthough Ġlate , Ġto Ġresolve Ġthe Ġconfusion . </s> R Q 3 . ĠDo Ġmodels Ġtrained Ġwith Ġdifferent Ġcontext Ġlengths Ġperform Ġbetter Ġthan Ġwhen Ġtrained Ġon Ġa Ġsingle Ġcontext Ġlength ? </s> We Ġfind Ġthat Ġcontext Ġlength Ġhas Ġa Ġregular ising Ġeffect Ġon Ġthe Ġmodel 's Ġperformance Ġat Ġtest Ġtime . ĠThis Ġis Ġnot Ġsurprising Ġsince Ġmodels Ġtrained Ġwith Ġthreads Ġof Ġsingle Ġcontext Ġlength Ġwill Ġnot Ġgeneral ise Ġto Ġinfer Ġdifferent Ġcontext Ġlengths . ĠRow Ġ4 Ġof ĠTable ĠT AB REF 17 Ġshows Ġa Ġsteep Ġperformance Ġdrop Ġin Ġtraining Ġby Ġclass ifier Ġwith Ġall Ġthreads Ġtrunc ated Ġto Ġa Ġcontext Ġof Ġjust Ġone Ġpost , ĠIN LINE FORM 0 Ġ, Ġthe Ġpost Ġimmediately Ġpreceding Ġthe Ġintervened Ġpost . ĠWe Ġalso Ġconducted Ġan Ġexperiment Ġwith Ġa Ġmulti - object ive Ġloss Ġfunction Ġwith Ġan Ġadditive Ġcross - ent ropy Ġterm Ġwhere Ġeach Ġterm Ġcomp utes Ġloss Ġfrom Ġa Ġmodel Ġwith Ġcontext Ġlimited Ġto Ġa Ġlength Ġof Ġ3 . ĠWe Ġchose Ġ3 Ġsince Ġintervened Ġthreads Ġin Ġall Ġthe Ġcourses Ġhad Ġa Ġmedian Ġlength Ġbetween Ġ3 Ġand Ġ4 . ĠWe Ġachieved Ġan ĠIN LINE FORM 1 Ġof Ġ0 . 45 Ġwith Ġa Ġprecision Ġof Ġ0 . 47 Ġand Ġrecall Ġof Ġ0 . 43 . ĠThis Ġachieves Ġa Ġperformance Ġcomparable Ġto Ġthat Ġof Ġthe ĠB IB REF 7 Ġwith Ġcontext Ġlength Ġset Ġto Ġonly Ġto Ġ3 . ĠThis Ġapproach Ġof Ġusing Ġinfinitely Ġmany Ġloss Ġterms Ġfor Ġeach Ġcontext Ġlength Ġfrom Ġ1 Ġthrough Ġthe Ġmaximum Ġthread Ġlength Ġin Ġa Ġcourse Ġis Ġnaive Ġand Ġnot Ġpractical . ĠWe Ġonly Ġuse Ġthis Ġmodel Ġto Ġshow Ġthe Ġimportance Ġof Ġtraining Ġthe Ġmodel Ġwith Ġloss Ġfrom Ġthreads Ġof Ġdifferent Ġlengths Ġto Ġprevent Ġmodels Ġover fitting Ġto Ġthreads Ġof Ġspecific Ġcontext Ġlengths . </s> Conclusion </s> We Ġpredict Ġinstructor Ġintervention Ġon Ġstudent Ġdiscussions Ġby Ġfirst Ġinfer ring Ġthe Ġoptimal Ġsize Ġof Ġthe Ġcontext Ġneeded Ġto Ġdecide Ġon Ġthe Ġintervention Ġdecision Ġfor Ġthe Ġintervened Ġpost . ĠWe Ġfirst Ġshow Ġthat Ġa Ġstructured Ġrepresentation Ġof Ġthe Ġcomplete Ġthread Ġas Ġthe Ġcontext Ġis Ġbetter Ġthan Ġa Ġbag - of - words , Ġfeature - rich Ġrepresentation . ĠWe Ġthen Ġpropose Ġattention - based Ġmodels Ġto Ġinfer Ġand Ġselect Ġa Ġcontext ĠâĢĵ Ġdefined Ġas Ġa Ġcontiguous Ġsubsequ ence Ġof Ġstudent Ġposts ĠâĢĵ Ġto Ġimprove Ġover Ġa Ġmodel Ġthat Ġalways Ġtakes Ġthe Ġcomplete Ġthread Ġas Ġa Ġcontext Ġto Ġprediction Ġintervention . ĠOur ĠAny ĠPost ĠAttention Ġ( AP A ) Ġmodel Ġenables Ġinstructors Ġto Ġtune Ġthe Ġmodel Ġto Ġpredict Ġintervention Ġearly Ġor Ġlate . ĠWe Ġposit Ġour ĠAP A Ġmodel Ġwill Ġenable ĠMO OC Ġinstructors Ġemploying Ġvarying Ġped ag ogical Ġstyles Ġto Ġuse Ġthe Ġmodel Ġequally Ġwell . ĠWe Ġintro spect Ġthe Ġattention Ġmodels ' Ġperformance Ġacross Ġthreads Ġof Ġvarying Ġlengths Ġand Ġshow Ġthat ĠAP A Ġpredicts Ġintervention Ġon Ġlonger Ġthreads , Ġwhich Ġpossesses Ġmore Ġcandidate Ġcontexts , Ġbetter . </s> We Ġnote Ġthat Ġthe Ġrecall Ġof Ġthe Ġpredictive Ġmodels Ġfor Ġlonger Ġthreads Ġ( that Ġis , Ġthreads Ġof Ġlength Ġgreater Ġ2 ) Ġcan Ġstill Ġbe Ġimproved . ĠModels Ġperform Ġdifferently Ġbetween Ġshorter Ġand Ġlonger Ġlength . ĠAn Ġensemble Ġmodel Ġor Ġa Ġmulti - object ive Ġloss Ġfunction Ġis Ġthus Ġplanned Ġin Ġour Ġfuture Ġwork Ġto Ġbetter Ġprediction Ġon Ġsuch Ġlonger Ġthreads .