By involving humans for post-hoc evaluation of model's interpretability
ABSTRACTIVE
directly solicits informative keywords from the crowd for model training, thereby providing human-understandable explanations for the improved model
EXTRACTIVE
Human-in-the-Loop Approaches. Our work extends weakly supervised learning methods by involving humans in the loop BIBREF13. Existing human-in-the-loop approaches mainly leverage crowds to label individual data instances BIBREF9, BIBREF10 or to debug the training data BIBREF30, BIBREF31 or components BIBREF32, BIBREF33, BIBREF34 of a machine learning system. Unlike these works, we leverage crowd workers to label sampled microposts in order to obtain keyword-specific expectations, which can then be generalized to help classify microposts containing the same keyword, thus amplifying the utility of the crowd. Our work is further connected to the topic of interpretability and transparency of machine learning models BIBREF11, BIBREF35, BIBREF12, for which humans are increasingly involved, for instance for post-hoc evaluations of the model's interpretability. In contrast, our approach directly solicits informative keywords from the crowd for model training, thereby providing human-understandable explanations for the improved model.
Human-in-the-Loop Approaches. Our work extends weakly supervised learning methods by involving humans in the loop BIBREF13. Existing human-in-the-loop approaches mainly leverage crowds to label individual data instances BIBREF9, BIBREF10 or to debug the training data BIBREF30, BIBREF31 or components BIBREF32, BIBREF33, BIBREF34 of a machine learning system. Unlike these works, we leverage crowd workers to label sampled microposts in order to obtain keyword-specific expectations, which can then be generalized to help classify microposts containing the same keyword, thus amplifying the utility of the crowd. Our work is further connected to the topic of interpretability and transparency of machine learning models BIBREF11, BIBREF35, BIBREF12, for which humans are increasingly involved, for instance for post-hoc evaluations of the model's interpretability. In contrast, our approach directly solicits informative keywords from the crowd for model training, thereby providing human-understandable explanations for the improved model.
1912.00667
Introduction </s> Event Ġdetection Ġon Ġmicro blog ging Ġplatforms Ġsuch Ġas ĠTwitter Ġaims Ġto Ġdetect Ġevents Ġpreempt ively . ĠA Ġmain Ġtask Ġin Ġevent Ġdetection Ġis Ġdetecting Ġevents Ġof Ġpredetermined Ġtypes ĠB IB REF 0 , Ġsuch Ġas Ġconcerts Ġor Ġcontroversial Ġevents Ġbased Ġon Ġmic rop ost s Ġmatching Ġspecific Ġevent Ġdescriptions . ĠThis Ġtask Ġhas Ġextensive Ġapplications Ġranging Ġfrom Ġcyber Ġsecurity ĠB IB REF 1 , ĠB IB REF 2 Ġto Ġpolitical Ġelections ĠB IB REF 3 Ġor Ġpublic Ġhealth ĠB IB REF 4 , ĠB IB REF 5 . ĠDue Ġto Ġthe Ġhigh Ġambiguity Ġand Ġinconsistency Ġof Ġthe Ġterms Ġused Ġin Ġmic rop ost s , Ġevent Ġdetection Ġis Ġgenerally Ġperformed Ġthough Ġstatistical Ġmachine Ġlearning Ġmodels , Ġwhich Ġrequire Ġa Ġlabeled Ġdataset Ġfor Ġmodel Ġtraining . ĠData Ġlabeling Ġis , Ġhowever , Ġa Ġlong , Ġlabor ious , Ġand Ġusually Ġcostly Ġprocess . ĠFor Ġthe Ġcase Ġof Ġmic rop ost Ġclassification , Ġthough Ġpositive Ġlabels Ġcan Ġbe Ġcollected Ġ( e . g ., Ġusing Ġspecific Ġhasht ags , Ġor Ġevent - related Ġdate - time Ġinformation ), Ġthere Ġis Ġno Ġstraightforward Ġway Ġto Ġgenerate Ġnegative Ġlabels Ġuseful Ġfor Ġmodel Ġtraining . ĠTo Ġtackle Ġthis Ġlack Ġof Ġnegative Ġlabels Ġand Ġthe Ġsignificant Ġmanual Ġefforts Ġin Ġdata Ġlabeling , ĠB IB REF 1 Ġ( BI B REF 1 , ĠB IB REF 3 ) Ġintroduced Ġa Ġweak Ġsupervision Ġbased Ġlearning Ġapproach , Ġwhich Ġuses Ġonly Ġpositively Ġlabeled Ġdata , Ġaccompanied Ġby Ġunl abel ed Ġexamples Ġby Ġfiltering Ġmic rop ost s Ġthat Ġcontain Ġa Ġcertain Ġkeyword Ġindicative Ġof Ġthe Ġevent Ġtype Ġunder Ġconsideration Ġ( e . g ., Ġ` hack ' Ġfor Ġcyber Ġsecurity ). ĠAnother Ġkey Ġtechnique Ġin Ġthis Ġcontext Ġis Ġexpectation Ġregular ization ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 1 . ĠHere , Ġthe Ġestimated Ġproportion Ġof Ġrelevant Ġmic rop ost s Ġin Ġan Ġunl abel ed Ġdataset Ġcontaining Ġa Ġkeyword Ġis Ġgiven Ġas Ġa Ġkeyword - specific Ġexpectation . ĠThis Ġexpectation Ġis Ġused Ġin Ġthe Ġregular ization Ġterm Ġof Ġthe Ġmodel 's Ġobjective Ġfunction Ġto Ġconst rain Ġthe Ġposterior Ġdistribution Ġof Ġthe Ġmodel Ġpredictions . ĠBy Ġdoing Ġso , Ġthe Ġmodel Ġis Ġtrained Ġwith Ġan Ġexpectation Ġon Ġits Ġprediction Ġfor Ġmic rop ost s Ġthat Ġcontain Ġthe Ġkeyword . ĠSuch Ġa Ġmethod , Ġhowever , Ġsuffers Ġfrom Ġtwo Ġkey Ġproblems : </s> Due Ġto Ġthe Ġunpredict ability Ġof Ġevent Ġoccurrences Ġand Ġthe Ġconstantly Ġchanging Ġdynamics Ġof Ġusers ' Ġposting Ġfrequency ĠB IB REF 8 , Ġestimating Ġthe Ġexpectation Ġassociated Ġwith Ġa Ġkeyword Ġis Ġa Ġchallenging Ġtask , Ġeven Ġfor Ġdomain Ġexperts ; </s> The Ġperformance Ġof Ġthe Ġevent Ġdetection Ġmodel Ġis Ġconstrained Ġby Ġthe Ġinform at iveness Ġof Ġthe Ġkeyword Ġused Ġfor Ġmodel Ġtraining . ĠAs Ġof Ġnow , Ġwe Ġlack Ġa Ġprincipled Ġmethod Ġfor Ġdiscovering Ġnew Ġkeywords Ġand Ġimprove Ġthe Ġmodel Ġperformance . </s> To Ġaddress Ġthe Ġabove Ġissues , Ġwe Ġadvocate Ġa Ġhuman - AI Ġloop Ġapproach Ġfor Ġdiscovering Ġinformative Ġkeywords Ġand Ġestimating Ġtheir Ġexpectations Ġreliably . ĠOur Ġapproach Ġiter atively Ġlever ages Ġ1 ) Ġcrowd Ġworkers Ġfor Ġestimating Ġkeyword - specific Ġexpectations , Ġand Ġ2 ) Ġthe Ġdisagreement Ġbetween Ġthe Ġmodel Ġand Ġthe Ġcrowd Ġfor Ġdiscovering Ġnew Ġinformative Ġkeywords . ĠMore Ġspecifically , Ġat Ġeach Ġiteration Ġafter Ġwe Ġobtain Ġa Ġkeyword - specific Ġexpectation Ġfrom Ġthe Ġcrowd , Ġwe Ġtrain Ġthe Ġmodel Ġusing Ġexpectation Ġregular ization Ġand Ġselect Ġthose Ġkeyword - related Ġmic rop ost s Ġfor Ġwhich Ġthe Ġmodel 's Ġprediction Ġdisagrees Ġthe Ġmost Ġwith Ġthe Ġcrowd 's Ġexpectation ; Ġsuch Ġmic rop ost s Ġare Ġthen Ġpresented Ġto Ġthe Ġcrowd Ġto Ġidentify Ġnew Ġkeywords Ġthat Ġbest Ġexplain Ġthe Ġdisagreement . ĠBy Ġdoing Ġso , Ġour Ġapproach Ġidentifies Ġnew Ġkeywords Ġwhich Ġconvey Ġmore Ġrelevant Ġinformation Ġwith Ġrespect Ġto Ġexisting Ġones , Ġthus Ġeffectively Ġboosting Ġmodel Ġperformance . ĠBy Ġexploiting Ġthe Ġdisagreement Ġbetween Ġthe Ġmodel Ġand Ġthe Ġcrowd , Ġour Ġapproach Ġcan Ġmake Ġefficient Ġuse Ġof Ġthe Ġcrowd , Ġwhich Ġis Ġof Ġcritical Ġimportance Ġin Ġa Ġhuman - in - the - loop Ġcontext ĠB IB REF 9 , ĠB IB REF 10 . ĠAn Ġadditional Ġadvantage Ġof Ġour Ġapproach Ġis Ġthat Ġby Ġobtaining Ġnew Ġkeywords Ġthat Ġimprove Ġmodel Ġperformance Ġover Ġtime , Ġwe Ġare Ġable Ġto Ġgain Ġinsight Ġinto Ġhow Ġthe Ġmodel Ġlearns Ġfor Ġspecific Ġevent Ġdetection Ġtasks . ĠSuch Ġan Ġadvantage Ġis Ġparticularly Ġuseful Ġfor Ġevent Ġdetection Ġusing Ġcomplex Ġmodels , Ġe . g ., Ġdeep Ġneural Ġnetworks , Ġwhich Ġare Ġintrinsically Ġhard Ġto Ġunderstand ĠB IB REF 11 , ĠB IB REF 12 . ĠAn Ġadditional Ġchallenge Ġin Ġinvolving Ġcrowd Ġworkers Ġis Ġthat Ġtheir Ġcontributions Ġare Ġnot Ġfully Ġreliable ĠB IB REF 13 . ĠIn Ġthe Ġcrowds ourcing Ġliterature , Ġthis Ġproblem Ġis Ġusually Ġtackled Ġwith Ġprob abil istic Ġlatent Ġvariable Ġmodels ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 , Ġwhich Ġare Ġused Ġto Ġperform Ġtruth Ġinference Ġby Ġaggreg ating Ġa Ġredundant Ġset Ġof Ġcrowd Ġcontributions . ĠOur Ġhuman - AI Ġloop Ġapproach Ġimproves Ġthe Ġinference Ġof Ġkeyword Ġexpectation Ġby Ġaggreg ating Ġcontributions Ġnot Ġonly Ġfrom Ġthe Ġcrowd Ġbut Ġalso Ġfrom Ġthe Ġmodel . ĠThis , Ġhowever , Ġcomes Ġwith Ġits Ġown Ġchallenge Ġas Ġthe Ġmodel 's Ġpredictions Ġare Ġfurther Ġdependent Ġon Ġthe Ġresults Ġof Ġexpectation Ġinference , Ġwhich Ġis Ġused Ġfor Ġmodel Ġtraining . ĠTo Ġaddress Ġthis Ġproblem , Ġwe Ġintroduce Ġa Ġunified Ġprob abil istic Ġmodel Ġthat Ġseamlessly Ġintegrates Ġexpectation Ġinference Ġand Ġmodel Ġtraining , Ġthereby Ġallowing Ġthe Ġformer Ġto Ġbenefit Ġfrom Ġthe Ġlatter Ġwhile Ġresolving Ġthe Ġinter - depend ency Ġbetween Ġthe Ġtwo . </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwe Ġare Ġthe Ġfirst Ġto Ġpropose Ġa Ġhuman - AI Ġloop Ġapproach Ġthat Ġiter atively Ġimproves Ġmachine Ġlearning Ġmodels Ġfor Ġevent Ġdetection . ĠIn Ġsummary , Ġour Ġwork Ġmakes Ġthe Ġfollowing Ġkey Ġcontributions : </s> A Ġnovel Ġhuman - AI Ġloop Ġapproach Ġfor Ġmic rop ost Ġevent Ġdetection Ġthat Ġjointly Ġdiscovers Ġinformative Ġkeywords Ġand Ġestimates Ġtheir Ġexpectation ; </s> A Ġunified Ġprob abil istic Ġmodel Ġthat Ġinf ers Ġkeyword Ġexpectation Ġand Ġsimultaneously Ġperforms Ġmodel Ġtraining ; </s> An Ġextensive Ġempirical Ġevaluation Ġof Ġour Ġapproach Ġon Ġmultiple Ġreal - world Ġdatasets Ġdemonstrating Ġthat Ġour Ġapproach Ġsignificantly Ġimproves Ġthe Ġstate Ġof Ġthe Ġart Ġby Ġan Ġaverage Ġof Ġ24 . 3 % ĠA UC . </s> The Ġrest Ġof Ġthis Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠFirst , Ġwe Ġpresent Ġour Ġhuman - AI Ġloop Ġapproach Ġin ĠSection ĠSEC REF 2 . ĠSub sequently , Ġwe Ġintroduce Ġour Ġproposed Ġprob abil istic Ġmodel Ġin ĠSection ĠSEC REF 3 . ĠThe Ġexperimental Ġsetup Ġand Ġresults Ġare Ġpresented Ġin ĠSection ĠSEC REF 4 . ĠFinally , Ġwe Ġbriefly Ġcover Ġrelated Ġwork Ġin ĠSection ĠSEC REF 5 Ġbefore Ġconcluding Ġour Ġwork Ġin ĠSection ĠSEC REF 6 . </s> The ĠHuman - AI ĠLoop ĠApproach </s> Given Ġa Ġset Ġof Ġlabeled Ġand Ġunl abel ed Ġmic rop ost s , Ġour Ġgoal Ġis Ġto Ġextract Ġinformative Ġkeywords Ġand Ġestimate Ġtheir Ġexpectations Ġin Ġorder Ġto Ġtrain Ġa Ġmachine Ġlearning Ġmodel . ĠTo Ġachieve Ġthis Ġgoal , Ġour Ġproposed Ġhuman - AI Ġloop Ġapproach Ġcomprises Ġtwo Ġcrowds ourcing Ġtasks , Ġi . e ., Ġmic rop ost Ġclassification Ġfollowed Ġby Ġkeyword Ġdiscovery , Ġand Ġa Ġunified Ġprob abil istic Ġmodel Ġfor Ġexpectation Ġinference Ġand Ġmodel Ġtraining . ĠFigure ĠFIG REF 6 Ġpresents Ġan Ġoverview Ġof Ġour Ġapproach . ĠNext , Ġwe Ġdescribe Ġour Ġapproach Ġfrom Ġa Ġprocess - centric Ġperspective . </s> Following Ġprevious Ġstudies ĠB IB REF 1 , ĠB IB REF 17 , ĠB IB REF 2 , Ġwe Ġcollect Ġa Ġset Ġof Ġunl abel ed Ġmic rop ost s Ġ$\ math cal Ġ{ U } $ Ġfrom Ġa Ġmicro blog ging Ġplatform Ġand Ġpost - filter , Ġusing Ġan Ġinitial Ġ( set Ġof ) Ġkeyword ( s ), Ġthose Ġmic rop ost s Ġthat Ġare Ġpotentially Ġrelevant Ġto Ġan Ġevent Ġcategory . ĠThen , Ġwe Ġcollect Ġa Ġset Ġof Ġevent - related Ġmic rop ost s Ġ( i . e ., Ġpositively Ġlabeled Ġmic rop ost s ) Ġ$\ math cal Ġ{ L } $, Ġpost - fil tering Ġwith Ġa Ġlist Ġof Ġseed Ġevents . Ġ$\ math cal Ġ{ U } $ Ġand Ġ$\ math cal Ġ{ L } $ Ġare Ġused Ġtogether Ġto Ġtrain Ġa Ġdiscrim in ative Ġmodel Ġ( e . g ., Ġa Ġdeep Ġneural Ġnetwork ) Ġfor Ġclass ifying Ġthe Ġrelevance Ġof Ġmic rop ost s Ġto Ġan Ġevent . ĠWe Ġdenote Ġthe Ġtarget Ġmodel Ġas Ġ$ p _ \ the ta Ġ( y | x ) $, Ġwhere Ġ$\ the ta Ġ$ Ġis Ġthe Ġmodel Ġparameter Ġto Ġbe Ġlearned Ġand Ġ$ y $ Ġis Ġthe Ġlabel Ġof Ġan Ġarbitrary Ġmic rop ost , Ġrepresented Ġby Ġa Ġbag - of - words Ġvector Ġ$ x $. ĠOur Ġapproach Ġiter ates Ġseveral Ġtimes Ġ$ t = \ l brace Ġ1 , Ġ2 , Ġ\ ld ots Ġ\ r brace Ġ$ Ġuntil Ġthe Ġperformance Ġof Ġthe Ġtarget Ġmodel Ġconver ges . ĠEach Ġiteration Ġstarts Ġfrom Ġthe Ġinitial Ġkeyword ( s ) Ġor Ġthe Ġnew Ġkeyword ( s ) Ġdiscovered Ġin Ġthe Ġprevious Ġiteration . ĠGiven Ġsuch Ġa Ġkeyword , Ġden oted Ġby Ġ$ w ^{ ( t )} $, Ġthe Ġiteration Ġstarts Ġby Ġsampling Ġmic rop ost s Ġcontaining Ġthe Ġkeyword Ġfrom Ġ$\ math cal Ġ{ U } $, Ġfollowed Ġby Ġdynamically Ġcreating Ġmic rop ost Ġclassification Ġtasks Ġand Ġpublishing Ġthem Ġon Ġa Ġcrowds ourcing Ġplatform . </s> Mic rop ost ĠClassification . ĠThe Ġmic rop ost Ġclassification Ġtask Ġrequires Ġcrowd Ġworkers Ġto Ġlabel Ġthe Ġselected Ġmic rop ost s Ġinto Ġtwo Ġclasses : Ġevent - related Ġand Ġnon Ġevent - related . ĠIn Ġparticular , Ġworkers Ġare Ġgiven Ġinstructions Ġand Ġexamples Ġto Ġdifferentiate Ġevent - instance Ġrelated Ġmic rop ost s Ġand Ġgeneral Ġevent - category Ġrelated Ġmic rop ost s . ĠConsider , Ġfor Ġexample , Ġthe Ġfollowing Ġmic rop ost s Ġin Ġthe Ġcontext Ġof ĠCyber Ġattack Ġevents , Ġboth Ġcontaining Ġthe Ġkeyword Ġ` hack ': </s> Credit Ġfirm ĠEqu ifax Ġsays Ġ143 m ĠAmericans ' Ġsocial Ġsecurity Ġnumbers Ġexposed Ġin Ġhack </s> This Ġmic rop ost Ġdescribes Ġan Ġinstance Ġof Ġa Ġcyber Ġattack Ġevent Ġthat Ġthe Ġtarget Ġmodel Ġshould Ġidentify . ĠThis Ġis , Ġtherefore , Ġan Ġevent - instance Ġrelated Ġmic rop ost Ġand Ġshould Ġbe Ġconsidered Ġas Ġa Ġpositive Ġexample . ĠContrast Ġthis Ġwith Ġthe Ġfollowing Ġexample : </s> Companies Ġneed Ġto Ġstep Ġtheir Ġcyber Ġsecurity Ġup </s> This Ġmic rop ost , Ġthough Ġrelated Ġto Ġcyber Ġsecurity Ġin Ġgeneral , Ġdoes Ġnot Ġmention Ġan Ġinstance Ġof Ġa Ġcyber Ġattack Ġevent , Ġand Ġis Ġof Ġno Ġinterest Ġto Ġus Ġfor Ġevent Ġdetection . ĠThis Ġis Ġan Ġexample Ġof Ġa Ġgeneral Ġevent - category Ġrelated Ġmic rop ost Ġand Ġshould Ġbe Ġconsidered Ġas Ġa Ġnegative Ġexample . </s> In Ġthis Ġtask , Ġeach Ġselected Ġmic rop ost Ġis Ġlabeled Ġby Ġmultiple Ġcrowd Ġworkers . ĠThe Ġannotations Ġare Ġpassed Ġto Ġour Ġprob abil istic Ġmodel Ġfor Ġexpectation Ġinference Ġand Ġmodel Ġtraining . </s> Ex pect ation ĠIn ference Ġ& ĠModel ĠTraining . ĠOur Ġprob abil istic Ġmodel Ġtakes Ġcrowd - cont ributed Ġlabels Ġand Ġthe Ġmodel Ġtrained Ġin Ġthe Ġprevious Ġiteration Ġas Ġinput . ĠAs Ġoutput , Ġit Ġgenerates Ġa Ġkeyword - specific Ġexpectation , Ġden oted Ġas Ġ$ e ^{ ( t )} $, Ġand Ġan Ġimproved Ġversion Ġof Ġthe Ġmic rop ost Ġclassification Ġmodel , Ġden oted Ġas Ġ$ p _{ \ the ta Ġ^ { ( t ) }} ( y | x ) $. ĠThe Ġdetails Ġof Ġour Ġprob abil istic Ġmodel Ġare Ġgiven Ġin ĠSection ĠSEC REF 3 . </s> Key word ĠDiscovery . ĠThe Ġkeyword Ġdiscovery Ġtask Ġaims Ġat Ġdiscovering Ġa Ġnew Ġkeyword Ġ( or Ġa Ġset Ġof Ġkeywords ) Ġthat Ġis Ġmost Ġinformative Ġfor Ġmodel Ġtraining Ġwith Ġrespect Ġto Ġexisting Ġkeywords . ĠTo Ġthis Ġend , Ġwe Ġfirst Ġapply Ġthe Ġcurrent Ġmodel Ġ$ p _{ \ the ta Ġ^ { ( t ) }} ( y | x ) $ Ġon Ġthe Ġunl abel ed Ġmic rop ost s Ġ$\ math cal Ġ{ U } $. ĠFor Ġthose Ġthat Ġcontain Ġthe Ġkeyword Ġ$ w ^{ ( t )} $, Ġwe Ġcalculate Ġthe Ġdisagreement Ġbetween Ġthe Ġmodel Ġpredictions Ġand Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $ : </s> and Ġselect Ġthe Ġones Ġwith Ġthe Ġhighest Ġdisagreement Ġfor Ġkeyword Ġdiscovery . ĠThese Ġselected Ġmic rop ost s Ġare Ġsupposed Ġto Ġcontain Ġinformation Ġthat Ġcan Ġexplain Ġthe Ġdisagreement Ġbetween Ġthe Ġmodel Ġprediction Ġand Ġkeyword - specific Ġexpectation , Ġand Ġcan Ġthus Ġprovide Ġinformation Ġthat Ġis Ġmost Ġdifferent Ġfrom Ġthe Ġexisting Ġset Ġof Ġkeywords Ġfor Ġmodel Ġtraining . </s> For Ġinstance , Ġour Ġstudy Ġshows Ġthat Ġthe Ġexpectation Ġfor Ġthe Ġkeyword Ġ` hack ' Ġis Ġ0 . 20 , Ġwhich Ġmeans Ġonly Ġ20 % Ġof Ġthe Ġinitial Ġset Ġof Ġmic rop ost s Ġretrieved Ġwith Ġthe Ġkeyword Ġare Ġevent - related . ĠA Ġmic rop ost Ġselected Ġwith Ġthe Ġhighest Ġdisagreement Ġ( E q . ĠDIS PLAY _ FORM 7 ), Ġwhose Ġlikelihood Ġof Ġbeing Ġevent - related Ġas Ġpredicted Ġby Ġthe Ġmodel Ġis Ġ$ 99 . 9 \ % $, Ġis Ġshown Ġas Ġan Ġexample Ġbelow : </s> RT Ġ@ xxx : ĠHong ĠKong Ġsecurities Ġbrokers Ġhit Ġby Ġcyber Ġattacks , Ġmay Ġface Ġmore : Ġregulator Ġ# cy ber Ġ# security Ġ# h acking Ġhttps :// t . co / r C 1 s 9 CB </s> This Ġmic rop ost Ġcontains Ġkeywords Ġthat Ġcan Ġbetter Ġindicate Ġthe Ġrelevance Ġto Ġa Ġcyber Ġsecurity Ġevent Ġthan Ġthe Ġinitial Ġkeyword Ġ` hack ', Ġe . g ., Ġ` sec urities ', Ġ` hit ', Ġand Ġ` attack '. </s> Note Ġthat Ġwhen Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 7 Ġis Ġhigh , Ġthe Ġselected Ġmic rop ost s Ġwill Ġbe Ġthe Ġones Ġthat Ġcontain Ġkeywords Ġindicating Ġthe Ġirre lev ance Ġof Ġthe Ġmic rop ost s Ġto Ġan Ġevent Ġcategory . ĠSuch Ġkeywords Ġare Ġalso Ġuseful Ġfor Ġmodel Ġtraining Ġas Ġthey Ġhelp Ġimprove Ġthe Ġmodel 's Ġability Ġto Ġidentify Ġirrelevant Ġmic rop ost s . </s> To Ġidentify Ġnew Ġkeywords Ġin Ġthe Ġselected Ġmic rop ost s , Ġwe Ġagain Ġleverage Ġcrowds ourcing , Ġas Ġhumans Ġare Ġtypically Ġbetter Ġthan Ġmachines Ġat Ġproviding Ġspecific Ġexplanations ĠB IB REF 18 , ĠB IB REF 19 . ĠIn Ġthe Ġcrowds ourcing Ġtask , Ġworkers Ġare Ġfirst Ġasked Ġto Ġfind Ġthose Ġmic rop ost s Ġwhere Ġthe Ġmodel Ġpredictions Ġare Ġdeemed Ġcorrect . ĠThen , Ġfrom Ġthose Ġmic rop ost s , Ġworkers Ġare Ġasked Ġto Ġfind Ġthe Ġkeyword Ġthat Ġbest Ġindicates Ġthe Ġclass Ġof Ġthe Ġmic rop ost s Ġas Ġpredicted Ġby Ġthe Ġmodel . ĠThe Ġkeyword Ġmost Ġfrequently Ġidentified Ġby Ġthe Ġworkers Ġis Ġthen Ġused Ġas Ġthe Ġinitial Ġkeyword Ġfor Ġthe Ġfollowing Ġiteration . ĠIn Ġcase Ġmultiple Ġkeywords Ġare Ġselected , Ġe . g ., Ġthe Ġtop -$ N $ Ġfrequent Ġones , Ġworkers Ġwill Ġbe Ġasked Ġto Ġperform Ġ$ N $ Ġmic rop ost Ġclassification Ġtasks Ġfor Ġeach Ġkeyword Ġin Ġthe Ġnext Ġiteration , Ġand Ġthe Ġmodel Ġtraining Ġwill Ġbe Ġperformed Ġon Ġmultiple Ġkeyword - specific Ġexpectations . </s> Un ified ĠProb abil istic ĠModel </s> This Ġsection Ġintroduces Ġour Ġprob abil istic Ġmodel Ġthat Ġinf ers Ġkeyword Ġexpectation Ġand Ġtrains Ġthe Ġtarget Ġmodel Ġsimultaneously . ĠWe Ġstart Ġby Ġformal izing Ġthe Ġproblem Ġand Ġintroducing Ġour Ġmodel , Ġbefore Ġdescribing Ġthe Ġmodel Ġlearning Ġmethod . </s> Problem ĠForm al ization . ĠWe Ġconsider Ġthe Ġproblem Ġat Ġiteration Ġ$ t $ Ġwhere Ġthe Ġcorresponding Ġkeyword Ġis Ġ$ w ^{ ( t )} $. ĠIn Ġthe Ġcurrent Ġiteration , Ġlet Ġ$\ math cal Ġ{ U } ^{ ( t )} Ġ\ sub set Ġ\ math cal Ġ{ U } $ Ġdenote Ġthe Ġset Ġof Ġall Ġmic rop ost s Ġcontaining Ġthe Ġkeyword Ġand Ġ$\ math cal Ġ{ M } ^{ ( t )} = Ġ\ l brace Ġx _{ m }\ r brace Ġ_ { m = 1 } ^ M \ sub set Ġ\ math cal Ġ{ U } ^{ ( t )} $ Ġbe Ġthe Ġrandomly Ġselected Ġsubset Ġof Ġ$ M $ Ġmic rop ost s Ġlabeled Ġby Ġ$ N $ Ġcrowd Ġworkers Ġ$\ math cal Ġ{ C } Ġ= Ġ\ l brace Ġc _ n \ r brace Ġ_ { n = 1 } ^ N $. ĠThe Ġannotations Ġform Ġa Ġmatrix Ġ$\ math bf Ġ{ A }\ in Ġ\ math bb Ġ{ R } ^{ M \ times ĠN } $ Ġwhere Ġ$\ math bf Ġ{ A } _{ mn } $ Ġis Ġthe Ġlabel Ġfor Ġthe Ġmic rop ost Ġ$ x _ m $ Ġcontributed Ġby Ġcrowd Ġworker Ġ$ c _ n $. ĠOur Ġgoal Ġis Ġto Ġinfer Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $ Ġand Ġtrain Ġthe Ġtarget Ġmodel Ġby Ġlearning Ġthe Ġmodel Ġparameter Ġ$\ the ta Ġ^ { ( t )} $. ĠAn Ġadditional Ġparameter Ġof Ġour Ġprob abil istic Ġmodel Ġis Ġthe Ġreliability Ġof Ġcrowd Ġworkers , Ġwhich Ġis Ġessential Ġwhen Ġinvolving Ġcrowds ourcing . ĠFollowing ĠDaw id Ġand ĠSk ene ĠB IB REF 14 , ĠB IB REF 16 , Ġwe Ġrepresent Ġthe Ġannotation Ġreliability Ġof Ġworker Ġ$ c _ n $ Ġby Ġa Ġlatent Ġconfusion Ġmatrix Ġ$\ pi Ġ^ { ( n )} $, Ġwhere Ġthe Ġ$ rs $ - th Ġelement Ġ$\ pi Ġ_ { rs } ^{ ( n )} $ Ġdenotes Ġthe Ġprobability Ġof Ġ$ c _ n $ Ġlabeling Ġa Ġmic rop ost Ġas Ġclass Ġ$ r $ Ġgiven Ġthe Ġtrue Ġclass Ġ$ s $. </s> Un ified ĠProb abil istic ĠModel Ġ: :: ĠExpect ation Ġas ĠModel ĠPoster ior </s> First , Ġwe Ġintroduce Ġan Ġexpectation Ġregular ization Ġtechnique Ġfor Ġthe Ġweak ly Ġsupervised Ġlearning Ġof Ġthe Ġtarget Ġmodel Ġ$ p _{ \ the ta Ġ^ { ( t ) }} ( y | x ) $. ĠIn Ġthis Ġsetting , Ġthe Ġobjective Ġfunction Ġof Ġthe Ġtarget Ġmodel Ġis Ġcomposed Ġof Ġtwo Ġparts , Ġcorresponding Ġto Ġthe Ġlabeled Ġmic rop ost s Ġ$\ math cal Ġ{ L } $ Ġand Ġthe Ġunl abel ed Ġones Ġ$\ math cal Ġ{ U } $. </s> The Ġformer Ġpart Ġaims Ġat Ġmaximizing Ġthe Ġlikelihood Ġof Ġthe Ġlabeled Ġmic rop ost s : </s> where Ġwe Ġassume Ġthat Ġ$\ the ta Ġ$ Ġis Ġgenerated Ġfrom Ġa Ġprior Ġdistribution Ġ( e . g ., ĠLa pl ac ian Ġor ĠGa ussian ) Ġparameter ized Ġby Ġ$\ s igma Ġ$ . </s> To Ġleverage Ġunl abel ed Ġdata Ġfor Ġmodel Ġtraining , Ġwe Ġmake Ġuse Ġof Ġthe Ġexpectations Ġof Ġexisting Ġkeywords , Ġi . e ., Ġ{ ($ w ^{ ( 1 )} $, Ġ$ e ^{ ( 1 )} $ ), Ġ... , Ġ($ w ^{ ( t - 1 )} $, Ġ$ e ^{ ( t - 1 )} $ ), Ġ($ w ^{ ( t )} $, Ġ$ e ^{ ( t )} $ )} Ġ( Note Ġthat Ġ$ e ^{ ( t )} $ Ġis Ġinferred ), Ġas Ġa Ġregular ization Ġterm Ġto Ġconst rain Ġmodel Ġtraining . ĠTo Ġdo Ġso , Ġwe Ġfirst Ġgive Ġthe Ġmodel 's Ġexpectation Ġfor Ġeach Ġkeyword Ġ$ w ^{ ( k )} $ Ġ($ 1 \ le Ġk \ le Ġt $ ) Ġas Ġfollows : </s> which Ġdenotes Ġthe Ġempirical Ġexpectation Ġof Ġthe Ġmodel âĢ Ļ s Ġposterior Ġpredictions Ġon Ġthe Ġunl abel ed Ġmic rop ost s Ġ$\ math cal Ġ{ U } ^{ ( k )} $ Ġcontaining Ġkeyword Ġ$ w ^{ ( k )} $. ĠExpect ation Ġregular ization Ġcan Ġthen Ġbe Ġformulated Ġas Ġthe Ġregular ization Ġof Ġthe Ġdistance Ġbetween Ġthe ĠBern ou lli Ġdistribution Ġparameter ized Ġby Ġthe Ġmodel 's Ġexpectation Ġand Ġthe Ġexpectation Ġof Ġthe Ġexisting Ġkeyword : </s> where Ġ$ D _{ K L } [ \ cd ot Ġ\ Vert Ġ\ cd ot Ġ] $ Ġdenotes Ġthe ĠKL - d iver gence Ġbetween Ġthe ĠBern ou lli Ġdistributions Ġ$ Ber ( e ^{ ( k ) }) $ Ġand Ġ$ Ber (\ math bb Ġ{ E } _{ x \ sim Ġ\ math cal Ġ{ U } ^{ ( k ) }} ( y )) $, Ġand Ġ$\ lambda Ġ$ Ġcontrols Ġthe Ġstrength Ġof Ġexpectation Ġregular ization . </s> Un ified ĠProb abil istic ĠModel Ġ: :: ĠExpect ation Ġas ĠClass ĠPrior </s> To Ġlearn Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $ Ġand Ġthe Ġcrowd Ġworker Ġreliability Ġ$\ pi Ġ^ { ( n )} $ Ġ($ 1 \ le Ġn \ le ĠN $ ), Ġwe Ġmodel Ġthe Ġlikelihood Ġof Ġthe Ġcrowd - cont ributed Ġlabels Ġ$\ math bf Ġ{ A } $ Ġas Ġa Ġfunction Ġof Ġthese Ġparameters . ĠIn Ġthis Ġcontext , Ġwe Ġview Ġthe Ġexpectation Ġas Ġthe Ġclass Ġprior , Ġthus Ġperforming Ġexpectation Ġinference Ġas Ġthe Ġlearning Ġof Ġthe Ġclass Ġprior . ĠBy Ġdoing Ġso , Ġwe Ġconnect Ġexpectation Ġinference Ġwith Ġmodel Ġtraining . </s> Specifically , Ġwe Ġmodel Ġthe Ġlikelihood Ġof Ġan Ġarbitrary Ġcrowd - cont ributed Ġlabel Ġ$\ math bf Ġ{ A } _{ mn } $ Ġas Ġa Ġmixture Ġof Ġmult in om ials Ġwhere Ġthe Ġprior Ġis Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $ : </s> where Ġ$ e _ s ^{ ( t )} $ Ġis Ġthe Ġprobability Ġof Ġthe Ġground Ġtruth Ġlabel Ġbeing Ġ$ s $ Ġgiven Ġthe Ġkeyword - specific Ġexpectation Ġas Ġthe Ġclass Ġprior ; Ġ$ K $ Ġis Ġthe Ġset Ġof Ġpossible Ġground Ġtruth Ġlabels Ġ( binary Ġin Ġour Ġcontext ); Ġand Ġ$ r = \ math bf Ġ{ A } _{ mn } $ Ġis Ġthe Ġcrowd - cont ributed Ġlabel . ĠThen , Ġfor Ġan Ġindividual Ġmic rop ost Ġ$ x _ m $, Ġthe Ġlikelihood Ġof Ġcrowd - cont ributed Ġlabels Ġ$\ math bf Ġ{ A } _{ m : } $ Ġis Ġgiven Ġby : </s> Therefore , Ġthe Ġobjective Ġfunction Ġfor Ġmaximizing Ġthe Ġlikelihood Ġof Ġthe Ġentire Ġannotation Ġmatrix Ġ$\ math bf Ġ{ A } $ Ġcan Ġbe Ġdescribed Ġas : </s> Un ified ĠProb abil istic ĠModel Ġ: :: ĠUnified ĠProb abil istic ĠModel </s> Integ rating Ġmodel Ġtraining Ġwith Ġexpectation Ġinference , Ġthe Ġoverall Ġobjective Ġfunction Ġof Ġour Ġproposed Ġmodel Ġis Ġgiven Ġby : </s> Figure ĠFIG REF 18 Ġdepicts Ġa Ġgraphical Ġrepresentation Ġof Ġour Ġmodel , Ġwhich Ġcombines Ġthe Ġtarget Ġmodel Ġfor Ġtraining Ġ( on Ġthe Ġleft ) Ġwith Ġthe Ġgener ative Ġmodel Ġfor Ġcrowd - cont ributed Ġlabels Ġ( on Ġthe Ġright ) Ġthrough Ġa Ġkeyword - specific Ġexpectation . </s> Model ĠLearning . ĠDue Ġto Ġthe Ġunknown Ġground Ġtruth Ġlabels Ġof Ġcrowd - annot ated Ġmic rop ost s Ġ($ y _ m $ Ġin ĠFigure ĠFIG REF 18 ), Ġwe Ġresort Ġto Ġexpectation Ġmaxim ization Ġfor Ġmodel Ġlearning . ĠThe Ġlearning Ġalgorithm Ġiter atively Ġtakes Ġtwo Ġsteps : Ġthe ĠE - step Ġand Ġthe ĠM - step . ĠThe ĠE - step Ġinf ers Ġthe Ġground Ġtruth Ġlabels Ġgiven Ġthe Ġcurrent Ġmodel Ġparameters . ĠThe ĠM - step Ġupdates Ġthe Ġmodel Ġparameters , Ġincluding Ġthe Ġcrowd Ġreliability Ġparameters Ġ$\ pi Ġ^ { ( n )} $ Ġ($ 1 \ le Ġn \ le ĠN $ ), Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $, Ġand Ġthe Ġparameter Ġof Ġthe Ġtarget Ġmodel Ġ$\ the ta Ġ^ { ( t )} $. ĠThe ĠE - step Ġand Ġthe Ġcrowd Ġparameter Ġupdate Ġin Ġthe ĠM - step Ġare Ġsimilar Ġto Ġthe ĠDaw id - Sk ene Ġmodel ĠB IB REF 14 . ĠThe Ġkeyword Ġexpectation Ġis Ġinferred Ġby Ġtaking Ġinto Ġaccount Ġboth Ġthe Ġcrowd - cont ributed Ġlabels Ġand Ġthe Ġmodel Ġprediction : </s> The Ġparameter Ġof Ġthe Ġtarget Ġmodel Ġis Ġupdated Ġby Ġgradient Ġdescent . ĠFor Ġexample , Ġwhen Ġthe Ġtarget Ġmodel Ġto Ġbe Ġtrained Ġis Ġa Ġdeep Ġneural Ġnetwork , Ġwe Ġuse Ġback - prop ag ation Ġwith Ġgradient Ġdescent Ġto Ġupdate Ġthe Ġweight Ġmat rices . </s> Exper iments Ġand ĠResults </s> This Ġsection Ġpresents Ġour Ġexperimental Ġsetup Ġand Ġresults Ġfor Ġevaluating Ġour Ġapproach . ĠWe Ġaim Ġat Ġanswering Ġthe Ġfollowing Ġquestions : </s> [ no item se p , left margin = * ] </s> Q 1 : ĠHow Ġeffectively Ġdoes Ġour Ġproposed Ġhuman - AI Ġloop Ġapproach Ġenhance Ġthe Ġstate - of - the - art Ġmachine Ġlearning Ġmodels Ġfor Ġevent Ġdetection ? </s> Q 2 : ĠHow Ġwell Ġdoes Ġour Ġkeyword Ġdiscovery Ġmethod Ġwork Ġcompare Ġto Ġexisting Ġkeyword Ġexpansion Ġmethods ? </s> Q 3 : ĠHow Ġeffective Ġis Ġour Ġapproach Ġusing Ġcrowds ourcing Ġat Ġobtaining Ġnew Ġkeywords Ġcompared Ġwith Ġan Ġapproach Ġlab elling Ġmic rop ost s Ġfor Ġmodel Ġtraining Ġunder Ġthe Ġsame Ġcost ? </s> Q 4 : ĠHow Ġmuch Ġbenefit Ġdoes Ġour Ġunified Ġprob abil istic Ġmodel Ġbring Ġcompared Ġto Ġmethods Ġthat Ġdo Ġnot Ġtake Ġcrowd Ġreliability Ġinto Ġaccount ? </s> Exper iments Ġand ĠResults Ġ: :: ĠExperimental ĠSetup </s> Dat as ets . ĠWe Ġperform Ġour Ġexperiments Ġwith Ġtwo Ġpredetermined Ġevent Ġcategories : Ġcyber Ġsecurity Ġ( Cy ber Attack ) Ġand Ġdeath Ġof Ġpoliticians Ġ( Polit ician Death ). ĠThese Ġevent Ġcategories Ġare Ġchosen Ġas Ġthey Ġare Ġrepresentative Ġof Ġimportant Ġevent Ġtypes Ġthat Ġare Ġof Ġinterest Ġto Ġmany Ġgovernments Ġand Ġcompanies . ĠThe Ġneed Ġto Ġcreate Ġour Ġown Ġdataset Ġwas Ġmotivated Ġby Ġthe Ġlack Ġof Ġpublic Ġdatasets Ġfor Ġevent Ġdetection Ġon Ġmic rop ost s . ĠThe Ġfew Ġavailable Ġdatasets Ġdo Ġnot Ġsuit Ġour Ġrequirements . ĠFor Ġexample , Ġthe Ġpublicly Ġavailable ĠEvents - 2012 ĠTwitter Ġdataset ĠB IB REF 20 Ġcontains Ġgeneric Ġevent Ġdescriptions Ġsuch Ġas ĠPolitics , ĠSports , ĠCulture Ġetc . ĠOur Ġwork Ġtargets Ġmore Ġspecific Ġevent Ġcategories ĠB IB REF 21 . ĠFollowing Ġprevious Ġstudies ĠB IB REF 1 , Ġwe Ġcollect Ġevent - related Ġmic rop ost s Ġfrom ĠTwitter Ġusing Ġ11 Ġand Ġ8 Ġseed Ġevents Ġ( see ĠSection ĠSEC REF 2 ) Ġfor ĠCyber Attack Ġand ĠPolit ician Death , Ġrespectively . ĠUn label ed Ġmic rop ost s Ġare Ġcollected Ġby Ġusing Ġthe Ġkeyword Ġ` hack ' Ġfor ĠCyber Attack , Ġwhile Ġfor ĠPolit ician Death , Ġwe Ġuse Ġa Ġset Ġof Ġkeywords Ġrelated Ġto Ġ` polit ician ' Ġand Ġ` death ' Ġ( such Ġas Ġ` b ureau cr at ', Ġ` dead ' Ġetc .) ĠFor Ġeach Ġdataset , Ġwe Ġrandomly Ġselect Ġ500 Ġtweets Ġfrom Ġthe Ġunl abel ed Ġsubset Ġand Ġmanually Ġlabel Ġthem Ġfor Ġevaluation . ĠTable ĠT AB REF 25 Ġshows Ġkey Ġstatistics Ġfrom Ġour Ġtwo Ġdatasets . </s> Compar ison ĠMethods . ĠTo Ġdemonstrate Ġthe Ġgener ality Ġof Ġour Ġapproach Ġon Ġdifferent Ġevent Ġdetection Ġmodels , Ġwe Ġconsider ĠLog istic ĠReg ression Ġ( LR ) ĠB IB REF 1 Ġand ĠMult il ayer ĠPer cept ron Ġ( ML P ) ĠB IB REF 2 Ġas Ġthe Ġtarget Ġmodels . ĠAs Ġthe Ġgoal Ġof Ġour Ġexperiments Ġis Ġto Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġour Ġapproach Ġas Ġa Ġnew Ġmodel Ġtraining Ġtechnique , Ġwe Ġuse Ġthese Ġwidely Ġused Ġmodels . ĠAlso , Ġwe Ġnote Ġthat Ġin Ġour Ġcase Ġother Ġneural Ġnetwork Ġmodels Ġwith Ġmore Ġcomplex Ġnetwork Ġarchitectures Ġfor Ġevent Ġdetection , Ġsuch Ġas Ġthe Ġbi - direction al ĠL ST M ĠB IB REF 17 , Ġturn Ġout Ġto Ġbe Ġless Ġeffective Ġthan Ġa Ġsimple Ġfeed forward Ġnetwork . ĠFor Ġboth ĠLR Ġand ĠML P , Ġwe Ġevaluate Ġour Ġproposed Ġhuman - AI Ġloop Ġapproach Ġfor Ġkeyword Ġdiscovery Ġand Ġexpectation Ġestimation Ġby Ġcomparing Ġagainst Ġthe Ġweak ly Ġsupervised Ġlearning Ġmethod Ġproposed Ġby ĠB IB REF 1 Ġ( BI B REF 1 ) Ġand ĠB IB REF 17 Ġ( BI B REF 17 ) Ġwhere Ġonly Ġone Ġinitial Ġkeyword Ġis Ġused Ġwith Ġan Ġexpectation Ġestimated Ġby Ġan Ġindividual Ġexpert . </s> Parameter ĠSettings . ĠWe Ġempir ically Ġset Ġoptimal Ġparameters Ġbased Ġon Ġa Ġheld - out Ġvalidation Ġset Ġthat Ġcontains Ġ20 % Ġof Ġthe Ġtest Ġdata . ĠThese Ġinclude Ġthe Ġhyper param ters Ġof Ġthe Ġtarget Ġmodel , Ġthose Ġof Ġour Ġproposed Ġprob abil istic Ġmodel , Ġand Ġthe Ġparameters Ġused Ġfor Ġtraining Ġthe Ġtarget Ġmodel . ĠWe Ġexplore ĠML P Ġwith Ġ1 , Ġ2 Ġand Ġ3 Ġhidden Ġlayers Ġand Ġapply Ġa Ġgrid Ġsearch Ġin Ġ32 , Ġ64 , Ġ128 , Ġ256 , Ġ512 Ġfor Ġthe Ġdimension Ġof Ġthe Ġembed d ings Ġand Ġthat Ġof Ġthe Ġhidden Ġlayers . ĠFor Ġthe Ġcoefficient Ġof Ġexpectation Ġregular ization , Ġwe Ġfollow ĠB IB REF 6 Ġ( BI B REF 6 ) Ġand Ġset Ġit Ġto Ġ$\ lambda Ġ= 10 Ġ\ times Ġ$ Ġ# label ed Ġexamples . ĠFor Ġmodel Ġtraining , Ġwe Ġuse Ġthe ĠAdam ĠB IB REF 22 Ġoptimization Ġalgorithm Ġfor Ġboth Ġmodels . </s> E val uation . ĠFollowing ĠB IB REF 1 Ġ( BI B REF 1 ) Ġand ĠB IB REF 3 Ġ( BI B REF 3 ), Ġwe Ġuse Ġaccuracy Ġand Ġarea Ġunder Ġthe Ġprecision - rec all Ġcurve Ġ( A UC ) Ġmetrics Ġto Ġmeasure Ġthe Ġperformance Ġof Ġour Ġproposed Ġapproach . ĠWe Ġnote Ġthat Ġdue Ġto Ġthe Ġimbalance Ġin Ġour Ġdatasets Ġ( 20 % Ġpositive Ġmic rop ost s Ġin ĠCyber Attack Ġand Ġ27 % Ġin ĠPolit ician Death ), Ġaccuracy Ġis Ġdominated Ġby Ġnegative Ġexamples ; ĠA UC , Ġin Ġcomparison , Ġbetter Ġcharacter izes Ġthe Ġdiscrim in ative Ġpower Ġof Ġthe Ġmodel . </s> C rowd s ourcing . ĠWe Ġchose ĠLevel Ġ3 Ġworkers Ġon Ġthe ĠFigure - Eight Ġcrowds ourcing Ġplatform Ġfor Ġour Ġexperiments . ĠThe Ġinter - annot ator Ġagreement Ġin Ġmic rop ost Ġclassification Ġis Ġtaken Ġinto Ġaccount Ġthrough Ġthe ĠEM Ġalgorithm . ĠFor Ġkeyword Ġdiscovery , Ġwe Ġfilter Ġkeywords Ġbased Ġon Ġthe Ġfrequency Ġof Ġthe Ġkeyword Ġbeing Ġselected Ġby Ġthe Ġcrowd . ĠIn Ġterms Ġof Ġcost - effect iveness , Ġour Ġapproach Ġis Ġmotivated Ġfrom Ġthe Ġfact Ġthat Ġcrowds ourced Ġdata Ġannotation Ġcan Ġbe Ġexpensive , Ġand Ġis Ġthus Ġdesigned Ġwith Ġminimal Ġcrowd Ġinvolvement . ĠFor Ġeach Ġiteration , Ġwe Ġselected Ġ50 Ġtweets Ġfor Ġkeyword Ġdiscovery Ġand Ġ50 Ġtweets Ġfor Ġmic rop ost Ġclassification Ġper Ġkeyword . ĠFor Ġa Ġdataset Ġwith Ġ80 k Ġtweets Ġ( e . g ., ĠCyber Attack ), Ġour Ġapproach Ġonly Ġrequires Ġto Ġmanually Ġinspect Ġ800 Ġtweets Ġ( for Ġ8 Ġkeywords ), Ġwhich Ġis Ġonly Ġ1 % Ġof Ġthe Ġentire Ġdataset . </s> Exper iments Ġand ĠResults Ġ: :: ĠResults Ġof Ġour ĠHuman - AI ĠLoop Ġ( Q 1 ) </s> Table ĠT AB REF 26 Ġreports Ġthe Ġevaluation Ġof Ġour Ġapproach Ġon Ġboth Ġthe ĠCyber Attack Ġand ĠPolit ician Death Ġevent Ġcategories . ĠOur Ġapproach Ġis Ġconfigured Ġsuch Ġthat Ġeach Ġiteration Ġstarts Ġwith Ġ1 Ġnew Ġkeyword Ġdiscovered Ġin Ġthe Ġprevious Ġiteration . </s> Our Ġapproach Ġimproves ĠLR Ġby Ġ5 . 17 % Ġ( Acc uracy ) Ġand Ġ18 . 38 % Ġ( A UC ), Ġand ĠML P Ġby Ġ10 . 71 % Ġ( Acc uracy ) Ġand Ġ30 . 27 % Ġ( A UC ) Ġon Ġaverage . ĠSuch Ġsignificant Ġimprovements Ġclearly Ġdemonstrate Ġthat Ġour Ġapproach Ġis Ġeffective Ġat Ġimproving Ġmodel Ġperformance . ĠWe Ġobserve Ġthat Ġthe Ġtarget Ġmodels Ġgenerally Ġconverge Ġbetween Ġthe Ġ7 th Ġand Ġ9 th Ġiteration Ġon Ġboth Ġdatasets Ġwhen Ġperformance Ġis Ġmeasured Ġby ĠA UC . ĠThe Ġperformance Ġcan Ġslightly Ġdegrade Ġwhen Ġthe Ġmodels Ġare Ġfurther Ġtrained Ġfor Ġmore Ġiterations Ġon Ġboth Ġdatasets . ĠThis Ġis Ġlikely Ġdue Ġto Ġthe Ġfact Ġthat Ġover Ġtime , Ġthe Ġnewly Ġdiscovered Ġkeywords Ġentail Ġlower Ġnovel Ġinformation Ġfor Ġmodel Ġtraining . ĠFor Ġinstance , Ġfor Ġthe ĠCyber Attack Ġdataset Ġthe Ġnew Ġkeyword Ġin Ġthe Ġ9 th Ġiteration Ġ` election ' Ġfrequently Ġco - occ urs Ġwith Ġthe Ġkeyword Ġ` r ussia ' Ġin Ġthe Ġ5 th Ġiteration Ġ( in Ġmic rop ost s Ġthat Ġconnect ĠRussian Ġhackers Ġwith ĠUS Ġelections ), Ġthus Ġbringing Ġlimited Ġnew Ġinformation Ġfor Ġimproving Ġthe Ġmodel Ġperformance . ĠAs Ġa Ġside Ġremark , Ġwe Ġnote Ġthat Ġthe Ġmodels Ġconverge Ġfaster Ġwhen Ġperformance Ġis Ġmeasured Ġby Ġaccuracy . ĠSuch Ġa Ġcomparison Ġresult Ġconfirms Ġthe Ġdifference Ġbetween Ġthe Ġmetrics Ġand Ġshows Ġthe Ġnecessity Ġfor Ġmore Ġkeywords Ġto Ġdiscriminate Ġevent - related Ġmic rop ost s Ġfrom Ġnon Ġevent - related Ġones . </s> Exper iments Ġand ĠResults Ġ: :: ĠCompar ative ĠResults Ġon ĠKey word ĠDiscovery Ġ( Q 2 ) </s> Figure ĠFIG REF 31 Ġshows Ġthe Ġevaluation Ġof Ġour Ġapproach Ġwhen Ġdiscovering Ġnew Ġinformative Ġkeywords Ġfor Ġmodel Ġtraining Ġ( see ĠSection ĠSEC REF 2 : ĠKey word ĠDiscovery ). ĠWe Ġcompare Ġour Ġhuman - AI Ġcollaborative Ġway Ġof Ġdiscovering Ġnew Ġkeywords Ġagainst Ġa Ġquery Ġexpansion Ġ( Q E ) Ġapproach ĠB IB REF 23 , ĠB IB REF 24 Ġthat Ġlever ages Ġword Ġembed d ings Ġto Ġfind Ġsimilar Ġwords Ġin Ġthe Ġlatent Ġsemantic Ġspace . ĠSpecifically , Ġwe Ġuse Ġpre - trained Ġword Ġembed d ings Ġbased Ġon Ġa Ġlarge ĠGoogle ĠNews Ġdataset Ġfor Ġquery Ġexpansion . ĠFor Ġinstance , Ġthe Ġtop Ġkeywords Ġresulting Ġfrom ĠQ E Ġfor Ġ` polit ician ' Ġare , Ġ` dep uty ', ` minist ry ', ` secret ary ', Ġand Ġ` min ister '. ĠFor Ġeach Ġof Ġthese Ġkeywords , Ġwe Ġuse Ġthe Ġcrowd Ġto Ġlabel Ġa Ġset Ġof Ġtweets Ġand Ġobtain Ġa Ġcorresponding Ġexpectation . </s> We Ġobserve Ġthat Ġour Ġapproach Ġconsistently Ġoutper forms ĠQ E Ġby Ġan Ġaverage Ġof Ġ$ 4 . 62 \ % $ Ġand Ġ$ 52 . 58 \ % $ ĠA UC Ġon ĠCyber Attack Ġand ĠPolit ician Death , Ġrespectively . ĠThe Ġlarge Ġgap Ġbetween Ġthe Ġperformance Ġimprovements Ġfor Ġthe Ġtwo Ġdatasets Ġis Ġmainly Ġdue Ġto Ġthe Ġfact Ġthat Ġmic rop ost s Ġthat Ġare Ġrelevant Ġfor ĠPolit ician Death Ġare Ġsem antically Ġmore Ġcomplex Ġthan Ġthose Ġfor ĠCyber Attack , Ġas Ġthey Ġencode Ġnoun - verb Ġrelationship Ġ( e . g ., ĠâĢ ľ the Ġking Ġof Ġ... Ġdied Ġ... âĢ Ŀ ) Ġrather Ġthan Ġa Ġsimple Ġverb Ġ( e . g ., ĠâĢ ľ ... Ġhacked . âĢ Ŀ ) Ġfor Ġthe ĠCyber Attack Ġmic rop ost s . ĠQ E Ġonly Ġfinds Ġsyn onyms Ġof Ġexisting Ġkeywords Ġrelated Ġto Ġeither Ġ` polit ician ' Ġor Ġ` death ', Ġhowever Ġcannot Ġfind Ġa Ġmeaningful Ġkeyword Ġthat Ġfully Ġcharacter izes Ġthe Ġdeath Ġof Ġa Ġpolitician . ĠFor Ġinstance , ĠQ E Ġfinds Ġthe Ġkeywords Ġ` kill ' Ġand Ġ` mur der ', Ġwhich Ġare Ġsem antically Ġclose Ġto Ġ` death ' Ġbut Ġare Ġnot Ġspecifically Ġrelevant Ġto Ġthe Ġdeath Ġof Ġa Ġpolitician . ĠUnlike ĠQ E , Ġour Ġapproach Ġidentifies Ġkeywords Ġthat Ġgo Ġbeyond Ġmere Ġsyn onyms Ġand Ġthat Ġare Ġmore Ġdirectly Ġrelated Ġto Ġthe Ġend Ġtask , Ġi . e ., Ġdiscriminating Ġevent - related Ġmic rop ost s Ġfrom Ġnon Ġrelated Ġones . ĠExamples Ġare Ġ` dem ise ' Ġand Ġ` cond ol ence '. ĠAs Ġa Ġremark , Ġwe Ġnote Ġthat Ġin ĠFigure ĠFIG REF 31 ( b ), Ġthe Ġincrease Ġin ĠQ E Ġperformance Ġon ĠPolit ician Death Ġis Ġdue Ġto Ġthe Ġkeywords Ġ` dep uty ' Ġand Ġ` min ister ', Ġwhich Ġhappen Ġto Ġbe Ġhighly Ġindicative Ġof Ġthe Ġdeath Ġof Ġa Ġpolitician Ġin Ġour Ġdataset ; Ġthese Ġkeywords Ġare Ġalso Ġidentified Ġby Ġour Ġapproach . </s> Exper iments Ġand ĠResults Ġ: :: ĠCost - Effect iveness ĠResults Ġ( Q 3 ) </s> To Ġdemonstrate Ġthe Ġcost - effect iveness Ġof Ġusing Ġcrowds ourcing Ġfor Ġobtaining Ġnew Ġkeywords Ġand Ġconsequently , Ġtheir Ġexpectations , Ġwe Ġcompare Ġthe Ġperformance Ġof Ġour Ġapproach Ġwith Ġan Ġapproach Ġusing Ġcrowds ourcing Ġto Ġonly Ġlabel Ġmic rop ost s Ġfor Ġmodel Ġtraining Ġat Ġthe Ġsame Ġcost . ĠSpecifically , Ġwe Ġconducted Ġan Ġadditional Ġcrowds ourcing Ġexperiment Ġwhere Ġthe Ġsame Ġcost Ġused Ġfor Ġkeyword Ġdiscovery Ġin Ġour Ġapproach Ġis Ġused Ġto Ġlabel Ġadditional Ġmic rop ost s Ġfor Ġmodel Ġtraining . ĠThese Ġnewly Ġlabeled Ġmic rop ost s Ġare Ġused Ġwith Ġthe Ġmic rop ost s Ġlabeled Ġin Ġthe Ġmic rop ost Ġclassification Ġtask Ġof Ġour Ġapproach Ġ( see ĠSection ĠSEC REF 2 : ĠMic rop ost ĠClassification ) Ġand Ġthe Ġexpectation Ġof Ġthe Ġinitial Ġkeyword Ġto Ġtrain Ġthe Ġmodel Ġfor Ġcomparison . ĠThe Ġmodel Ġtrained Ġin Ġthis Ġway Ġincreases ĠA UC Ġby Ġ0 . 87 % Ġfor ĠCyber Attack , Ġand Ġby Ġ1 . 06 % Ġfor ĠPolit ician Death ; Ġin Ġcomparison , Ġour Ġproposed Ġapproach Ġincreases ĠA UC Ġby Ġ33 . 42 % Ġfor ĠPolit ician Death Ġand Ġby Ġ15 . 23 % Ġfor ĠCyber Attack Ġover Ġthe Ġbaseline Ġpresented Ġby ĠB IB REF 1 ). ĠThese Ġresults Ġshow Ġthat Ġusing Ġcrowds ourcing Ġfor Ġkeyword Ġdiscovery Ġis Ġsignificantly Ġmore Ġcost - effective Ġthan Ġsimply Ġusing Ġcrowds ourcing Ġto Ġget Ġadditional Ġlabels Ġwhen Ġtraining Ġthe Ġmodel . </s> Exper iments Ġand ĠResults Ġ: :: ĠExpect ation ĠIn ference ĠResults Ġ( Q 4 ) </s> To Ġinvestigate Ġthe Ġeffectiveness Ġof Ġour Ġexpectation Ġinference Ġmethod , Ġwe Ġcompare Ġit Ġagainst Ġa Ġmajority Ġvoting Ġapproach , Ġa Ġstrong Ġbaseline Ġin Ġtruth Ġinference ĠB IB REF 16 . ĠFigure ĠFIG REF 36 Ġshows Ġthe Ġresult Ġof Ġthis Ġevaluation . ĠWe Ġobserve Ġthat Ġour Ġapproach Ġresults Ġin Ġbetter Ġmodels Ġfor Ġboth ĠCyber Attack Ġand ĠPolit ician Death . ĠOur Ġmanual Ġinvestigation Ġreveals Ġthat Ġworkers ' Ġannotations Ġare Ġof Ġhigh Ġreliability , Ġwhich Ġexplains Ġthe Ġrelatively Ġgood Ġperformance Ġof Ġmajority Ġvoting . ĠDespite Ġlimited Ġmargin Ġfor Ġimprovement , Ġour Ġmethod Ġof Ġexpectation Ġinference Ġimproves Ġthe Ġperformance Ġof Ġmajority Ġvoting Ġby Ġ$ 0 . 4 \ % $ Ġand Ġ$ 1 . 19 \ % $ ĠA UC Ġon ĠCyber Attack Ġand ĠPolit ician Death , Ġrespectively . </s> Related ĠWork </s> Event ĠDetection . ĠThe Ġtechniques Ġfor Ġevent Ġextraction Ġfrom Ġmicro blog ging Ġplatforms Ġcan Ġbe Ġclassified Ġaccording Ġto Ġtheir Ġdomain Ġspecificity Ġand Ġtheir Ġdetection Ġmethod ĠB IB REF 0 . ĠEarly Ġworks Ġmainly Ġfocus Ġon Ġopen Ġdomain Ġevent Ġdetection ĠB IB REF 25 , ĠB IB REF 26 , ĠB IB REF 27 . ĠOur Ġwork Ġfalls Ġinto Ġthe Ġcategory Ġof Ġdomain - specific Ġevent Ġdetection ĠB IB REF 21 , Ġwhich Ġhas Ġdrawn Ġincreasing Ġattention Ġdue Ġto Ġits Ġrelevance Ġfor Ġvarious Ġapplications Ġsuch Ġas Ġcyber Ġsecurity ĠB IB REF 1 , ĠB IB REF 2 Ġand Ġpublic Ġhealth ĠB IB REF 4 , ĠB IB REF 5 . ĠIn Ġterms Ġof Ġtechnique , Ġour Ġproposed Ġdetection Ġmethod Ġis Ġrelated Ġto Ġthe Ġrecently Ġproposed Ġweak ly Ġsupervised Ġlearning Ġmethods ĠB IB REF 1 , ĠB IB REF 17 , ĠB IB REF 3 . ĠThis Ġcomes Ġin Ġcontrast Ġwith Ġfully - super vised Ġlearning Ġmethods , Ġwhich Ġare Ġoften Ġlimited Ġby Ġthe Ġsize Ġof Ġthe Ġtraining Ġdata Ġ( e . g ., Ġa Ġfew Ġhundred Ġexamples ) ĠB IB REF 28 , ĠB IB REF 29 . </s> Human - in - the - Loop ĠAppro aches . ĠOur Ġwork Ġextends Ġweak ly Ġsupervised Ġlearning Ġmethods Ġby Ġinvolving Ġhumans Ġin Ġthe Ġloop ĠB IB REF 13 . ĠEx isting Ġhuman - in - the - loop Ġapproaches Ġmainly Ġleverage Ġcrowds Ġto Ġlabel Ġindividual Ġdata Ġinstances ĠB IB REF 9 , ĠB IB REF 10 Ġor Ġto Ġdebug Ġthe Ġtraining Ġdata ĠB IB REF 30 , ĠB IB REF 31 Ġor Ġcomponents ĠB IB REF 32 , ĠB IB REF 33 , ĠB IB REF 34 Ġof Ġa Ġmachine Ġlearning Ġsystem . ĠUnlike Ġthese Ġworks , Ġwe Ġleverage Ġcrowd Ġworkers Ġto Ġlabel Ġsampled Ġmic rop ost s Ġin Ġorder Ġto Ġobtain Ġkeyword - specific Ġexpectations , Ġwhich Ġcan Ġthen Ġbe Ġgeneralized Ġto Ġhelp Ġclassify Ġmic rop ost s Ġcontaining Ġthe Ġsame Ġkeyword , Ġthus Ġampl ifying Ġthe Ġutility Ġof Ġthe Ġcrowd . ĠOur Ġwork Ġis Ġfurther Ġconnected Ġto Ġthe Ġtopic Ġof Ġinterpret ability Ġand Ġtransparency Ġof Ġmachine Ġlearning Ġmodels ĠB IB REF 11 , ĠB IB REF 35 , ĠB IB REF 12 , Ġfor Ġwhich Ġhumans Ġare Ġincreasingly Ġinvolved , Ġfor Ġinstance Ġfor Ġpost - h oc Ġevaluations Ġof Ġthe Ġmodel 's Ġinterpret ability . ĠIn Ġcontrast , Ġour Ġapproach Ġdirectly Ġsolic its Ġinformative Ġkeywords Ġfrom Ġthe Ġcrowd Ġfor Ġmodel Ġtraining , Ġthereby Ġproviding Ġhuman - under stand able Ġexplanations Ġfor Ġthe Ġimproved Ġmodel . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġa Ġnew Ġhuman - AI Ġloop Ġapproach Ġfor Ġkeyword Ġdiscovery Ġand Ġexpectation Ġestimation Ġto Ġbetter Ġtrain Ġevent Ġdetection Ġmodels . ĠOur Ġapproach Ġtakes Ġadvantage Ġof Ġthe Ġdisagreement Ġbetween Ġthe Ġcrowd Ġand Ġthe Ġmodel Ġto Ġdiscover Ġinformative Ġkeywords Ġand Ġlever ages Ġthe Ġjoint Ġpower Ġof Ġthe Ġcrowd Ġand Ġthe Ġmodel Ġin Ġexpectation Ġinference . ĠWe Ġevaluated Ġour Ġapproach Ġon Ġreal - world Ġdatasets Ġand Ġshowed Ġthat Ġit Ġsignificantly Ġoutper forms Ġthe Ġstate Ġof Ġthe Ġart Ġand Ġthat Ġit Ġis Ġparticularly Ġuseful Ġfor Ġdetecting Ġevents Ġwhere Ġrelevant Ġmic rop ost s Ġare Ġsem antically Ġcomplex , Ġe . g ., Ġthe Ġdeath Ġof Ġa Ġpolitician . ĠAs Ġfuture Ġwork , Ġwe Ġplan Ġto Ġparallel ize Ġthe Ġcrowds ourcing Ġtasks Ġand Ġoptimize Ġour Ġpipeline Ġin Ġorder Ġto Ġuse Ġour Ġevent Ġdetection Ġapproach Ġin Ġreal - time . </s> Acknowled gements </s> This Ġproject Ġhas Ġreceived Ġfunding Ġfrom Ġthe ĠSwiss ĠNational ĠScience ĠFoundation Ġ( gr ant Ġ# 40 75 40 _ 16 73 20 ĠTight en - it - All ) Ġand Ġfrom Ġthe ĠEuropean ĠResearch ĠCouncil Ġ( ERC ) Ġunder Ġthe ĠEuropean ĠUnion 's ĠHorizon Ġ2020 Ġresearch Ġand Ġinnovation Ġprogramme Ġ( gr ant Ġagreement Ġ6 83 253 / Graph Int ).
Introduction Event detection on microblogging platforms such as Twitter aims to detect events preemptively. A main task in event detection is detecting events of predetermined types BIBREF0, such as concerts or controversial events based on microposts matching specific event descriptions. This task has extensive applications ranging from cyber security BIBREF1, BIBREF2 to political elections BIBREF3 or public health BIBREF4, BIBREF5. Due to the high ambiguity and inconsistency of the terms used in microposts, event detection is generally performed though statistical machine learning models, which require a labeled dataset for model training. Data labeling is, however, a long, laborious, and usually costly process. For the case of micropost classification, though positive labels can be collected (e.g., using specific hashtags, or event-related date-time information), there is no straightforward way to generate negative labels useful for model training. To tackle this lack of negative labels and the significant manual efforts in data labeling, BIBREF1 (BIBREF1, BIBREF3) introduced a weak supervision based learning approach, which uses only positively labeled data, accompanied by unlabeled examples by filtering microposts that contain a certain keyword indicative of the event type under consideration (e.g., `hack' for cyber security). Another key technique in this context is expectation regularization BIBREF6, BIBREF7, BIBREF1. Here, the estimated proportion of relevant microposts in an unlabeled dataset containing a keyword is given as a keyword-specific expectation. This expectation is used in the regularization term of the model's objective function to constrain the posterior distribution of the model predictions. By doing so, the model is trained with an expectation on its prediction for microposts that contain the keyword. Such a method, however, suffers from two key problems: Due to the unpredictability of event occurrences and the constantly changing dynamics of users' posting frequency BIBREF8, estimating the expectation associated with a keyword is a challenging task, even for domain experts; The performance of the event detection model is constrained by the informativeness of the keyword used for model training. As of now, we lack a principled method for discovering new keywords and improve the model performance. To address the above issues, we advocate a human-AI loop approach for discovering informative keywords and estimating their expectations reliably. Our approach iteratively leverages 1) crowd workers for estimating keyword-specific expectations, and 2) the disagreement between the model and the crowd for discovering new informative keywords. More specifically, at each iteration after we obtain a keyword-specific expectation from the crowd, we train the model using expectation regularization and select those keyword-related microposts for which the model's prediction disagrees the most with the crowd's expectation; such microposts are then presented to the crowd to identify new keywords that best explain the disagreement. By doing so, our approach identifies new keywords which convey more relevant information with respect to existing ones, thus effectively boosting model performance. By exploiting the disagreement between the model and the crowd, our approach can make efficient use of the crowd, which is of critical importance in a human-in-the-loop context BIBREF9, BIBREF10. An additional advantage of our approach is that by obtaining new keywords that improve model performance over time, we are able to gain insight into how the model learns for specific event detection tasks. Such an advantage is particularly useful for event detection using complex models, e.g., deep neural networks, which are intrinsically hard to understand BIBREF11, BIBREF12. An additional challenge in involving crowd workers is that their contributions are not fully reliable BIBREF13. In the crowdsourcing literature, this problem is usually tackled with probabilistic latent variable models BIBREF14, BIBREF15, BIBREF16, which are used to perform truth inference by aggregating a redundant set of crowd contributions. Our human-AI loop approach improves the inference of keyword expectation by aggregating contributions not only from the crowd but also from the model. This, however, comes with its own challenge as the model's predictions are further dependent on the results of expectation inference, which is used for model training. To address this problem, we introduce a unified probabilistic model that seamlessly integrates expectation inference and model training, thereby allowing the former to benefit from the latter while resolving the inter-dependency between the two. To the best of our knowledge, we are the first to propose a human-AI loop approach that iteratively improves machine learning models for event detection. In summary, our work makes the following key contributions: A novel human-AI loop approach for micropost event detection that jointly discovers informative keywords and estimates their expectation; A unified probabilistic model that infers keyword expectation and simultaneously performs model training; An extensive empirical evaluation of our approach on multiple real-world datasets demonstrating that our approach significantly improves the state of the art by an average of 24.3% AUC. The rest of this paper is organized as follows. First, we present our human-AI loop approach in Section SECREF2. Subsequently, we introduce our proposed probabilistic model in Section SECREF3. The experimental setup and results are presented in Section SECREF4. Finally, we briefly cover related work in Section SECREF5 before concluding our work in Section SECREF6. The Human-AI Loop Approach Given a set of labeled and unlabeled microposts, our goal is to extract informative keywords and estimate their expectations in order to train a machine learning model. To achieve this goal, our proposed human-AI loop approach comprises two crowdsourcing tasks, i.e., micropost classification followed by keyword discovery, and a unified probabilistic model for expectation inference and model training. Figure FIGREF6 presents an overview of our approach. Next, we describe our approach from a process-centric perspective. Following previous studies BIBREF1, BIBREF17, BIBREF2, we collect a set of unlabeled microposts $\mathcal {U}$ from a microblogging platform and post-filter, using an initial (set of) keyword(s), those microposts that are potentially relevant to an event category. Then, we collect a set of event-related microposts (i.e., positively labeled microposts) $\mathcal {L}$, post-filtering with a list of seed events. $\mathcal {U}$ and $\mathcal {L}$ are used together to train a discriminative model (e.g., a deep neural network) for classifying the relevance of microposts to an event. We denote the target model as $p_\theta (y|x)$, where $\theta $ is the model parameter to be learned and $y$ is the label of an arbitrary micropost, represented by a bag-of-words vector $x$. Our approach iterates several times $t=\lbrace 1, 2, \ldots \rbrace $ until the performance of the target model converges. Each iteration starts from the initial keyword(s) or the new keyword(s) discovered in the previous iteration. Given such a keyword, denoted by $w^{(t)}$, the iteration starts by sampling microposts containing the keyword from $\mathcal {U}$, followed by dynamically creating micropost classification tasks and publishing them on a crowdsourcing platform. Micropost Classification. The micropost classification task requires crowd workers to label the selected microposts into two classes: event-related and non event-related. In particular, workers are given instructions and examples to differentiate event-instance related microposts and general event-category related microposts. Consider, for example, the following microposts in the context of Cyber attack events, both containing the keyword `hack': Credit firm Equifax says 143m Americans' social security numbers exposed in hack This micropost describes an instance of a cyber attack event that the target model should identify. This is, therefore, an event-instance related micropost and should be considered as a positive example. Contrast this with the following example: Companies need to step their cyber security up This micropost, though related to cyber security in general, does not mention an instance of a cyber attack event, and is of no interest to us for event detection. This is an example of a general event-category related micropost and should be considered as a negative example. In this task, each selected micropost is labeled by multiple crowd workers. The annotations are passed to our probabilistic model for expectation inference and model training. Expectation Inference & Model Training. Our probabilistic model takes crowd-contributed labels and the model trained in the previous iteration as input. As output, it generates a keyword-specific expectation, denoted as $e^{(t)}$, and an improved version of the micropost classification model, denoted as $p_{\theta ^{(t)}}(y|x)$. The details of our probabilistic model are given in Section SECREF3. Keyword Discovery. The keyword discovery task aims at discovering a new keyword (or a set of keywords) that is most informative for model training with respect to existing keywords. To this end, we first apply the current model $p_{\theta ^{(t)}}(y|x)$ on the unlabeled microposts $\mathcal {U}$. For those that contain the keyword $w^{(t)}$, we calculate the disagreement between the model predictions and the keyword-specific expectation $e^{(t)}$: and select the ones with the highest disagreement for keyword discovery. These selected microposts are supposed to contain information that can explain the disagreement between the model prediction and keyword-specific expectation, and can thus provide information that is most different from the existing set of keywords for model training. For instance, our study shows that the expectation for the keyword `hack' is 0.20, which means only 20% of the initial set of microposts retrieved with the keyword are event-related. A micropost selected with the highest disagreement (Eq. DISPLAY_FORM7), whose likelihood of being event-related as predicted by the model is $99.9\%$, is shown as an example below: RT @xxx: Hong Kong securities brokers hit by cyber attacks, may face more: regulator #cyber #security #hacking https://t.co/rC1s9CB This micropost contains keywords that can better indicate the relevance to a cyber security event than the initial keyword `hack', e.g., `securities', `hit', and `attack'. Note that when the keyword-specific expectation $e^{(t)}$ in Equation DISPLAY_FORM7 is high, the selected microposts will be the ones that contain keywords indicating the irrelevance of the microposts to an event category. Such keywords are also useful for model training as they help improve the model's ability to identify irrelevant microposts. To identify new keywords in the selected microposts, we again leverage crowdsourcing, as humans are typically better than machines at providing specific explanations BIBREF18, BIBREF19. In the crowdsourcing task, workers are first asked to find those microposts where the model predictions are deemed correct. Then, from those microposts, workers are asked to find the keyword that best indicates the class of the microposts as predicted by the model. The keyword most frequently identified by the workers is then used as the initial keyword for the following iteration. In case multiple keywords are selected, e.g., the top-$N$ frequent ones, workers will be asked to perform $N$ micropost classification tasks for each keyword in the next iteration, and the model training will be performed on multiple keyword-specific expectations. Unified Probabilistic Model This section introduces our probabilistic model that infers keyword expectation and trains the target model simultaneously. We start by formalizing the problem and introducing our model, before describing the model learning method. Problem Formalization. We consider the problem at iteration $t$ where the corresponding keyword is $w^{(t)}$. In the current iteration, let $\mathcal {U}^{(t)} \subset \mathcal {U}$ denote the set of all microposts containing the keyword and $\mathcal {M}^{(t)}= \lbrace x_{m}\rbrace _{m=1}^M\subset \mathcal {U}^{(t)}$ be the randomly selected subset of $M$ microposts labeled by $N$ crowd workers $\mathcal {C} = \lbrace c_n\rbrace _{n=1}^N$. The annotations form a matrix $\mathbf {A}\in \mathbb {R}^{M\times N}$ where $\mathbf {A}_{mn}$ is the label for the micropost $x_m$ contributed by crowd worker $c_n$. Our goal is to infer the keyword-specific expectation $e^{(t)}$ and train the target model by learning the model parameter $\theta ^{(t)}$. An additional parameter of our probabilistic model is the reliability of crowd workers, which is essential when involving crowdsourcing. Following Dawid and Skene BIBREF14, BIBREF16, we represent the annotation reliability of worker $c_n$ by a latent confusion matrix $\pi ^{(n)}$, where the $rs$-th element $\pi _{rs}^{(n)}$ denotes the probability of $c_n$ labeling a micropost as class $r$ given the true class $s$. Unified Probabilistic Model ::: Expectation as Model Posterior First, we introduce an expectation regularization technique for the weakly supervised learning of the target model $p_{\theta ^{(t)}}(y|x)$. In this setting, the objective function of the target model is composed of two parts, corresponding to the labeled microposts $\mathcal {L}$ and the unlabeled ones $\mathcal {U}$. The former part aims at maximizing the likelihood of the labeled microposts: where we assume that $\theta $ is generated from a prior distribution (e.g., Laplacian or Gaussian) parameterized by $\sigma $. To leverage unlabeled data for model training, we make use of the expectations of existing keywords, i.e., {($w^{(1)}$, $e^{(1)}$), ..., ($w^{(t-1)}$, $e^{(t-1)}$), ($w^{(t)}$, $e^{(t)}$)} (Note that $e^{(t)}$ is inferred), as a regularization term to constrain model training. To do so, we first give the model's expectation for each keyword $w^{(k)}$ ($1\le k\le t$) as follows: which denotes the empirical expectation of the model’s posterior predictions on the unlabeled microposts $\mathcal {U}^{(k)}$ containing keyword $w^{(k)}$. Expectation regularization can then be formulated as the regularization of the distance between the Bernoulli distribution parameterized by the model's expectation and the expectation of the existing keyword: where $D_{KL}[\cdot \Vert \cdot ]$ denotes the KL-divergence between the Bernoulli distributions $Ber(e^{(k)})$ and $Ber(\mathbb {E}_{x\sim \mathcal {U}^{(k)}}(y))$, and $\lambda $ controls the strength of expectation regularization. Unified Probabilistic Model ::: Expectation as Class Prior To learn the keyword-specific expectation $e^{(t)}$ and the crowd worker reliability $\pi ^{(n)}$ ($1\le n\le N$), we model the likelihood of the crowd-contributed labels $\mathbf {A}$ as a function of these parameters. In this context, we view the expectation as the class prior, thus performing expectation inference as the learning of the class prior. By doing so, we connect expectation inference with model training. Specifically, we model the likelihood of an arbitrary crowd-contributed label $\mathbf {A}_{mn}$ as a mixture of multinomials where the prior is the keyword-specific expectation $e^{(t)}$: where $e_s^{(t)}$ is the probability of the ground truth label being $s$ given the keyword-specific expectation as the class prior; $K$ is the set of possible ground truth labels (binary in our context); and $r=\mathbf {A}_{mn}$ is the crowd-contributed label. Then, for an individual micropost $x_m$, the likelihood of crowd-contributed labels $\mathbf {A}_{m:}$ is given by: Therefore, the objective function for maximizing the likelihood of the entire annotation matrix $\mathbf {A}$ can be described as: Unified Probabilistic Model ::: Unified Probabilistic Model Integrating model training with expectation inference, the overall objective function of our proposed model is given by: Figure FIGREF18 depicts a graphical representation of our model, which combines the target model for training (on the left) with the generative model for crowd-contributed labels (on the right) through a keyword-specific expectation. Model Learning. Due to the unknown ground truth labels of crowd-annotated microposts ($y_m$ in Figure FIGREF18), we resort to expectation maximization for model learning. The learning algorithm iteratively takes two steps: the E-step and the M-step. The E-step infers the ground truth labels given the current model parameters. The M-step updates the model parameters, including the crowd reliability parameters $\pi ^{(n)}$ ($1\le n\le N$), the keyword-specific expectation $e^{(t)}$, and the parameter of the target model $\theta ^{(t)}$. The E-step and the crowd parameter update in the M-step are similar to the Dawid-Skene model BIBREF14. The keyword expectation is inferred by taking into account both the crowd-contributed labels and the model prediction: The parameter of the target model is updated by gradient descent. For example, when the target model to be trained is a deep neural network, we use back-propagation with gradient descent to update the weight matrices. Experiments and Results This section presents our experimental setup and results for evaluating our approach. We aim at answering the following questions: [noitemsep,leftmargin=*] Q1: How effectively does our proposed human-AI loop approach enhance the state-of-the-art machine learning models for event detection? Q2: How well does our keyword discovery method work compare to existing keyword expansion methods? Q3: How effective is our approach using crowdsourcing at obtaining new keywords compared with an approach labelling microposts for model training under the same cost? Q4: How much benefit does our unified probabilistic model bring compared to methods that do not take crowd reliability into account? Experiments and Results ::: Experimental Setup Datasets. We perform our experiments with two predetermined event categories: cyber security (CyberAttack) and death of politicians (PoliticianDeath). These event categories are chosen as they are representative of important event types that are of interest to many governments and companies. The need to create our own dataset was motivated by the lack of public datasets for event detection on microposts. The few available datasets do not suit our requirements. For example, the publicly available Events-2012 Twitter dataset BIBREF20 contains generic event descriptions such as Politics, Sports, Culture etc. Our work targets more specific event categories BIBREF21. Following previous studies BIBREF1, we collect event-related microposts from Twitter using 11 and 8 seed events (see Section SECREF2) for CyberAttack and PoliticianDeath, respectively. Unlabeled microposts are collected by using the keyword `hack' for CyberAttack, while for PoliticianDeath, we use a set of keywords related to `politician' and `death' (such as `bureaucrat', `dead' etc.) For each dataset, we randomly select 500 tweets from the unlabeled subset and manually label them for evaluation. Table TABREF25 shows key statistics from our two datasets. Comparison Methods. To demonstrate the generality of our approach on different event detection models, we consider Logistic Regression (LR) BIBREF1 and Multilayer Perceptron (MLP) BIBREF2 as the target models. As the goal of our experiments is to demonstrate the effectiveness of our approach as a new model training technique, we use these widely used models. Also, we note that in our case other neural network models with more complex network architectures for event detection, such as the bi-directional LSTM BIBREF17, turn out to be less effective than a simple feedforward network. For both LR and MLP, we evaluate our proposed human-AI loop approach for keyword discovery and expectation estimation by comparing against the weakly supervised learning method proposed by BIBREF1 (BIBREF1) and BIBREF17 (BIBREF17) where only one initial keyword is used with an expectation estimated by an individual expert. Parameter Settings. We empirically set optimal parameters based on a held-out validation set that contains 20% of the test data. These include the hyperparamters of the target model, those of our proposed probabilistic model, and the parameters used for training the target model. We explore MLP with 1, 2 and 3 hidden layers and apply a grid search in 32, 64, 128, 256, 512 for the dimension of the embeddings and that of the hidden layers. For the coefficient of expectation regularization, we follow BIBREF6 (BIBREF6) and set it to $\lambda =10 \times $ #labeled examples. For model training, we use the Adam BIBREF22 optimization algorithm for both models. Evaluation. Following BIBREF1 (BIBREF1) and BIBREF3 (BIBREF3), we use accuracy and area under the precision-recall curve (AUC) metrics to measure the performance of our proposed approach. We note that due to the imbalance in our datasets (20% positive microposts in CyberAttack and 27% in PoliticianDeath), accuracy is dominated by negative examples; AUC, in comparison, better characterizes the discriminative power of the model. Crowdsourcing. We chose Level 3 workers on the Figure-Eight crowdsourcing platform for our experiments. The inter-annotator agreement in micropost classification is taken into account through the EM algorithm. For keyword discovery, we filter keywords based on the frequency of the keyword being selected by the crowd. In terms of cost-effectiveness, our approach is motivated from the fact that crowdsourced data annotation can be expensive, and is thus designed with minimal crowd involvement. For each iteration, we selected 50 tweets for keyword discovery and 50 tweets for micropost classification per keyword. For a dataset with 80k tweets (e.g., CyberAttack), our approach only requires to manually inspect 800 tweets (for 8 keywords), which is only 1% of the entire dataset. Experiments and Results ::: Results of our Human-AI Loop (Q1) Table TABREF26 reports the evaluation of our approach on both the CyberAttack and PoliticianDeath event categories. Our approach is configured such that each iteration starts with 1 new keyword discovered in the previous iteration. Our approach improves LR by 5.17% (Accuracy) and 18.38% (AUC), and MLP by 10.71% (Accuracy) and 30.27% (AUC) on average. Such significant improvements clearly demonstrate that our approach is effective at improving model performance. We observe that the target models generally converge between the 7th and 9th iteration on both datasets when performance is measured by AUC. The performance can slightly degrade when the models are further trained for more iterations on both datasets. This is likely due to the fact that over time, the newly discovered keywords entail lower novel information for model training. For instance, for the CyberAttack dataset the new keyword in the 9th iteration `election' frequently co-occurs with the keyword `russia' in the 5th iteration (in microposts that connect Russian hackers with US elections), thus bringing limited new information for improving the model performance. As a side remark, we note that the models converge faster when performance is measured by accuracy. Such a comparison result confirms the difference between the metrics and shows the necessity for more keywords to discriminate event-related microposts from non event-related ones. Experiments and Results ::: Comparative Results on Keyword Discovery (Q2) Figure FIGREF31 shows the evaluation of our approach when discovering new informative keywords for model training (see Section SECREF2: Keyword Discovery). We compare our human-AI collaborative way of discovering new keywords against a query expansion (QE) approach BIBREF23, BIBREF24 that leverages word embeddings to find similar words in the latent semantic space. Specifically, we use pre-trained word embeddings based on a large Google News dataset for query expansion. For instance, the top keywords resulting from QE for `politician' are, `deputy',`ministry',`secretary', and `minister'. For each of these keywords, we use the crowd to label a set of tweets and obtain a corresponding expectation. We observe that our approach consistently outperforms QE by an average of $4.62\%$ and $52.58\%$ AUC on CyberAttack and PoliticianDeath, respectively. The large gap between the performance improvements for the two datasets is mainly due to the fact that microposts that are relevant for PoliticianDeath are semantically more complex than those for CyberAttack, as they encode noun-verb relationship (e.g., “the king of ... died ...”) rather than a simple verb (e.g., “... hacked.”) for the CyberAttack microposts. QE only finds synonyms of existing keywords related to either `politician' or `death', however cannot find a meaningful keyword that fully characterizes the death of a politician. For instance, QE finds the keywords `kill' and `murder', which are semantically close to `death' but are not specifically relevant to the death of a politician. Unlike QE, our approach identifies keywords that go beyond mere synonyms and that are more directly related to the end task, i.e., discriminating event-related microposts from non related ones. Examples are `demise' and `condolence'. As a remark, we note that in Figure FIGREF31(b), the increase in QE performance on PoliticianDeath is due to the keywords `deputy' and `minister', which happen to be highly indicative of the death of a politician in our dataset; these keywords are also identified by our approach. Experiments and Results ::: Cost-Effectiveness Results (Q3) To demonstrate the cost-effectiveness of using crowdsourcing for obtaining new keywords and consequently, their expectations, we compare the performance of our approach with an approach using crowdsourcing to only label microposts for model training at the same cost. Specifically, we conducted an additional crowdsourcing experiment where the same cost used for keyword discovery in our approach is used to label additional microposts for model training. These newly labeled microposts are used with the microposts labeled in the micropost classification task of our approach (see Section SECREF2: Micropost Classification) and the expectation of the initial keyword to train the model for comparison. The model trained in this way increases AUC by 0.87% for CyberAttack, and by 1.06% for PoliticianDeath; in comparison, our proposed approach increases AUC by 33.42% for PoliticianDeath and by 15.23% for CyberAttack over the baseline presented by BIBREF1). These results show that using crowdsourcing for keyword discovery is significantly more cost-effective than simply using crowdsourcing to get additional labels when training the model. Experiments and Results ::: Expectation Inference Results (Q4) To investigate the effectiveness of our expectation inference method, we compare it against a majority voting approach, a strong baseline in truth inference BIBREF16. Figure FIGREF36 shows the result of this evaluation. We observe that our approach results in better models for both CyberAttack and PoliticianDeath. Our manual investigation reveals that workers' annotations are of high reliability, which explains the relatively good performance of majority voting. Despite limited margin for improvement, our method of expectation inference improves the performance of majority voting by $0.4\%$ and $1.19\%$ AUC on CyberAttack and PoliticianDeath, respectively. Related Work Event Detection. The techniques for event extraction from microblogging platforms can be classified according to their domain specificity and their detection method BIBREF0. Early works mainly focus on open domain event detection BIBREF25, BIBREF26, BIBREF27. Our work falls into the category of domain-specific event detection BIBREF21, which has drawn increasing attention due to its relevance for various applications such as cyber security BIBREF1, BIBREF2 and public health BIBREF4, BIBREF5. In terms of technique, our proposed detection method is related to the recently proposed weakly supervised learning methods BIBREF1, BIBREF17, BIBREF3. This comes in contrast with fully-supervised learning methods, which are often limited by the size of the training data (e.g., a few hundred examples) BIBREF28, BIBREF29. Human-in-the-Loop Approaches. Our work extends weakly supervised learning methods by involving humans in the loop BIBREF13. Existing human-in-the-loop approaches mainly leverage crowds to label individual data instances BIBREF9, BIBREF10 or to debug the training data BIBREF30, BIBREF31 or components BIBREF32, BIBREF33, BIBREF34 of a machine learning system. Unlike these works, we leverage crowd workers to label sampled microposts in order to obtain keyword-specific expectations, which can then be generalized to help classify microposts containing the same keyword, thus amplifying the utility of the crowd. Our work is further connected to the topic of interpretability and transparency of machine learning models BIBREF11, BIBREF35, BIBREF12, for which humans are increasingly involved, for instance for post-hoc evaluations of the model's interpretability. In contrast, our approach directly solicits informative keywords from the crowd for model training, thereby providing human-understandable explanations for the improved model. Conclusion In this paper, we presented a new human-AI loop approach for keyword discovery and expectation estimation to better train event detection models. Our approach takes advantage of the disagreement between the crowd and the model to discover informative keywords and leverages the joint power of the crowd and the model in expectation inference. We evaluated our approach on real-world datasets and showed that it significantly outperforms the state of the art and that it is particularly useful for detecting events where relevant microposts are semantically complex, e.g., the death of a politician. As future work, we plan to parallelize the crowdsourcing tasks and optimize our pipeline in order to use our event detection approach in real-time. Acknowledgements This project has received funding from the Swiss National Science Foundation (grant #407540_167320 Tighten-it-All) and from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement 683253/GraphInt).
How are the interpretability merits of the approach demonstrated?
67e9e147b2cab5ba43572ce8a17fc863690172f0
How Ġare Ġthe Ġinterpret ability Ġmerits Ġof Ġthe Ġapproach Ġdemonstrated ?
By Ġinvolving Ġhumans Ġfor Ġpost - h oc Ġevaluation Ġof Ġmodel 's Ġinterpret ability
<s> How Ġare Ġthe Ġinterpret ability Ġmerits Ġof Ġthe Ġapproach Ġdemonstrated ? </s> Introduction </s> Event Ġdetection Ġon Ġmicro blog ging Ġplatforms Ġsuch Ġas ĠTwitter Ġaims Ġto Ġdetect Ġevents Ġpreempt ively . ĠA Ġmain Ġtask Ġin Ġevent Ġdetection Ġis Ġdetecting Ġevents Ġof Ġpredetermined Ġtypes ĠB IB REF 0 , Ġsuch Ġas Ġconcerts Ġor Ġcontroversial Ġevents Ġbased Ġon Ġmic rop ost s Ġmatching Ġspecific Ġevent Ġdescriptions . ĠThis Ġtask Ġhas Ġextensive Ġapplications Ġranging Ġfrom Ġcyber Ġsecurity ĠB IB REF 1 , ĠB IB REF 2 Ġto Ġpolitical Ġelections ĠB IB REF 3 Ġor Ġpublic Ġhealth ĠB IB REF 4 , ĠB IB REF 5 . ĠDue Ġto Ġthe Ġhigh Ġambiguity Ġand Ġinconsistency Ġof Ġthe Ġterms Ġused Ġin Ġmic rop ost s , Ġevent Ġdetection Ġis Ġgenerally Ġperformed Ġthough Ġstatistical Ġmachine Ġlearning Ġmodels , Ġwhich Ġrequire Ġa Ġlabeled Ġdataset Ġfor Ġmodel Ġtraining . ĠData Ġlabeling Ġis , Ġhowever , Ġa Ġlong , Ġlabor ious , Ġand Ġusually Ġcostly Ġprocess . ĠFor Ġthe Ġcase Ġof Ġmic rop ost Ġclassification , Ġthough Ġpositive Ġlabels Ġcan Ġbe Ġcollected Ġ( e . g ., Ġusing Ġspecific Ġhasht ags , Ġor Ġevent - related Ġdate - time Ġinformation ), Ġthere Ġis Ġno Ġstraightforward Ġway Ġto Ġgenerate Ġnegative Ġlabels Ġuseful Ġfor Ġmodel Ġtraining . ĠTo Ġtackle Ġthis Ġlack Ġof Ġnegative Ġlabels Ġand Ġthe Ġsignificant Ġmanual Ġefforts Ġin Ġdata Ġlabeling , ĠB IB REF 1 Ġ( BI B REF 1 , ĠB IB REF 3 ) Ġintroduced Ġa Ġweak Ġsupervision Ġbased Ġlearning Ġapproach , Ġwhich Ġuses Ġonly Ġpositively Ġlabeled Ġdata , Ġaccompanied Ġby Ġunl abel ed Ġexamples Ġby Ġfiltering Ġmic rop ost s Ġthat Ġcontain Ġa Ġcertain Ġkeyword Ġindicative Ġof Ġthe Ġevent Ġtype Ġunder Ġconsideration Ġ( e . g ., Ġ` hack ' Ġfor Ġcyber Ġsecurity ). ĠAnother Ġkey Ġtechnique Ġin Ġthis Ġcontext Ġis Ġexpectation Ġregular ization ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 1 . ĠHere , Ġthe Ġestimated Ġproportion Ġof Ġrelevant Ġmic rop ost s Ġin Ġan Ġunl abel ed Ġdataset Ġcontaining Ġa Ġkeyword Ġis Ġgiven Ġas Ġa Ġkeyword - specific Ġexpectation . ĠThis Ġexpectation Ġis Ġused Ġin Ġthe Ġregular ization Ġterm Ġof Ġthe Ġmodel 's Ġobjective Ġfunction Ġto Ġconst rain Ġthe Ġposterior Ġdistribution Ġof Ġthe Ġmodel Ġpredictions . ĠBy Ġdoing Ġso , Ġthe Ġmodel Ġis Ġtrained Ġwith Ġan Ġexpectation Ġon Ġits Ġprediction Ġfor Ġmic rop ost s Ġthat Ġcontain Ġthe Ġkeyword . ĠSuch Ġa Ġmethod , Ġhowever , Ġsuffers Ġfrom Ġtwo Ġkey Ġproblems : </s> Due Ġto Ġthe Ġunpredict ability Ġof Ġevent Ġoccurrences Ġand Ġthe Ġconstantly Ġchanging Ġdynamics Ġof Ġusers ' Ġposting Ġfrequency ĠB IB REF 8 , Ġestimating Ġthe Ġexpectation Ġassociated Ġwith Ġa Ġkeyword Ġis Ġa Ġchallenging Ġtask , Ġeven Ġfor Ġdomain Ġexperts ; </s> The Ġperformance Ġof Ġthe Ġevent Ġdetection Ġmodel Ġis Ġconstrained Ġby Ġthe Ġinform at iveness Ġof Ġthe Ġkeyword Ġused Ġfor Ġmodel Ġtraining . ĠAs Ġof Ġnow , Ġwe Ġlack Ġa Ġprincipled Ġmethod Ġfor Ġdiscovering Ġnew Ġkeywords Ġand Ġimprove Ġthe Ġmodel Ġperformance . </s> To Ġaddress Ġthe Ġabove Ġissues , Ġwe Ġadvocate Ġa Ġhuman - AI Ġloop Ġapproach Ġfor Ġdiscovering Ġinformative Ġkeywords Ġand Ġestimating Ġtheir Ġexpectations Ġreliably . ĠOur Ġapproach Ġiter atively Ġlever ages Ġ1 ) Ġcrowd Ġworkers Ġfor Ġestimating Ġkeyword - specific Ġexpectations , Ġand Ġ2 ) Ġthe Ġdisagreement Ġbetween Ġthe Ġmodel Ġand Ġthe Ġcrowd Ġfor Ġdiscovering Ġnew Ġinformative Ġkeywords . ĠMore Ġspecifically , Ġat Ġeach Ġiteration Ġafter Ġwe Ġobtain Ġa Ġkeyword - specific Ġexpectation Ġfrom Ġthe Ġcrowd , Ġwe Ġtrain Ġthe Ġmodel Ġusing Ġexpectation Ġregular ization Ġand Ġselect Ġthose Ġkeyword - related Ġmic rop ost s Ġfor Ġwhich Ġthe Ġmodel 's Ġprediction Ġdisagrees Ġthe Ġmost Ġwith Ġthe Ġcrowd 's Ġexpectation ; Ġsuch Ġmic rop ost s Ġare Ġthen Ġpresented Ġto Ġthe Ġcrowd Ġto Ġidentify Ġnew Ġkeywords Ġthat Ġbest Ġexplain Ġthe Ġdisagreement . ĠBy Ġdoing Ġso , Ġour Ġapproach Ġidentifies Ġnew Ġkeywords Ġwhich Ġconvey Ġmore Ġrelevant Ġinformation Ġwith Ġrespect Ġto Ġexisting Ġones , Ġthus Ġeffectively Ġboosting Ġmodel Ġperformance . ĠBy Ġexploiting Ġthe Ġdisagreement Ġbetween Ġthe Ġmodel Ġand Ġthe Ġcrowd , Ġour Ġapproach Ġcan Ġmake Ġefficient Ġuse Ġof Ġthe Ġcrowd , Ġwhich Ġis Ġof Ġcritical Ġimportance Ġin Ġa Ġhuman - in - the - loop Ġcontext ĠB IB REF 9 , ĠB IB REF 10 . ĠAn Ġadditional Ġadvantage Ġof Ġour Ġapproach Ġis Ġthat Ġby Ġobtaining Ġnew Ġkeywords Ġthat Ġimprove Ġmodel Ġperformance Ġover Ġtime , Ġwe Ġare Ġable Ġto Ġgain Ġinsight Ġinto Ġhow Ġthe Ġmodel Ġlearns Ġfor Ġspecific Ġevent Ġdetection Ġtasks . ĠSuch Ġan Ġadvantage Ġis Ġparticularly Ġuseful Ġfor Ġevent Ġdetection Ġusing Ġcomplex Ġmodels , Ġe . g ., Ġdeep Ġneural Ġnetworks , Ġwhich Ġare Ġintrinsically Ġhard Ġto Ġunderstand ĠB IB REF 11 , ĠB IB REF 12 . ĠAn Ġadditional Ġchallenge Ġin Ġinvolving Ġcrowd Ġworkers Ġis Ġthat Ġtheir Ġcontributions Ġare Ġnot Ġfully Ġreliable ĠB IB REF 13 . ĠIn Ġthe Ġcrowds ourcing Ġliterature , Ġthis Ġproblem Ġis Ġusually Ġtackled Ġwith Ġprob abil istic Ġlatent Ġvariable Ġmodels ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 , Ġwhich Ġare Ġused Ġto Ġperform Ġtruth Ġinference Ġby Ġaggreg ating Ġa Ġredundant Ġset Ġof Ġcrowd Ġcontributions . ĠOur Ġhuman - AI Ġloop Ġapproach Ġimproves Ġthe Ġinference Ġof Ġkeyword Ġexpectation Ġby Ġaggreg ating Ġcontributions Ġnot Ġonly Ġfrom Ġthe Ġcrowd Ġbut Ġalso Ġfrom Ġthe Ġmodel . ĠThis , Ġhowever , Ġcomes Ġwith Ġits Ġown Ġchallenge Ġas Ġthe Ġmodel 's Ġpredictions Ġare Ġfurther Ġdependent Ġon Ġthe Ġresults Ġof Ġexpectation Ġinference , Ġwhich Ġis Ġused Ġfor Ġmodel Ġtraining . ĠTo Ġaddress Ġthis Ġproblem , Ġwe Ġintroduce Ġa Ġunified Ġprob abil istic Ġmodel Ġthat Ġseamlessly Ġintegrates Ġexpectation Ġinference Ġand Ġmodel Ġtraining , Ġthereby Ġallowing Ġthe Ġformer Ġto Ġbenefit Ġfrom Ġthe Ġlatter Ġwhile Ġresolving Ġthe Ġinter - depend ency Ġbetween Ġthe Ġtwo . </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwe Ġare Ġthe Ġfirst Ġto Ġpropose Ġa Ġhuman - AI Ġloop Ġapproach Ġthat Ġiter atively Ġimproves Ġmachine Ġlearning Ġmodels Ġfor Ġevent Ġdetection . ĠIn Ġsummary , Ġour Ġwork Ġmakes Ġthe Ġfollowing Ġkey Ġcontributions : </s> A Ġnovel Ġhuman - AI Ġloop Ġapproach Ġfor Ġmic rop ost Ġevent Ġdetection Ġthat Ġjointly Ġdiscovers Ġinformative Ġkeywords Ġand Ġestimates Ġtheir Ġexpectation ; </s> A Ġunified Ġprob abil istic Ġmodel Ġthat Ġinf ers Ġkeyword Ġexpectation Ġand Ġsimultaneously Ġperforms Ġmodel Ġtraining ; </s> An Ġextensive Ġempirical Ġevaluation Ġof Ġour Ġapproach Ġon Ġmultiple Ġreal - world Ġdatasets Ġdemonstrating Ġthat Ġour Ġapproach Ġsignificantly Ġimproves Ġthe Ġstate Ġof Ġthe Ġart Ġby Ġan Ġaverage Ġof Ġ24 . 3 % ĠA UC . </s> The Ġrest Ġof Ġthis Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠFirst , Ġwe Ġpresent Ġour Ġhuman - AI Ġloop Ġapproach Ġin ĠSection ĠSEC REF 2 . ĠSub sequently , Ġwe Ġintroduce Ġour Ġproposed Ġprob abil istic Ġmodel Ġin ĠSection ĠSEC REF 3 . ĠThe Ġexperimental Ġsetup Ġand Ġresults Ġare Ġpresented Ġin ĠSection ĠSEC REF 4 . ĠFinally , Ġwe Ġbriefly Ġcover Ġrelated Ġwork Ġin ĠSection ĠSEC REF 5 Ġbefore Ġconcluding Ġour Ġwork Ġin ĠSection ĠSEC REF 6 . </s> The ĠHuman - AI ĠLoop ĠApproach </s> Given Ġa Ġset Ġof Ġlabeled Ġand Ġunl abel ed Ġmic rop ost s , Ġour Ġgoal Ġis Ġto Ġextract Ġinformative Ġkeywords Ġand Ġestimate Ġtheir Ġexpectations Ġin Ġorder Ġto Ġtrain Ġa Ġmachine Ġlearning Ġmodel . ĠTo Ġachieve Ġthis Ġgoal , Ġour Ġproposed Ġhuman - AI Ġloop Ġapproach Ġcomprises Ġtwo Ġcrowds ourcing Ġtasks , Ġi . e ., Ġmic rop ost Ġclassification Ġfollowed Ġby Ġkeyword Ġdiscovery , Ġand Ġa Ġunified Ġprob abil istic Ġmodel Ġfor Ġexpectation Ġinference Ġand Ġmodel Ġtraining . ĠFigure ĠFIG REF 6 Ġpresents Ġan Ġoverview Ġof Ġour Ġapproach . ĠNext , Ġwe Ġdescribe Ġour Ġapproach Ġfrom Ġa Ġprocess - centric Ġperspective . </s> Following Ġprevious Ġstudies ĠB IB REF 1 , ĠB IB REF 17 , ĠB IB REF 2 , Ġwe Ġcollect Ġa Ġset Ġof Ġunl abel ed Ġmic rop ost s Ġ$\ math cal Ġ{ U } $ Ġfrom Ġa Ġmicro blog ging Ġplatform Ġand Ġpost - filter , Ġusing Ġan Ġinitial Ġ( set Ġof ) Ġkeyword ( s ), Ġthose Ġmic rop ost s Ġthat Ġare Ġpotentially Ġrelevant Ġto Ġan Ġevent Ġcategory . ĠThen , Ġwe Ġcollect Ġa Ġset Ġof Ġevent - related Ġmic rop ost s Ġ( i . e ., Ġpositively Ġlabeled Ġmic rop ost s ) Ġ$\ math cal Ġ{ L } $, Ġpost - fil tering Ġwith Ġa Ġlist Ġof Ġseed Ġevents . Ġ$\ math cal Ġ{ U } $ Ġand Ġ$\ math cal Ġ{ L } $ Ġare Ġused Ġtogether Ġto Ġtrain Ġa Ġdiscrim in ative Ġmodel Ġ( e . g ., Ġa Ġdeep Ġneural Ġnetwork ) Ġfor Ġclass ifying Ġthe Ġrelevance Ġof Ġmic rop ost s Ġto Ġan Ġevent . ĠWe Ġdenote Ġthe Ġtarget Ġmodel Ġas Ġ$ p _ \ the ta Ġ( y | x ) $, Ġwhere Ġ$\ the ta Ġ$ Ġis Ġthe Ġmodel Ġparameter Ġto Ġbe Ġlearned Ġand Ġ$ y $ Ġis Ġthe Ġlabel Ġof Ġan Ġarbitrary Ġmic rop ost , Ġrepresented Ġby Ġa Ġbag - of - words Ġvector Ġ$ x $. ĠOur Ġapproach Ġiter ates Ġseveral Ġtimes Ġ$ t = \ l brace Ġ1 , Ġ2 , Ġ\ ld ots Ġ\ r brace Ġ$ Ġuntil Ġthe Ġperformance Ġof Ġthe Ġtarget Ġmodel Ġconver ges . ĠEach Ġiteration Ġstarts Ġfrom Ġthe Ġinitial Ġkeyword ( s ) Ġor Ġthe Ġnew Ġkeyword ( s ) Ġdiscovered Ġin Ġthe Ġprevious Ġiteration . ĠGiven Ġsuch Ġa Ġkeyword , Ġden oted Ġby Ġ$ w ^{ ( t )} $, Ġthe Ġiteration Ġstarts Ġby Ġsampling Ġmic rop ost s Ġcontaining Ġthe Ġkeyword Ġfrom Ġ$\ math cal Ġ{ U } $, Ġfollowed Ġby Ġdynamically Ġcreating Ġmic rop ost Ġclassification Ġtasks Ġand Ġpublishing Ġthem Ġon Ġa Ġcrowds ourcing Ġplatform . </s> Mic rop ost ĠClassification . ĠThe Ġmic rop ost Ġclassification Ġtask Ġrequires Ġcrowd Ġworkers Ġto Ġlabel Ġthe Ġselected Ġmic rop ost s Ġinto Ġtwo Ġclasses : Ġevent - related Ġand Ġnon Ġevent - related . ĠIn Ġparticular , Ġworkers Ġare Ġgiven Ġinstructions Ġand Ġexamples Ġto Ġdifferentiate Ġevent - instance Ġrelated Ġmic rop ost s Ġand Ġgeneral Ġevent - category Ġrelated Ġmic rop ost s . ĠConsider , Ġfor Ġexample , Ġthe Ġfollowing Ġmic rop ost s Ġin Ġthe Ġcontext Ġof ĠCyber Ġattack Ġevents , Ġboth Ġcontaining Ġthe Ġkeyword Ġ` hack ': </s> Credit Ġfirm ĠEqu ifax Ġsays Ġ143 m ĠAmericans ' Ġsocial Ġsecurity Ġnumbers Ġexposed Ġin Ġhack </s> This Ġmic rop ost Ġdescribes Ġan Ġinstance Ġof Ġa Ġcyber Ġattack Ġevent Ġthat Ġthe Ġtarget Ġmodel Ġshould Ġidentify . ĠThis Ġis , Ġtherefore , Ġan Ġevent - instance Ġrelated Ġmic rop ost Ġand Ġshould Ġbe Ġconsidered Ġas Ġa Ġpositive Ġexample . ĠContrast Ġthis Ġwith Ġthe Ġfollowing Ġexample : </s> Companies Ġneed Ġto Ġstep Ġtheir Ġcyber Ġsecurity Ġup </s> This Ġmic rop ost , Ġthough Ġrelated Ġto Ġcyber Ġsecurity Ġin Ġgeneral , Ġdoes Ġnot Ġmention Ġan Ġinstance Ġof Ġa Ġcyber Ġattack Ġevent , Ġand Ġis Ġof Ġno Ġinterest Ġto Ġus Ġfor Ġevent Ġdetection . ĠThis Ġis Ġan Ġexample Ġof Ġa Ġgeneral Ġevent - category Ġrelated Ġmic rop ost Ġand Ġshould Ġbe Ġconsidered Ġas Ġa Ġnegative Ġexample . </s> In Ġthis Ġtask , Ġeach Ġselected Ġmic rop ost Ġis Ġlabeled Ġby Ġmultiple Ġcrowd Ġworkers . ĠThe Ġannotations Ġare Ġpassed Ġto Ġour Ġprob abil istic Ġmodel Ġfor Ġexpectation Ġinference Ġand Ġmodel Ġtraining . </s> Ex pect ation ĠIn ference Ġ& ĠModel ĠTraining . ĠOur Ġprob abil istic Ġmodel Ġtakes Ġcrowd - cont ributed Ġlabels Ġand Ġthe Ġmodel Ġtrained Ġin Ġthe Ġprevious Ġiteration Ġas Ġinput . ĠAs Ġoutput , Ġit Ġgenerates Ġa Ġkeyword - specific Ġexpectation , Ġden oted Ġas Ġ$ e ^{ ( t )} $, Ġand Ġan Ġimproved Ġversion Ġof Ġthe Ġmic rop ost Ġclassification Ġmodel , Ġden oted Ġas Ġ$ p _{ \ the ta Ġ^ { ( t ) }} ( y | x ) $. ĠThe Ġdetails Ġof Ġour Ġprob abil istic Ġmodel Ġare Ġgiven Ġin ĠSection ĠSEC REF 3 . </s> Key word ĠDiscovery . ĠThe Ġkeyword Ġdiscovery Ġtask Ġaims Ġat Ġdiscovering Ġa Ġnew Ġkeyword Ġ( or Ġa Ġset Ġof Ġkeywords ) Ġthat Ġis Ġmost Ġinformative Ġfor Ġmodel Ġtraining Ġwith Ġrespect Ġto Ġexisting Ġkeywords . ĠTo Ġthis Ġend , Ġwe Ġfirst Ġapply Ġthe Ġcurrent Ġmodel Ġ$ p _{ \ the ta Ġ^ { ( t ) }} ( y | x ) $ Ġon Ġthe Ġunl abel ed Ġmic rop ost s Ġ$\ math cal Ġ{ U } $. ĠFor Ġthose Ġthat Ġcontain Ġthe Ġkeyword Ġ$ w ^{ ( t )} $, Ġwe Ġcalculate Ġthe Ġdisagreement Ġbetween Ġthe Ġmodel Ġpredictions Ġand Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $ : </s> and Ġselect Ġthe Ġones Ġwith Ġthe Ġhighest Ġdisagreement Ġfor Ġkeyword Ġdiscovery . ĠThese Ġselected Ġmic rop ost s Ġare Ġsupposed Ġto Ġcontain Ġinformation Ġthat Ġcan Ġexplain Ġthe Ġdisagreement Ġbetween Ġthe Ġmodel Ġprediction Ġand Ġkeyword - specific Ġexpectation , Ġand Ġcan Ġthus Ġprovide Ġinformation Ġthat Ġis Ġmost Ġdifferent Ġfrom Ġthe Ġexisting Ġset Ġof Ġkeywords Ġfor Ġmodel Ġtraining . </s> For Ġinstance , Ġour Ġstudy Ġshows Ġthat Ġthe Ġexpectation Ġfor Ġthe Ġkeyword Ġ` hack ' Ġis Ġ0 . 20 , Ġwhich Ġmeans Ġonly Ġ20 % Ġof Ġthe Ġinitial Ġset Ġof Ġmic rop ost s Ġretrieved Ġwith Ġthe Ġkeyword Ġare Ġevent - related . ĠA Ġmic rop ost Ġselected Ġwith Ġthe Ġhighest Ġdisagreement Ġ( E q . ĠDIS PLAY _ FORM 7 ), Ġwhose Ġlikelihood Ġof Ġbeing Ġevent - related Ġas Ġpredicted Ġby Ġthe Ġmodel Ġis Ġ$ 99 . 9 \ % $, Ġis Ġshown Ġas Ġan Ġexample Ġbelow : </s> RT Ġ@ xxx : ĠHong ĠKong Ġsecurities Ġbrokers Ġhit Ġby Ġcyber Ġattacks , Ġmay Ġface Ġmore : Ġregulator Ġ# cy ber Ġ# security Ġ# h acking Ġhttps :// t . co / r C 1 s 9 CB </s> This Ġmic rop ost Ġcontains Ġkeywords Ġthat Ġcan Ġbetter Ġindicate Ġthe Ġrelevance Ġto Ġa Ġcyber Ġsecurity Ġevent Ġthan Ġthe Ġinitial Ġkeyword Ġ` hack ', Ġe . g ., Ġ` sec urities ', Ġ` hit ', Ġand Ġ` attack '. </s> Note Ġthat Ġwhen Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 7 Ġis Ġhigh , Ġthe Ġselected Ġmic rop ost s Ġwill Ġbe Ġthe Ġones Ġthat Ġcontain Ġkeywords Ġindicating Ġthe Ġirre lev ance Ġof Ġthe Ġmic rop ost s Ġto Ġan Ġevent Ġcategory . ĠSuch Ġkeywords Ġare Ġalso Ġuseful Ġfor Ġmodel Ġtraining Ġas Ġthey Ġhelp Ġimprove Ġthe Ġmodel 's Ġability Ġto Ġidentify Ġirrelevant Ġmic rop ost s . </s> To Ġidentify Ġnew Ġkeywords Ġin Ġthe Ġselected Ġmic rop ost s , Ġwe Ġagain Ġleverage Ġcrowds ourcing , Ġas Ġhumans Ġare Ġtypically Ġbetter Ġthan Ġmachines Ġat Ġproviding Ġspecific Ġexplanations ĠB IB REF 18 , ĠB IB REF 19 . ĠIn Ġthe Ġcrowds ourcing Ġtask , Ġworkers Ġare Ġfirst Ġasked Ġto Ġfind Ġthose Ġmic rop ost s Ġwhere Ġthe Ġmodel Ġpredictions Ġare Ġdeemed Ġcorrect . ĠThen , Ġfrom Ġthose Ġmic rop ost s , Ġworkers Ġare Ġasked Ġto Ġfind Ġthe Ġkeyword Ġthat Ġbest Ġindicates Ġthe Ġclass Ġof Ġthe Ġmic rop ost s Ġas Ġpredicted Ġby Ġthe Ġmodel . ĠThe Ġkeyword Ġmost Ġfrequently Ġidentified Ġby Ġthe Ġworkers Ġis Ġthen Ġused Ġas Ġthe Ġinitial Ġkeyword Ġfor Ġthe Ġfollowing Ġiteration . ĠIn Ġcase Ġmultiple Ġkeywords Ġare Ġselected , Ġe . g ., Ġthe Ġtop -$ N $ Ġfrequent Ġones , Ġworkers Ġwill Ġbe Ġasked Ġto Ġperform Ġ$ N $ Ġmic rop ost Ġclassification Ġtasks Ġfor Ġeach Ġkeyword Ġin Ġthe Ġnext Ġiteration , Ġand Ġthe Ġmodel Ġtraining Ġwill Ġbe Ġperformed Ġon Ġmultiple Ġkeyword - specific Ġexpectations . </s> Un ified ĠProb abil istic ĠModel </s> This Ġsection Ġintroduces Ġour Ġprob abil istic Ġmodel Ġthat Ġinf ers Ġkeyword Ġexpectation Ġand Ġtrains Ġthe Ġtarget Ġmodel Ġsimultaneously . ĠWe Ġstart Ġby Ġformal izing Ġthe Ġproblem Ġand Ġintroducing Ġour Ġmodel , Ġbefore Ġdescribing Ġthe Ġmodel Ġlearning Ġmethod . </s> Problem ĠForm al ization . ĠWe Ġconsider Ġthe Ġproblem Ġat Ġiteration Ġ$ t $ Ġwhere Ġthe Ġcorresponding Ġkeyword Ġis Ġ$ w ^{ ( t )} $. ĠIn Ġthe Ġcurrent Ġiteration , Ġlet Ġ$\ math cal Ġ{ U } ^{ ( t )} Ġ\ sub set Ġ\ math cal Ġ{ U } $ Ġdenote Ġthe Ġset Ġof Ġall Ġmic rop ost s Ġcontaining Ġthe Ġkeyword Ġand Ġ$\ math cal Ġ{ M } ^{ ( t )} = Ġ\ l brace Ġx _{ m }\ r brace Ġ_ { m = 1 } ^ M \ sub set Ġ\ math cal Ġ{ U } ^{ ( t )} $ Ġbe Ġthe Ġrandomly Ġselected Ġsubset Ġof Ġ$ M $ Ġmic rop ost s Ġlabeled Ġby Ġ$ N $ Ġcrowd Ġworkers Ġ$\ math cal Ġ{ C } Ġ= Ġ\ l brace Ġc _ n \ r brace Ġ_ { n = 1 } ^ N $. ĠThe Ġannotations Ġform Ġa Ġmatrix Ġ$\ math bf Ġ{ A }\ in Ġ\ math bb Ġ{ R } ^{ M \ times ĠN } $ Ġwhere Ġ$\ math bf Ġ{ A } _{ mn } $ Ġis Ġthe Ġlabel Ġfor Ġthe Ġmic rop ost Ġ$ x _ m $ Ġcontributed Ġby Ġcrowd Ġworker Ġ$ c _ n $. ĠOur Ġgoal Ġis Ġto Ġinfer Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $ Ġand Ġtrain Ġthe Ġtarget Ġmodel Ġby Ġlearning Ġthe Ġmodel Ġparameter Ġ$\ the ta Ġ^ { ( t )} $. ĠAn Ġadditional Ġparameter Ġof Ġour Ġprob abil istic Ġmodel Ġis Ġthe Ġreliability Ġof Ġcrowd Ġworkers , Ġwhich Ġis Ġessential Ġwhen Ġinvolving Ġcrowds ourcing . ĠFollowing ĠDaw id Ġand ĠSk ene ĠB IB REF 14 , ĠB IB REF 16 , Ġwe Ġrepresent Ġthe Ġannotation Ġreliability Ġof Ġworker Ġ$ c _ n $ Ġby Ġa Ġlatent Ġconfusion Ġmatrix Ġ$\ pi Ġ^ { ( n )} $, Ġwhere Ġthe Ġ$ rs $ - th Ġelement Ġ$\ pi Ġ_ { rs } ^{ ( n )} $ Ġdenotes Ġthe Ġprobability Ġof Ġ$ c _ n $ Ġlabeling Ġa Ġmic rop ost Ġas Ġclass Ġ$ r $ Ġgiven Ġthe Ġtrue Ġclass Ġ$ s $. </s> Un ified ĠProb abil istic ĠModel Ġ: :: ĠExpect ation Ġas ĠModel ĠPoster ior </s> First , Ġwe Ġintroduce Ġan Ġexpectation Ġregular ization Ġtechnique Ġfor Ġthe Ġweak ly Ġsupervised Ġlearning Ġof Ġthe Ġtarget Ġmodel Ġ$ p _{ \ the ta Ġ^ { ( t ) }} ( y | x ) $. ĠIn Ġthis Ġsetting , Ġthe Ġobjective Ġfunction Ġof Ġthe Ġtarget Ġmodel Ġis Ġcomposed Ġof Ġtwo Ġparts , Ġcorresponding Ġto Ġthe Ġlabeled Ġmic rop ost s Ġ$\ math cal Ġ{ L } $ Ġand Ġthe Ġunl abel ed Ġones Ġ$\ math cal Ġ{ U } $. </s> The Ġformer Ġpart Ġaims Ġat Ġmaximizing Ġthe Ġlikelihood Ġof Ġthe Ġlabeled Ġmic rop ost s : </s> where Ġwe Ġassume Ġthat Ġ$\ the ta Ġ$ Ġis Ġgenerated Ġfrom Ġa Ġprior Ġdistribution Ġ( e . g ., ĠLa pl ac ian Ġor ĠGa ussian ) Ġparameter ized Ġby Ġ$\ s igma Ġ$ . </s> To Ġleverage Ġunl abel ed Ġdata Ġfor Ġmodel Ġtraining , Ġwe Ġmake Ġuse Ġof Ġthe Ġexpectations Ġof Ġexisting Ġkeywords , Ġi . e ., Ġ{ ($ w ^{ ( 1 )} $, Ġ$ e ^{ ( 1 )} $ ), Ġ... , Ġ($ w ^{ ( t - 1 )} $, Ġ$ e ^{ ( t - 1 )} $ ), Ġ($ w ^{ ( t )} $, Ġ$ e ^{ ( t )} $ )} Ġ( Note Ġthat Ġ$ e ^{ ( t )} $ Ġis Ġinferred ), Ġas Ġa Ġregular ization Ġterm Ġto Ġconst rain Ġmodel Ġtraining . ĠTo Ġdo Ġso , Ġwe Ġfirst Ġgive Ġthe Ġmodel 's Ġexpectation Ġfor Ġeach Ġkeyword Ġ$ w ^{ ( k )} $ Ġ($ 1 \ le Ġk \ le Ġt $ ) Ġas Ġfollows : </s> which Ġdenotes Ġthe Ġempirical Ġexpectation Ġof Ġthe Ġmodel âĢ Ļ s Ġposterior Ġpredictions Ġon Ġthe Ġunl abel ed Ġmic rop ost s Ġ$\ math cal Ġ{ U } ^{ ( k )} $ Ġcontaining Ġkeyword Ġ$ w ^{ ( k )} $. ĠExpect ation Ġregular ization Ġcan Ġthen Ġbe Ġformulated Ġas Ġthe Ġregular ization Ġof Ġthe Ġdistance Ġbetween Ġthe ĠBern ou lli Ġdistribution Ġparameter ized Ġby Ġthe Ġmodel 's Ġexpectation Ġand Ġthe Ġexpectation Ġof Ġthe Ġexisting Ġkeyword : </s> where Ġ$ D _{ K L } [ \ cd ot Ġ\ Vert Ġ\ cd ot Ġ] $ Ġdenotes Ġthe ĠKL - d iver gence Ġbetween Ġthe ĠBern ou lli Ġdistributions Ġ$ Ber ( e ^{ ( k ) }) $ Ġand Ġ$ Ber (\ math bb Ġ{ E } _{ x \ sim Ġ\ math cal Ġ{ U } ^{ ( k ) }} ( y )) $, Ġand Ġ$\ lambda Ġ$ Ġcontrols Ġthe Ġstrength Ġof Ġexpectation Ġregular ization . </s> Un ified ĠProb abil istic ĠModel Ġ: :: ĠExpect ation Ġas ĠClass ĠPrior </s> To Ġlearn Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $ Ġand Ġthe Ġcrowd Ġworker Ġreliability Ġ$\ pi Ġ^ { ( n )} $ Ġ($ 1 \ le Ġn \ le ĠN $ ), Ġwe Ġmodel Ġthe Ġlikelihood Ġof Ġthe Ġcrowd - cont ributed Ġlabels Ġ$\ math bf Ġ{ A } $ Ġas Ġa Ġfunction Ġof Ġthese Ġparameters . ĠIn Ġthis Ġcontext , Ġwe Ġview Ġthe Ġexpectation Ġas Ġthe Ġclass Ġprior , Ġthus Ġperforming Ġexpectation Ġinference Ġas Ġthe Ġlearning Ġof Ġthe Ġclass Ġprior . ĠBy Ġdoing Ġso , Ġwe Ġconnect Ġexpectation Ġinference Ġwith Ġmodel Ġtraining . </s> Specifically , Ġwe Ġmodel Ġthe Ġlikelihood Ġof Ġan Ġarbitrary Ġcrowd - cont ributed Ġlabel Ġ$\ math bf Ġ{ A } _{ mn } $ Ġas Ġa Ġmixture Ġof Ġmult in om ials Ġwhere Ġthe Ġprior Ġis Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $ : </s> where Ġ$ e _ s ^{ ( t )} $ Ġis Ġthe Ġprobability Ġof Ġthe Ġground Ġtruth Ġlabel Ġbeing Ġ$ s $ Ġgiven Ġthe Ġkeyword - specific Ġexpectation Ġas Ġthe Ġclass Ġprior ; Ġ$ K $ Ġis Ġthe Ġset Ġof Ġpossible Ġground Ġtruth Ġlabels Ġ( binary Ġin Ġour Ġcontext ); Ġand Ġ$ r = \ math bf Ġ{ A } _{ mn } $ Ġis Ġthe Ġcrowd - cont ributed Ġlabel . ĠThen , Ġfor Ġan Ġindividual Ġmic rop ost Ġ$ x _ m $, Ġthe Ġlikelihood Ġof Ġcrowd - cont ributed Ġlabels Ġ$\ math bf Ġ{ A } _{ m : } $ Ġis Ġgiven Ġby : </s> Therefore , Ġthe Ġobjective Ġfunction Ġfor Ġmaximizing Ġthe Ġlikelihood Ġof Ġthe Ġentire Ġannotation Ġmatrix Ġ$\ math bf Ġ{ A } $ Ġcan Ġbe Ġdescribed Ġas : </s> Un ified ĠProb abil istic ĠModel Ġ: :: ĠUnified ĠProb abil istic ĠModel </s> Integ rating Ġmodel Ġtraining Ġwith Ġexpectation Ġinference , Ġthe Ġoverall Ġobjective Ġfunction Ġof Ġour Ġproposed Ġmodel Ġis Ġgiven Ġby : </s> Figure ĠFIG REF 18 Ġdepicts Ġa Ġgraphical Ġrepresentation Ġof Ġour Ġmodel , Ġwhich Ġcombines Ġthe Ġtarget Ġmodel Ġfor Ġtraining Ġ( on Ġthe Ġleft ) Ġwith Ġthe Ġgener ative Ġmodel Ġfor Ġcrowd - cont ributed Ġlabels Ġ( on Ġthe Ġright ) Ġthrough Ġa Ġkeyword - specific Ġexpectation . </s> Model ĠLearning . ĠDue Ġto Ġthe Ġunknown Ġground Ġtruth Ġlabels Ġof Ġcrowd - annot ated Ġmic rop ost s Ġ($ y _ m $ Ġin ĠFigure ĠFIG REF 18 ), Ġwe Ġresort Ġto Ġexpectation Ġmaxim ization Ġfor Ġmodel Ġlearning . ĠThe Ġlearning Ġalgorithm Ġiter atively Ġtakes Ġtwo Ġsteps : Ġthe ĠE - step Ġand Ġthe ĠM - step . ĠThe ĠE - step Ġinf ers Ġthe Ġground Ġtruth Ġlabels Ġgiven Ġthe Ġcurrent Ġmodel Ġparameters . ĠThe ĠM - step Ġupdates Ġthe Ġmodel Ġparameters , Ġincluding Ġthe Ġcrowd Ġreliability Ġparameters Ġ$\ pi Ġ^ { ( n )} $ Ġ($ 1 \ le Ġn \ le ĠN $ ), Ġthe Ġkeyword - specific Ġexpectation Ġ$ e ^{ ( t )} $, Ġand Ġthe Ġparameter Ġof Ġthe Ġtarget Ġmodel Ġ$\ the ta Ġ^ { ( t )} $. ĠThe ĠE - step Ġand Ġthe Ġcrowd Ġparameter Ġupdate Ġin Ġthe ĠM - step Ġare Ġsimilar Ġto Ġthe ĠDaw id - Sk ene Ġmodel ĠB IB REF 14 . ĠThe Ġkeyword Ġexpectation Ġis Ġinferred Ġby Ġtaking Ġinto Ġaccount Ġboth Ġthe Ġcrowd - cont ributed Ġlabels Ġand Ġthe Ġmodel Ġprediction : </s> The Ġparameter Ġof Ġthe Ġtarget Ġmodel Ġis Ġupdated Ġby Ġgradient Ġdescent . ĠFor Ġexample , Ġwhen Ġthe Ġtarget Ġmodel Ġto Ġbe Ġtrained Ġis Ġa Ġdeep Ġneural Ġnetwork , Ġwe Ġuse Ġback - prop ag ation Ġwith Ġgradient Ġdescent Ġto Ġupdate Ġthe Ġweight Ġmat rices . </s> Exper iments Ġand ĠResults </s> This Ġsection Ġpresents Ġour Ġexperimental Ġsetup Ġand Ġresults Ġfor Ġevaluating Ġour Ġapproach . ĠWe Ġaim Ġat Ġanswering Ġthe Ġfollowing Ġquestions : </s> [ no item se p , left margin = * ] </s> Q 1 : ĠHow Ġeffectively Ġdoes Ġour Ġproposed Ġhuman - AI Ġloop Ġapproach Ġenhance Ġthe Ġstate - of - the - art Ġmachine Ġlearning Ġmodels Ġfor Ġevent Ġdetection ? </s> Q 2 : ĠHow Ġwell Ġdoes Ġour Ġkeyword Ġdiscovery Ġmethod Ġwork Ġcompare Ġto Ġexisting Ġkeyword Ġexpansion Ġmethods ? </s> Q 3 : ĠHow Ġeffective Ġis Ġour Ġapproach Ġusing Ġcrowds ourcing Ġat Ġobtaining Ġnew Ġkeywords Ġcompared Ġwith Ġan Ġapproach Ġlab elling Ġmic rop ost s Ġfor Ġmodel Ġtraining Ġunder Ġthe Ġsame Ġcost ? </s> Q 4 : ĠHow Ġmuch Ġbenefit Ġdoes Ġour Ġunified Ġprob abil istic Ġmodel Ġbring Ġcompared Ġto Ġmethods Ġthat Ġdo Ġnot Ġtake Ġcrowd Ġreliability Ġinto Ġaccount ? </s> Exper iments Ġand ĠResults Ġ: :: ĠExperimental ĠSetup </s> Dat as ets . ĠWe Ġperform Ġour Ġexperiments Ġwith Ġtwo Ġpredetermined Ġevent Ġcategories : Ġcyber Ġsecurity Ġ( Cy ber Attack ) Ġand Ġdeath Ġof Ġpoliticians Ġ( Polit ician Death ). ĠThese Ġevent Ġcategories Ġare Ġchosen Ġas Ġthey Ġare Ġrepresentative Ġof Ġimportant Ġevent Ġtypes Ġthat Ġare Ġof Ġinterest Ġto Ġmany Ġgovernments Ġand Ġcompanies . ĠThe Ġneed Ġto Ġcreate Ġour Ġown Ġdataset Ġwas Ġmotivated Ġby Ġthe Ġlack Ġof Ġpublic Ġdatasets Ġfor Ġevent Ġdetection Ġon Ġmic rop ost s . ĠThe Ġfew Ġavailable Ġdatasets Ġdo Ġnot Ġsuit Ġour Ġrequirements . ĠFor Ġexample , Ġthe Ġpublicly Ġavailable ĠEvents - 2012 ĠTwitter Ġdataset ĠB IB REF 20 Ġcontains Ġgeneric Ġevent Ġdescriptions Ġsuch Ġas ĠPolitics , ĠSports , ĠCulture Ġetc . ĠOur Ġwork Ġtargets Ġmore Ġspecific Ġevent Ġcategories ĠB IB REF 21 . ĠFollowing Ġprevious Ġstudies ĠB IB REF 1 , Ġwe Ġcollect Ġevent - related Ġmic rop ost s Ġfrom ĠTwitter Ġusing Ġ11 Ġand Ġ8 Ġseed Ġevents Ġ( see ĠSection ĠSEC REF 2 ) Ġfor ĠCyber Attack Ġand ĠPolit ician Death , Ġrespectively . ĠUn label ed Ġmic rop ost s Ġare Ġcollected Ġby Ġusing Ġthe Ġkeyword Ġ` hack ' Ġfor ĠCyber Attack , Ġwhile Ġfor ĠPolit ician Death , Ġwe Ġuse Ġa Ġset Ġof Ġkeywords Ġrelated Ġto Ġ` polit ician ' Ġand Ġ` death ' Ġ( such Ġas Ġ` b ureau cr at ', Ġ` dead ' Ġetc .) ĠFor Ġeach Ġdataset , Ġwe Ġrandomly Ġselect Ġ500 Ġtweets Ġfrom Ġthe Ġunl abel ed Ġsubset Ġand Ġmanually Ġlabel Ġthem Ġfor Ġevaluation . ĠTable ĠT AB REF 25 Ġshows Ġkey Ġstatistics Ġfrom Ġour Ġtwo Ġdatasets . </s> Compar ison ĠMethods . ĠTo Ġdemonstrate Ġthe Ġgener ality Ġof Ġour Ġapproach Ġon Ġdifferent Ġevent Ġdetection Ġmodels , Ġwe Ġconsider ĠLog istic ĠReg ression Ġ( LR ) ĠB IB REF 1 Ġand ĠMult il ayer ĠPer cept ron Ġ( ML P ) ĠB IB REF 2 Ġas Ġthe Ġtarget Ġmodels . ĠAs Ġthe Ġgoal Ġof Ġour Ġexperiments Ġis Ġto Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġour Ġapproach Ġas Ġa Ġnew Ġmodel Ġtraining Ġtechnique , Ġwe Ġuse Ġthese Ġwidely Ġused Ġmodels . ĠAlso , Ġwe Ġnote Ġthat Ġin Ġour Ġcase Ġother Ġneural Ġnetwork Ġmodels Ġwith Ġmore Ġcomplex Ġnetwork Ġarchitectures Ġfor Ġevent Ġdetection , Ġsuch Ġas Ġthe Ġbi - direction al ĠL ST M ĠB IB REF 17 , Ġturn Ġout Ġto Ġbe Ġless Ġeffective Ġthan Ġa Ġsimple Ġfeed forward Ġnetwork . ĠFor Ġboth ĠLR Ġand ĠML P , Ġwe Ġevaluate Ġour Ġproposed Ġhuman - AI Ġloop Ġapproach Ġfor Ġkeyword Ġdiscovery Ġand Ġexpectation Ġestimation Ġby Ġcomparing Ġagainst Ġthe Ġweak ly Ġsupervised Ġlearning Ġmethod Ġproposed Ġby ĠB IB REF 1 Ġ( BI B REF 1 ) Ġand ĠB IB REF 17 Ġ( BI B REF 17 ) Ġwhere Ġonly Ġone Ġinitial Ġkeyword Ġis Ġused Ġwith Ġan Ġexpectation Ġestimated Ġby Ġan Ġindividual Ġexpert . </s> Parameter ĠSettings . ĠWe Ġempir ically Ġset Ġoptimal Ġparameters Ġbased Ġon Ġa Ġheld - out Ġvalidation Ġset Ġthat Ġcontains Ġ20 % Ġof Ġthe Ġtest Ġdata . ĠThese Ġinclude Ġthe Ġhyper param ters Ġof Ġthe Ġtarget Ġmodel , Ġthose Ġof Ġour Ġproposed Ġprob abil istic Ġmodel , Ġand Ġthe Ġparameters Ġused Ġfor Ġtraining Ġthe Ġtarget Ġmodel . ĠWe Ġexplore ĠML P Ġwith Ġ1 , Ġ2 Ġand Ġ3 Ġhidden Ġlayers Ġand Ġapply Ġa Ġgrid Ġsearch Ġin Ġ32 , Ġ64 , Ġ128 , Ġ256 , Ġ512 Ġfor Ġthe Ġdimension Ġof Ġthe Ġembed d ings Ġand Ġthat Ġof Ġthe Ġhidden Ġlayers . ĠFor Ġthe Ġcoefficient Ġof Ġexpectation Ġregular ization , Ġwe Ġfollow ĠB IB REF 6 Ġ( BI B REF 6 ) Ġand Ġset Ġit Ġto Ġ$\ lambda Ġ= 10 Ġ\ times Ġ$ Ġ# label ed Ġexamples . ĠFor Ġmodel Ġtraining , Ġwe Ġuse Ġthe ĠAdam ĠB IB REF 22 Ġoptimization Ġalgorithm Ġfor Ġboth Ġmodels . </s> E val uation . ĠFollowing ĠB IB REF 1 Ġ( BI B REF 1 ) Ġand ĠB IB REF 3 Ġ( BI B REF 3 ), Ġwe Ġuse Ġaccuracy Ġand Ġarea Ġunder Ġthe Ġprecision - rec all Ġcurve Ġ( A UC ) Ġmetrics Ġto Ġmeasure Ġthe Ġperformance Ġof Ġour Ġproposed Ġapproach . ĠWe Ġnote Ġthat Ġdue Ġto Ġthe Ġimbalance Ġin Ġour Ġdatasets Ġ( 20 % Ġpositive Ġmic rop ost s Ġin ĠCyber Attack Ġand Ġ27 % Ġin ĠPolit ician Death ), Ġaccuracy Ġis Ġdominated Ġby Ġnegative Ġexamples ; ĠA UC , Ġin Ġcomparison , Ġbetter Ġcharacter izes Ġthe Ġdiscrim in ative Ġpower Ġof Ġthe Ġmodel . </s> C rowd s ourcing . ĠWe Ġchose ĠLevel Ġ3 Ġworkers Ġon Ġthe ĠFigure - Eight Ġcrowds ourcing Ġplatform Ġfor Ġour Ġexperiments . ĠThe Ġinter - annot ator Ġagreement Ġin Ġmic rop ost Ġclassification Ġis Ġtaken Ġinto Ġaccount Ġthrough Ġthe ĠEM Ġalgorithm . ĠFor Ġkeyword Ġdiscovery , Ġwe Ġfilter Ġkeywords Ġbased Ġon Ġthe Ġfrequency Ġof Ġthe Ġkeyword Ġbeing Ġselected Ġby Ġthe Ġcrowd . ĠIn Ġterms Ġof Ġcost - effect iveness , Ġour Ġapproach Ġis Ġmotivated Ġfrom Ġthe Ġfact Ġthat Ġcrowds ourced Ġdata Ġannotation Ġcan Ġbe Ġexpensive , Ġand Ġis Ġthus Ġdesigned Ġwith Ġminimal Ġcrowd Ġinvolvement . ĠFor Ġeach Ġiteration , Ġwe Ġselected Ġ50 Ġtweets Ġfor Ġkeyword Ġdiscovery Ġand Ġ50 Ġtweets Ġfor Ġmic rop ost Ġclassification Ġper Ġkeyword . ĠFor Ġa Ġdataset Ġwith Ġ80 k Ġtweets Ġ( e . g ., ĠCyber Attack ), Ġour Ġapproach Ġonly Ġrequires Ġto Ġmanually Ġinspect Ġ800 Ġtweets Ġ( for Ġ8 Ġkeywords ), Ġwhich Ġis Ġonly Ġ1 % Ġof Ġthe Ġentire Ġdataset . </s> Exper iments Ġand ĠResults Ġ: :: ĠResults Ġof Ġour ĠHuman - AI ĠLoop Ġ( Q 1 ) </s> Table ĠT AB REF 26 Ġreports Ġthe Ġevaluation Ġof Ġour Ġapproach Ġon Ġboth Ġthe ĠCyber Attack Ġand ĠPolit ician Death Ġevent Ġcategories . ĠOur Ġapproach Ġis Ġconfigured Ġsuch Ġthat Ġeach Ġiteration Ġstarts Ġwith Ġ1 Ġnew Ġkeyword Ġdiscovered Ġin Ġthe Ġprevious Ġiteration . </s> Our Ġapproach Ġimproves ĠLR Ġby Ġ5 . 17 % Ġ( Acc uracy ) Ġand Ġ18 . 38 % Ġ( A UC ), Ġand ĠML P Ġby Ġ10 . 71 % Ġ( Acc uracy ) Ġand Ġ30 . 27 % Ġ( A UC ) Ġon Ġaverage . ĠSuch Ġsignificant Ġimprovements Ġclearly Ġdemonstrate Ġthat Ġour Ġapproach Ġis Ġeffective Ġat Ġimproving Ġmodel Ġperformance . ĠWe Ġobserve Ġthat Ġthe Ġtarget Ġmodels Ġgenerally Ġconverge Ġbetween Ġthe Ġ7 th Ġand Ġ9 th Ġiteration Ġon Ġboth Ġdatasets Ġwhen Ġperformance Ġis Ġmeasured Ġby ĠA UC . ĠThe Ġperformance Ġcan Ġslightly Ġdegrade Ġwhen Ġthe Ġmodels Ġare Ġfurther Ġtrained Ġfor Ġmore Ġiterations Ġon Ġboth Ġdatasets . ĠThis Ġis Ġlikely Ġdue Ġto Ġthe Ġfact Ġthat Ġover Ġtime , Ġthe Ġnewly Ġdiscovered Ġkeywords Ġentail Ġlower Ġnovel Ġinformation Ġfor Ġmodel Ġtraining . ĠFor Ġinstance , Ġfor Ġthe ĠCyber Attack Ġdataset Ġthe Ġnew Ġkeyword Ġin Ġthe Ġ9 th Ġiteration Ġ` election ' Ġfrequently Ġco - occ urs Ġwith Ġthe Ġkeyword Ġ` r ussia ' Ġin Ġthe Ġ5 th Ġiteration Ġ( in Ġmic rop ost s Ġthat Ġconnect ĠRussian Ġhackers Ġwith ĠUS Ġelections ), Ġthus Ġbringing Ġlimited Ġnew Ġinformation Ġfor Ġimproving Ġthe Ġmodel Ġperformance . ĠAs Ġa Ġside Ġremark , Ġwe Ġnote Ġthat Ġthe Ġmodels Ġconverge Ġfaster Ġwhen Ġperformance Ġis Ġmeasured Ġby Ġaccuracy . ĠSuch Ġa Ġcomparison Ġresult Ġconfirms Ġthe Ġdifference Ġbetween Ġthe Ġmetrics Ġand Ġshows Ġthe Ġnecessity Ġfor Ġmore Ġkeywords Ġto Ġdiscriminate Ġevent - related Ġmic rop ost s Ġfrom Ġnon Ġevent - related Ġones . </s> Exper iments Ġand ĠResults Ġ: :: ĠCompar ative ĠResults Ġon ĠKey word ĠDiscovery Ġ( Q 2 ) </s> Figure ĠFIG REF 31 Ġshows Ġthe Ġevaluation Ġof Ġour Ġapproach Ġwhen Ġdiscovering Ġnew Ġinformative Ġkeywords Ġfor Ġmodel Ġtraining Ġ( see ĠSection ĠSEC REF 2 : ĠKey word ĠDiscovery ). ĠWe Ġcompare Ġour Ġhuman - AI Ġcollaborative Ġway Ġof Ġdiscovering Ġnew Ġkeywords Ġagainst Ġa Ġquery Ġexpansion Ġ( Q E ) Ġapproach ĠB IB REF 23 , ĠB IB REF 24 Ġthat Ġlever ages Ġword Ġembed d ings Ġto Ġfind Ġsimilar Ġwords Ġin Ġthe Ġlatent Ġsemantic Ġspace . ĠSpecifically , Ġwe Ġuse Ġpre - trained Ġword Ġembed d ings Ġbased Ġon Ġa Ġlarge ĠGoogle ĠNews Ġdataset Ġfor Ġquery Ġexpansion . ĠFor Ġinstance , Ġthe Ġtop Ġkeywords Ġresulting Ġfrom ĠQ E Ġfor Ġ` polit ician ' Ġare , Ġ` dep uty ', ` minist ry ', ` secret ary ', Ġand Ġ` min ister '. ĠFor Ġeach Ġof Ġthese Ġkeywords , Ġwe Ġuse Ġthe Ġcrowd Ġto Ġlabel Ġa Ġset Ġof Ġtweets Ġand Ġobtain Ġa Ġcorresponding Ġexpectation . </s> We Ġobserve Ġthat Ġour Ġapproach Ġconsistently Ġoutper forms ĠQ E Ġby Ġan Ġaverage Ġof Ġ$ 4 . 62 \ % $ Ġand Ġ$ 52 . 58 \ % $ ĠA UC Ġon ĠCyber Attack Ġand ĠPolit ician Death , Ġrespectively . ĠThe Ġlarge Ġgap Ġbetween Ġthe Ġperformance Ġimprovements Ġfor Ġthe Ġtwo Ġdatasets Ġis Ġmainly Ġdue Ġto Ġthe Ġfact Ġthat Ġmic rop ost s Ġthat Ġare Ġrelevant Ġfor ĠPolit ician Death Ġare Ġsem antically Ġmore Ġcomplex Ġthan Ġthose Ġfor ĠCyber Attack , Ġas Ġthey Ġencode Ġnoun - verb Ġrelationship Ġ( e . g ., ĠâĢ ľ the Ġking Ġof Ġ... Ġdied Ġ... âĢ Ŀ ) Ġrather Ġthan Ġa Ġsimple Ġverb Ġ( e . g ., ĠâĢ ľ ... Ġhacked . âĢ Ŀ ) Ġfor Ġthe ĠCyber Attack Ġmic rop ost s . ĠQ E Ġonly Ġfinds Ġsyn onyms Ġof Ġexisting Ġkeywords Ġrelated Ġto Ġeither Ġ` polit ician ' Ġor Ġ` death ', Ġhowever Ġcannot Ġfind Ġa Ġmeaningful Ġkeyword Ġthat Ġfully Ġcharacter izes Ġthe Ġdeath Ġof Ġa Ġpolitician . ĠFor Ġinstance , ĠQ E Ġfinds Ġthe Ġkeywords Ġ` kill ' Ġand Ġ` mur der ', Ġwhich Ġare Ġsem antically Ġclose Ġto Ġ` death ' Ġbut Ġare Ġnot Ġspecifically Ġrelevant Ġto Ġthe Ġdeath Ġof Ġa Ġpolitician . ĠUnlike ĠQ E , Ġour Ġapproach Ġidentifies Ġkeywords Ġthat Ġgo Ġbeyond Ġmere Ġsyn onyms Ġand Ġthat Ġare Ġmore Ġdirectly Ġrelated Ġto Ġthe Ġend Ġtask , Ġi . e ., Ġdiscriminating Ġevent - related Ġmic rop ost s Ġfrom Ġnon Ġrelated Ġones . ĠExamples Ġare Ġ` dem ise ' Ġand Ġ` cond ol ence '. ĠAs Ġa Ġremark , Ġwe Ġnote Ġthat Ġin ĠFigure ĠFIG REF 31 ( b ), Ġthe Ġincrease Ġin ĠQ E Ġperformance Ġon ĠPolit ician Death Ġis Ġdue Ġto Ġthe Ġkeywords Ġ` dep uty ' Ġand Ġ` min ister ', Ġwhich Ġhappen Ġto Ġbe Ġhighly Ġindicative Ġof Ġthe Ġdeath Ġof Ġa Ġpolitician Ġin Ġour Ġdataset ; Ġthese Ġkeywords Ġare Ġalso Ġidentified Ġby Ġour Ġapproach . </s> Exper iments Ġand ĠResults Ġ: :: ĠCost - Effect iveness ĠResults Ġ( Q 3 ) </s> To Ġdemonstrate Ġthe Ġcost - effect iveness Ġof Ġusing Ġcrowds ourcing Ġfor Ġobtaining Ġnew Ġkeywords Ġand Ġconsequently , Ġtheir Ġexpectations , Ġwe Ġcompare Ġthe Ġperformance Ġof Ġour Ġapproach Ġwith Ġan Ġapproach Ġusing Ġcrowds ourcing Ġto Ġonly Ġlabel Ġmic rop ost s Ġfor Ġmodel Ġtraining Ġat Ġthe Ġsame Ġcost . ĠSpecifically , Ġwe Ġconducted Ġan Ġadditional Ġcrowds ourcing Ġexperiment Ġwhere Ġthe Ġsame Ġcost Ġused Ġfor Ġkeyword Ġdiscovery Ġin Ġour Ġapproach Ġis Ġused Ġto Ġlabel Ġadditional Ġmic rop ost s Ġfor Ġmodel Ġtraining . ĠThese Ġnewly Ġlabeled Ġmic rop ost s Ġare Ġused Ġwith Ġthe Ġmic rop ost s Ġlabeled Ġin Ġthe Ġmic rop ost Ġclassification Ġtask Ġof Ġour Ġapproach Ġ( see ĠSection ĠSEC REF 2 : ĠMic rop ost ĠClassification ) Ġand Ġthe Ġexpectation Ġof Ġthe Ġinitial Ġkeyword Ġto Ġtrain Ġthe Ġmodel Ġfor Ġcomparison . ĠThe Ġmodel Ġtrained Ġin Ġthis Ġway Ġincreases ĠA UC Ġby Ġ0 . 87 % Ġfor ĠCyber Attack , Ġand Ġby Ġ1 . 06 % Ġfor ĠPolit ician Death ; Ġin Ġcomparison , Ġour Ġproposed Ġapproach Ġincreases ĠA UC Ġby Ġ33 . 42 % Ġfor ĠPolit ician Death Ġand Ġby Ġ15 . 23 % Ġfor ĠCyber Attack Ġover Ġthe Ġbaseline Ġpresented Ġby ĠB IB REF 1 ). ĠThese Ġresults Ġshow Ġthat Ġusing Ġcrowds ourcing Ġfor Ġkeyword Ġdiscovery Ġis Ġsignificantly Ġmore Ġcost - effective Ġthan Ġsimply Ġusing Ġcrowds ourcing Ġto Ġget Ġadditional Ġlabels Ġwhen Ġtraining Ġthe Ġmodel . </s> Exper iments Ġand ĠResults Ġ: :: ĠExpect ation ĠIn ference ĠResults Ġ( Q 4 ) </s> To Ġinvestigate Ġthe Ġeffectiveness Ġof Ġour Ġexpectation Ġinference Ġmethod , Ġwe Ġcompare Ġit Ġagainst Ġa Ġmajority Ġvoting Ġapproach , Ġa Ġstrong Ġbaseline Ġin Ġtruth Ġinference ĠB IB REF 16 . ĠFigure ĠFIG REF 36 Ġshows Ġthe Ġresult Ġof Ġthis Ġevaluation . ĠWe Ġobserve Ġthat Ġour Ġapproach Ġresults Ġin Ġbetter Ġmodels Ġfor Ġboth ĠCyber Attack Ġand ĠPolit ician Death . ĠOur Ġmanual Ġinvestigation Ġreveals Ġthat Ġworkers ' Ġannotations Ġare Ġof Ġhigh Ġreliability , Ġwhich Ġexplains Ġthe Ġrelatively Ġgood Ġperformance Ġof Ġmajority Ġvoting . ĠDespite Ġlimited Ġmargin Ġfor Ġimprovement , Ġour Ġmethod Ġof Ġexpectation Ġinference Ġimproves Ġthe Ġperformance Ġof Ġmajority Ġvoting Ġby Ġ$ 0 . 4 \ % $ Ġand Ġ$ 1 . 19 \ % $ ĠA UC Ġon ĠCyber Attack Ġand ĠPolit ician Death , Ġrespectively . </s> Related ĠWork </s> Event ĠDetection . ĠThe Ġtechniques Ġfor Ġevent Ġextraction Ġfrom Ġmicro blog ging Ġplatforms Ġcan Ġbe Ġclassified Ġaccording Ġto Ġtheir Ġdomain Ġspecificity Ġand Ġtheir Ġdetection Ġmethod ĠB IB REF 0 . ĠEarly Ġworks Ġmainly Ġfocus Ġon Ġopen Ġdomain Ġevent Ġdetection ĠB IB REF 25 , ĠB IB REF 26 , ĠB IB REF 27 . ĠOur Ġwork Ġfalls Ġinto Ġthe Ġcategory Ġof Ġdomain - specific Ġevent Ġdetection ĠB IB REF 21 , Ġwhich Ġhas Ġdrawn Ġincreasing Ġattention Ġdue Ġto Ġits Ġrelevance Ġfor Ġvarious Ġapplications Ġsuch Ġas Ġcyber Ġsecurity ĠB IB REF 1 , ĠB IB REF 2 Ġand Ġpublic Ġhealth ĠB IB REF 4 , ĠB IB REF 5 . ĠIn Ġterms Ġof Ġtechnique , Ġour Ġproposed Ġdetection Ġmethod Ġis Ġrelated Ġto Ġthe Ġrecently Ġproposed Ġweak ly Ġsupervised Ġlearning Ġmethods ĠB IB REF 1 , ĠB IB REF 17 , ĠB IB REF 3 . ĠThis Ġcomes Ġin Ġcontrast Ġwith Ġfully - super vised Ġlearning Ġmethods , Ġwhich Ġare Ġoften Ġlimited Ġby Ġthe Ġsize Ġof Ġthe Ġtraining Ġdata Ġ( e . g ., Ġa Ġfew Ġhundred Ġexamples ) ĠB IB REF 28 , ĠB IB REF 29 . </s> Human - in - the - Loop ĠAppro aches . ĠOur Ġwork Ġextends Ġweak ly Ġsupervised Ġlearning Ġmethods Ġby Ġinvolving Ġhumans Ġin Ġthe Ġloop ĠB IB REF 13 . ĠEx isting Ġhuman - in - the - loop Ġapproaches Ġmainly Ġleverage Ġcrowds Ġto Ġlabel Ġindividual Ġdata Ġinstances ĠB IB REF 9 , ĠB IB REF 10 Ġor Ġto Ġdebug Ġthe Ġtraining Ġdata ĠB IB REF 30 , ĠB IB REF 31 Ġor Ġcomponents ĠB IB REF 32 , ĠB IB REF 33 , ĠB IB REF 34 Ġof Ġa Ġmachine Ġlearning Ġsystem . ĠUnlike Ġthese Ġworks , Ġwe Ġleverage Ġcrowd Ġworkers Ġto Ġlabel Ġsampled Ġmic rop ost s Ġin Ġorder Ġto Ġobtain Ġkeyword - specific Ġexpectations , Ġwhich Ġcan Ġthen Ġbe Ġgeneralized Ġto Ġhelp Ġclassify Ġmic rop ost s Ġcontaining Ġthe Ġsame Ġkeyword , Ġthus Ġampl ifying Ġthe Ġutility Ġof Ġthe Ġcrowd . ĠOur Ġwork Ġis Ġfurther Ġconnected Ġto Ġthe Ġtopic Ġof Ġinterpret ability Ġand Ġtransparency Ġof Ġmachine Ġlearning Ġmodels ĠB IB REF 11 , ĠB IB REF 35 , ĠB IB REF 12 , Ġfor Ġwhich Ġhumans Ġare Ġincreasingly Ġinvolved , Ġfor Ġinstance Ġfor Ġpost - h oc Ġevaluations Ġof Ġthe Ġmodel 's Ġinterpret ability . ĠIn Ġcontrast , Ġour Ġapproach Ġdirectly Ġsolic its Ġinformative Ġkeywords Ġfrom Ġthe Ġcrowd Ġfor Ġmodel Ġtraining , Ġthereby Ġproviding Ġhuman - under stand able Ġexplanations Ġfor Ġthe Ġimproved Ġmodel . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġa Ġnew Ġhuman - AI Ġloop Ġapproach Ġfor Ġkeyword Ġdiscovery Ġand Ġexpectation Ġestimation Ġto Ġbetter Ġtrain Ġevent Ġdetection Ġmodels . ĠOur Ġapproach Ġtakes Ġadvantage Ġof Ġthe Ġdisagreement Ġbetween Ġthe Ġcrowd Ġand Ġthe Ġmodel Ġto Ġdiscover Ġinformative Ġkeywords Ġand Ġlever ages Ġthe Ġjoint Ġpower Ġof Ġthe Ġcrowd Ġand Ġthe Ġmodel Ġin Ġexpectation Ġinference . ĠWe Ġevaluated Ġour Ġapproach Ġon Ġreal - world Ġdatasets Ġand Ġshowed Ġthat Ġit Ġsignificantly Ġoutper forms Ġthe Ġstate Ġof Ġthe Ġart Ġand Ġthat Ġit Ġis Ġparticularly Ġuseful Ġfor Ġdetecting Ġevents Ġwhere Ġrelevant Ġmic rop ost s Ġare Ġsem antically Ġcomplex , Ġe . g ., Ġthe Ġdeath Ġof Ġa Ġpolitician . ĠAs Ġfuture Ġwork , Ġwe Ġplan Ġto Ġparallel ize Ġthe Ġcrowds ourcing Ġtasks Ġand Ġoptimize Ġour Ġpipeline Ġin Ġorder Ġto Ġuse Ġour Ġevent Ġdetection Ġapproach Ġin Ġreal - time . </s> Acknowled gements </s> This Ġproject Ġhas Ġreceived Ġfunding Ġfrom Ġthe ĠSwiss ĠNational ĠScience ĠFoundation Ġ( gr ant Ġ# 40 75 40 _ 16 73 20 ĠTight en - it - All ) Ġand Ġfrom Ġthe ĠEuropean ĠResearch ĠCouncil Ġ( ERC ) Ġunder Ġthe ĠEuropean ĠUnion 's ĠHorizon Ġ2020 Ġresearch Ġand Ġinnovation Ġprogramme Ġ( gr ant Ġagreement Ġ6 83 253 / Graph Int ).