Unanswerable
NONE
They did not use any platform, instead they hired undergraduate students to do the annotation.
ABSTRACTIVE
With the purpose of obtaining evaluation datasets from crowd annotators, we collect the sentences from two domains: Dialog and E-commerce domain. We hire undergraduate students to annotate the sentences. They are required to identify the predefined types of entities in the sentences. Together with the guideline document, the annotators are educated some tips in fifteen minutes and also provided with 20 exemplifying sentences.
1801.05147
Introduction </s> There Ġhas Ġbeen Ġsignificant Ġprogress Ġon ĠNamed ĠEntity ĠRecogn ition Ġ( NER ) Ġin Ġrecent Ġyears Ġusing Ġmodels Ġbased Ġon Ġmachine Ġlearning Ġalgorithms ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠAs Ġwith Ġother ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġtasks , Ġbuilding ĠN ER Ġsystems Ġtypically Ġrequires Ġa Ġmassive Ġamount Ġof Ġlabeled Ġtraining Ġdata Ġwhich Ġare Ġannot ated Ġby Ġexperts . ĠIn Ġreal Ġapplications , Ġwe Ġoften Ġneed Ġto Ġconsider Ġnew Ġtypes Ġof Ġentities Ġin Ġnew Ġdomains Ġwhere Ġwe Ġdo Ġnot Ġhave Ġexisting Ġannot ated Ġdata . ĠFor Ġsuch Ġnew Ġtypes Ġof Ġentities , Ġhowever , Ġit Ġis Ġvery Ġhard Ġto Ġfind Ġexperts Ġto Ġannot ate Ġthe Ġdata Ġwithin Ġshort Ġtime Ġlimits Ġand Ġhiring Ġexperts Ġis Ġcostly Ġand Ġnon - sc al able , Ġboth Ġin Ġterms Ġof Ġtime Ġand Ġmoney . </s> In Ġorder Ġto Ġquickly Ġobtain Ġnew Ġtraining Ġdata , Ġwe Ġcan Ġuse Ġcrowds ourcing Ġas Ġone Ġalternative Ġway Ġat Ġlower Ġcost Ġin Ġa Ġshort Ġtime . ĠBut Ġas Ġan Ġexchange , Ġcrowd Ġannotations Ġfrom Ġnon - exper ts Ġmay Ġbe Ġof Ġlower Ġquality Ġthan Ġthose Ġfrom Ġexperts . ĠIt Ġis Ġone Ġbiggest Ġchallenge Ġto Ġbuild Ġa Ġpowerful ĠN ER Ġsystem Ġon Ġsuch Ġa Ġlow Ġquality Ġannot ated Ġdata . ĠAlthough Ġwe Ġcan Ġobtain Ġhigh Ġquality Ġannotations Ġfor Ġeach Ġinput Ġsentence Ġby Ġmajority Ġvoting , Ġit Ġcan Ġbe Ġa Ġwaste Ġof Ġhuman Ġlab ors Ġto Ġachieve Ġsuch Ġa Ġgoal , Ġespecially Ġfor Ġsome Ġambiguous Ġsentences Ġwhich Ġmay Ġrequire Ġa Ġnumber Ġof Ġannotations Ġto Ġreach Ġan Ġagreement . ĠThus Ġmajority Ġwork Ġdirectly Ġbuild Ġmodels Ġon Ġcrowd Ġannotations , Ġtrying Ġto Ġmodel Ġthe Ġdifferences Ġamong Ġannot ators , Ġfor Ġexample , Ġsome Ġof Ġthe Ġannot ators Ġmay Ġbe Ġmore Ġtrust ful ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. </s> Here Ġwe Ġfocus Ġmainly Ġon Ġthe ĠChinese ĠN ER , Ġwhich Ġis Ġmore Ġdifficult Ġthan ĠN ER Ġfor Ġother Ġlanguages Ġsuch Ġas ĠEnglish Ġfor Ġthe Ġlack Ġof Ġmorph ological Ġvariations Ġsuch Ġas Ġcapital ization Ġand Ġin Ġparticular Ġthe Ġuncertainty Ġin Ġword Ġsegment ation . ĠThe ĠChinese ĠNE Ġtag gers Ġtrained Ġon Ġnews Ġdomain Ġoften Ġperform Ġpoor Ġin Ġother Ġdomains . ĠAlthough Ġwe Ġcan Ġalleviate Ġthe Ġproblem Ġby Ġusing Ġcharacter - level Ġtagging Ġto Ġresolve Ġthe Ġproblem Ġof Ġpoor Ġword Ġsegment ation Ġperformances ĠB IB REF 5 Ġ, Ġstill Ġthere Ġexists Ġa Ġlarge Ġgap Ġwhen Ġthe Ġtarget Ġdomain Ġchanges , Ġespecially Ġfor Ġthe Ġtexts Ġof Ġsocial Ġmedia . ĠThus , Ġin Ġorder Ġto Ġget Ġa Ġgood Ġtag ger Ġfor Ġnew Ġdomains Ġand Ġalso Ġfor Ġthe Ġconditions Ġof Ġnew Ġentity Ġtypes , Ġwe Ġrequire Ġlarge Ġamounts Ġof Ġlabeled Ġdata . ĠTherefore , Ġcrowds ourcing Ġis Ġa Ġreasonable Ġsolution Ġfor Ġthese Ġsituations . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġan Ġapproach Ġto Ġtraining Ġa ĠChinese ĠN ER Ġsystem Ġon Ġthe Ġcrowd - annot ated Ġdata . ĠOur Ġgoal Ġis Ġto Ġextract Ġadditional Ġannot ator Ġindependent Ġfeatures Ġby Ġadvers arial Ġtraining , Ġallev iating Ġthe Ġannotation Ġnoises Ġof Ġnon - exper ts . ĠThe Ġidea Ġof Ġadvers arial Ġtraining Ġin Ġneural Ġnetworks Ġhas Ġbeen Ġused Ġsuccessfully Ġin Ġseveral ĠN LP Ġtasks , Ġsuch Ġas Ġcross - ling ual ĠPOS Ġtagging ĠB IB REF 6 Ġand Ġcross - domain ĠPOS Ġtagging ĠB IB REF 7 Ġ. ĠThey Ġuse Ġit Ġto Ġreduce Ġthe Ġnegative Ġinfluences Ġof Ġthe Ġinput Ġdiver g ences Ġamong Ġdifferent Ġdomains Ġor Ġlanguages , Ġwhile Ġwe Ġuse Ġadvers arial Ġtraining Ġto Ġreduce Ġthe Ġnegative Ġinfluences Ġbrought Ġby Ġdifferent Ġcrowd Ġannot ators . ĠTo Ġour Ġbest Ġknowledge , Ġwe Ġare Ġthe Ġfirst Ġto Ġapply Ġadvers arial Ġtraining Ġfor Ġcrowd Ġannotation Ġlearning . </s> In Ġthe Ġlearning Ġframework , Ġwe Ġperform Ġadvers arial Ġtraining Ġbetween Ġthe Ġbasic ĠN ER Ġand Ġan Ġadditional Ġworker Ġdiscrim inator . ĠWe Ġhave Ġa Ġcommon ĠBi - L ST M Ġfor Ġrepresenting Ġannot ator - generic Ġinformation Ġand Ġa Ġprivate ĠBi - L ST M Ġfor Ġrepresenting Ġannot ator - specific Ġinformation . ĠWe Ġbuild Ġanother Ġlabel ĠBi - L ST M Ġby Ġthe Ġcrowd - annot ated ĠNE Ġlabel Ġsequence Ġwhich Ġreflects Ġthe Ġmind Ġof Ġthe Ġcrowd Ġannot ators Ġwho Ġlearn Ġentity Ġdefinitions Ġby Ġreading Ġthe Ġannotation Ġguide book . ĠThe Ġcommon Ġand Ġprivate ĠBi - L ST Ms Ġare Ġused Ġfor ĠN ER , Ġwhile Ġthe Ġcommon Ġand Ġlabel ĠBi - L ST Ms Ġare Ġused Ġas Ġinputs Ġfor Ġthe Ġworker Ġdiscrim inator . ĠThe Ġparameters Ġof Ġthe Ġcommon ĠBi - L ST M Ġare Ġlearned Ġby Ġadvers arial Ġtraining , Ġmaximizing Ġthe Ġworker Ġdiscrim inator Ġloss Ġand Ġmeanwhile Ġminimizing Ġthe ĠN ER Ġloss . ĠThus Ġthe Ġresulting Ġfeatures Ġof Ġthe Ġcommon ĠBi - L ST M Ġare Ġworker Ġinvari ant Ġand ĠN ER Ġsensitive . </s> For Ġevaluation , Ġwe Ġcreate Ġtwo ĠChinese ĠN ER Ġdatasets Ġin Ġtwo Ġdomains : Ġdialog Ġand Ġe - commerce . ĠWe Ġrequire Ġthe Ġcrowd Ġannot ators Ġto Ġlabel Ġthe Ġtypes Ġof Ġentities , Ġincluding Ġperson , Ġsong , Ġbrand , Ġproduct , Ġand Ġso Ġon . ĠIdent ifying Ġthese Ġentities Ġis Ġuseful Ġfor Ġchat bot Ġand Ġe - commerce Ġplatforms ĠB IB REF 8 Ġ. ĠThen Ġwe Ġconduct Ġexperiments Ġon Ġthe Ġnewly Ġcreated Ġdatasets Ġto Ġverify Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġadvers arial Ġneural Ġnetwork Ġmodel . ĠThe Ġresults Ġshow Ġthat Ġour Ġsystem Ġoutper forms Ġvery Ġstrong Ġbaseline Ġsystems . ĠIn Ġsummary , Ġwe Ġmake Ġthe Ġfollowing Ġcontributions : </s> Related ĠWork </s> Our Ġwork Ġis Ġrelated Ġto Ġthree Ġlines Ġof Ġresearch : ĠSequence Ġlabeling , ĠAd vers arial Ġtraining , Ġand ĠCrowd s ourcing . </s> Sequ ence Ġlabeling . ĠN ER Ġis Ġwidely Ġtreated Ġas Ġa Ġsequence Ġlabeling Ġproblem , Ġby Ġassigning Ġa Ġunique Ġlabel Ġover Ġeach Ġsent ential Ġword ĠB IB REF 9 Ġ. ĠEarly Ġstudies Ġon Ġsequence Ġlabeling Ġoften Ġuse Ġthe Ġmodels Ġof ĠH MM , ĠMEM M , Ġand ĠCR F ĠB IB REF 10 Ġbased Ġon Ġmanually - crafted Ġdiscrete Ġfeatures , Ġwhich Ġcan Ġsuffer Ġthe Ġfeature Ġsp arsity Ġproblem Ġand Ġrequire Ġheavy Ġfeature Ġengineering . ĠRecently , Ġneural Ġnetwork Ġmodels Ġhave Ġbeen Ġsuccessfully Ġapplied Ġto Ġsequence Ġlabeling ĠB IB REF 1 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 2 Ġ. ĠAmong Ġthese Ġwork , Ġthe Ġmodel Ġwhich Ġuses ĠBi - L ST M Ġfor Ġfeature Ġextraction Ġand ĠCR F Ġfor Ġdecoding Ġhas Ġachieved Ġstate - of - the - art Ġperformances ĠB IB REF 11 Ġ, ĠB IB REF 2 Ġ, Ġwhich Ġis Ġexploited Ġas Ġthe Ġbaseline Ġmodel Ġin Ġour Ġwork . </s> Ad vers arial ĠTraining . ĠAd vers arial ĠNetworks Ġhave Ġachieved Ġgreat Ġsuccess Ġin Ġcomputer Ġvision Ġsuch Ġas Ġimage Ġgeneration ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠIn Ġthe ĠN LP Ġcommunity , Ġthe Ġmethod Ġis Ġmainly Ġexploited Ġunder Ġthe Ġsettings Ġof Ġdomain Ġad a ption ĠB IB REF 14 Ġ, ĠB IB REF 7 Ġ, Ġcross - ling ual ĠB IB REF 15 Ġ, ĠB IB REF 6 Ġand Ġmulti - task Ġlearning ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. ĠAll Ġthese Ġsettings Ġinvolve Ġthe Ġfeature Ġdiver g ences Ġbetween Ġthe Ġtraining Ġand Ġtest Ġexamples , Ġand Ġaim Ġto Ġlearn Ġinvari ant Ġfeatures Ġacross Ġthe Ġdiver g ences Ġby Ġan Ġadditional Ġadvers arial Ġdiscrim inator , Ġsuch Ġas Ġdomain Ġdiscrim inator . ĠOur Ġwork Ġis Ġsimilar Ġto Ġthese Ġwork Ġbut Ġis Ġapplies Ġon Ġcrowds ourcing Ġlearning , Ġaiming Ġto Ġfind Ġinvari ant Ġfeatures Ġamong Ġdifferent Ġcrowds ourcing Ġworkers . </s> C rowd s ourcing . ĠMost ĠN LP Ġtasks Ġrequire Ġa Ġmassive Ġamount Ġof Ġlabeled Ġtraining Ġdata Ġwhich Ġare Ġannot ated Ġby Ġexperts . ĠHowever , Ġhiring Ġexperts Ġis Ġcostly Ġand Ġnon - sc al able , Ġboth Ġin Ġterms Ġof Ġtime Ġand Ġmoney . ĠInstead , Ġcrowds ourcing Ġis Ġanother Ġsolution Ġto Ġobtain Ġlabeled Ġdata Ġat Ġa Ġlower Ġcost Ġbut Ġwith Ġrelative Ġlower Ġquality Ġthan Ġthose Ġfrom Ġexperts . ĠB IB REF 18 Ġsnow 2008 che ap Ġcollected Ġlabeled Ġresults Ġfor Ġseveral ĠN LP Ġtasks Ġfrom ĠAmazon ĠMechanical ĠTurk Ġand Ġdemonstrated Ġthat Ġnon - exper ts Ġannotations Ġwere Ġquite Ġuseful Ġfor Ġtraining Ġnew Ġsystems . ĠIn Ġrecent Ġyears , Ġa Ġseries Ġof Ġwork Ġhave Ġfocused Ġon Ġhow Ġto Ġuse Ġcrowds ourcing Ġdata Ġefficiently Ġin Ġtasks Ġsuch Ġas Ġclassification ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, Ġand Ġcompare Ġquality Ġof Ġcrowd Ġand Ġexpert Ġlabels ĠB IB REF 21 Ġ. </s> In Ġsequence Ġlabeling Ġtasks , ĠB IB REF 22 Ġdred ze 2009 sequence Ġviewed Ġthis Ġtask Ġas Ġa Ġmulti - label Ġproblem Ġwhile ĠB IB REF 3 Ġrod rig ues 2014 sequence Ġtook Ġworkers Ġidentities Ġinto Ġaccount Ġby Ġassuming Ġthat Ġeach Ġsent ential Ġword Ġwas Ġtagged Ġcorrectly Ġby Ġone Ġof Ġthe Ġcrowds ourcing Ġworkers Ġand Ġproposed Ġa ĠCR F - based Ġmodel Ġwith Ġmultiple Ġannot ators . ĠB IB REF 4 Ġn guyen 2017 agg reg ating Ġintroduced Ġa Ġcrowd Ġrepresentation Ġin Ġwhich Ġthe Ġcrowd Ġvectors Ġwere Ġadded Ġinto Ġthe ĠL ST M - CR F Ġmodel Ġat Ġtrain Ġtime , Ġbut Ġignored Ġthem Ġat Ġtest Ġtime . ĠIn Ġthis Ġpaper , Ġwe Ġapply Ġadvers arial Ġtraining Ġon Ġcrowd Ġannotations Ġon ĠChinese ĠN ER Ġin Ġnew Ġdomains , Ġand Ġachieve Ġbetter Ġperformances Ġthan Ġprevious Ġstudies Ġon Ġcrowds ourcing Ġlearning . </s> Bas eline : ĠL ST M - CR F </s> We Ġuse Ġa Ġneural ĠCR F Ġmodel Ġas Ġthe Ġbaseline Ġsystem ĠB IB REF 9 Ġ, Ġtreating ĠN ER Ġas Ġa Ġsequence Ġlabeling Ġproblem Ġover ĠChinese Ġcharacters , Ġwhich Ġhas Ġachieved Ġstate - of - the - art Ġperformances ĠB IB REF 5 Ġ. ĠTo Ġthis Ġend , Ġwe Ġexplore Ġthe ĠBI EO Ġschema Ġto Ġconvert ĠN ER Ġinto Ġsequence Ġlabeling , Ġfollowing ĠB IB REF 2 Ġl ample - E t Al : 2016 : N 16 - 1 , Ġwhere Ġsent ential Ġcharacter Ġis Ġassigned Ġwith Ġone Ġunique Ġtag . ĠCon crete ly , Ġwe Ġtag Ġthe Ġnon - entity Ġcharacter Ġby Ġlabel ĠâĢ ľ O âĢ Ŀ , Ġthe Ġbeginning Ġcharacter Ġof Ġan Ġentity Ġby ĠâĢ ľ B - XX âĢ Ŀ , Ġthe Ġending Ġcharacter Ġof Ġan Ġentity Ġby ĠâĢ ľ E - XX âĢ Ŀ Ġand Ġthe Ġother Ġcharacter Ġof Ġan Ġentity Ġby ĠâĢ ľ I - XX âĢ Ŀ , Ġwhere ĠâĢ ľ XX âĢ Ŀ Ġdenotes Ġthe Ġentity Ġtype . </s> We Ġbuild Ġhigh - level Ġneural Ġfeatures Ġfrom Ġthe Ġinput Ġcharacter Ġsequence Ġby Ġa Ġbi - direction al ĠL ST M ĠB IB REF 2 Ġ. ĠThe Ġresulting Ġfeatures Ġare Ġcombined Ġand Ġthen Ġare Ġfed Ġinto Ġan Ġoutput ĠCR F Ġlayer Ġfor Ġdecoding . ĠIn Ġsummary , Ġthe Ġbaseline Ġmodel Ġhas Ġthree Ġmain Ġcomponents . ĠFirst , Ġwe Ġmake Ġvector Ġrepresentations Ġfor Ġsent ential Ġcharacters Ġ$\ math bf Ġ{ x } _ 1 \ math bf Ġ{ x } _ 2 \ cd ots Ġ\ math bf Ġ{ x } _ n $ Ġ, Ġtransforming Ġthe Ġdiscrete Ġinputs Ġinto Ġlow - dimensional Ġneural Ġinputs . ĠSecond , Ġfeature Ġextraction Ġis Ġperformed Ġto Ġobtain Ġhigh - level Ġfeatures Ġ$\ math bf Ġ{ h } _ 1 ^ {\ text { ner }} \ math bf Ġ{ h } _ 2 ^ {\ text { ner }} \ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { ner }} $ Ġ, Ġby Ġusing Ġa Ġbi - direction al ĠL ST M Ġ( Bi - L ST M ) Ġstructure Ġtogether Ġwith Ġa Ġlinear Ġtransformation Ġover Ġ$\ math bf Ġ{ x } _ 1 \ math bf Ġ{ x } _ 2 \ cd ots Ġ\ math bf Ġ{ x } _ n $ Ġ. ĠThird , Ġwe Ġapply Ġa ĠCR F Ġtagging Ġmodule Ġover Ġ$\ math bf Ġ{ h } _ 1 ^ {\ text { ner }} \ math bf Ġ{ h } _ 2 ^ {\ text { ner }} \ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { ner }} $ Ġ, Ġobtaining Ġthe Ġfinal Ġoutput ĠNE Ġlabels . ĠThe Ġoverall Ġframework Ġof Ġthe Ġbaseline Ġmodel Ġis Ġshown Ġby Ġthe Ġright Ġpart Ġof ĠFigure Ġ1 Ġ. </s> Vector ĠRepresent ation Ġof ĠCharacters </s> To Ġrepresent ĠChinese Ġcharacters , Ġwe Ġsimply Ġexploit Ġa Ġneural Ġembed ding Ġlayer Ġto Ġmap Ġdiscrete Ġcharacters Ġinto Ġthe Ġlow - dimensional Ġvector Ġrepresentations . ĠThe Ġgoal Ġis Ġachieved Ġby Ġa Ġlooking - up Ġtable Ġ$\ math bf Ġ{ E } ^ W $ Ġ, Ġwhich Ġis Ġa Ġmodel Ġparameter Ġand Ġwill Ġbe Ġfine - tun ed Ġduring Ġtraining . ĠThe Ġlooking - up Ġtable Ġcan Ġbe Ġinitialized Ġeither Ġby Ġrandom Ġor Ġby Ġusing Ġa Ġpret rained Ġembed d ings Ġfrom Ġlarge Ġscale Ġraw Ġcorpus . ĠFor Ġa Ġgiven ĠChinese Ġcharacter Ġsequence Ġ$ c _ 1 c _ 2 \ cd ots Ġc _ n $ Ġ, Ġwe Ġobtain Ġthe Ġvector Ġrepresentation Ġof Ġeach Ġsent ential Ġcharacter Ġby : Ġ$ Ġ\ math bf Ġ{ x } _ t Ġ= Ġ\ text { look - up } ( c _ t , Ġ\ math bf Ġ{ E } ^ W ), Ġ\ text { ~~ ~ } Ġt Ġ\ in Ġ[ 1 , Ġn ] $ Ġ. </s> Feature ĠExt raction </s> Based Ġon Ġthe Ġvector Ġsequence Ġ$\ math bf Ġ{ x } _ 1 \ math bf Ġ{ x } _ 2 \ cd ots Ġ\ math bf Ġ{ x } _ n $ Ġ, Ġwe Ġextract Ġhigher - level Ġfeatures Ġ$\ math bf Ġ{ h } _ 1 ^ {\ text { ner }} \ math bf Ġ{ h } _ 2 ^ {\ text { ner }} \ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { ner }} $ Ġby Ġusing Ġa Ġbid irection al ĠL ST M Ġmodule Ġand Ġa Ġsimple Ġfeed - forward Ġneural Ġlayer , Ġwhich Ġare Ġthen Ġused Ġfor ĠCR F Ġtagging Ġat Ġthe Ġnext Ġstep . </s> L ST M Ġis Ġa Ġtype Ġof Ġrecurrent Ġneural Ġnetwork Ġ( R NN ), Ġwhich Ġis Ġdesigned Ġfor Ġsolving Ġthe Ġexploding Ġand Ġdiminishing Ġgrad ients Ġof Ġbasic ĠR NN s ĠB IB REF 23 Ġ. ĠIt Ġhas Ġbeen Ġwidely Ġused Ġin Ġa Ġnumber Ġof ĠN LP Ġtasks , Ġincluding ĠPOS - tag ging ĠB IB REF 11 Ġ, ĠB IB REF 24 Ġ, Ġparsing ĠB IB REF 25 Ġand Ġmachine Ġtranslation ĠB IB REF 26 Ġ, Ġbecause Ġof Ġits Ġstrong Ġcapabilities Ġof Ġmodeling Ġnatural Ġlanguage Ġsentences . </s> By Ġtravers ing Ġ$\ math bf Ġ{ x } _ 1 \ math bf Ġ{ x } _ 2 \ cd ots Ġ\ math bf Ġ{ x } _ n $ Ġby Ġorder Ġand Ġreverse ly , Ġwe Ġobtain Ġthe Ġoutput Ġfeatures Ġ$\ math bf Ġ{ h } _ 1 ^ {\ text { private }} \ math bf Ġ{ h } _ 2 ^ {\ text { private }} \ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { private }} $ Ġof Ġthe Ġbi - L ST M , Ġwhere Ġ$\ math bf Ġ{ h } _ t ^ {\ text { private }} Ġ= Ġ\ over right arrow {\ math bf Ġ{ h }} _ t Ġ\ opl us Ġ\ over left arrow {\ math bf Ġ{ h }} _ t Ġ$ Ġ. ĠHere Ġwe Ġrefer Ġthis ĠBi - L ST M Ġas Ġprivate Ġin Ġorder Ġto Ġdifferentiate Ġit Ġwith Ġthe Ġcommon ĠBi - L ST M Ġover Ġthe Ġsame Ġcharacter Ġinputs Ġwhich Ġwill Ġbe Ġintroduced Ġin Ġthe Ġnext Ġsection . </s> Further Ġwe Ġmake Ġan Ġintegration Ġof Ġthe Ġoutput Ġvectors Ġof Ġbi - direction al ĠL ST M Ġby Ġa Ġlinear Ġfeed - forward Ġneural Ġlayer , Ġresulting Ġin Ġthe Ġfeatures Ġ$\ math bf Ġ{ h } _ 1 ^ {\ text { ner }} \ math bf Ġ{ h } _ 2 ^ {\ text { ner }} \ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { ner }} $ Ġby Ġequation : </s> $$ \ math bf Ġ{ h } _ t ^ {\ text { ner }} Ġ= Ġ\ math bf Ġ{ W } Ġ\ math bf Ġ{ h } _ t ^ {\ text { private }} Ġ+ Ġ\ math bf Ġ{ b }, $$ Ġ Ġ Ġ( E q . Ġ6 ) </s> where Ġ$\ math bf Ġ{ W } $ Ġand Ġ$\ math bf Ġ{ b } $ Ġare Ġboth Ġmodel Ġparameters . </s> CR F ĠT agging </s> Finally Ġwe Ġfeed Ġthe Ġresulting Ġfeatures Ġ$\ math bf Ġ{ h } _ t ^ {\ text { ner } }, Ġt \ in Ġ[ 1 , Ġn ] $ Ġinto Ġa ĠCR F Ġlayer Ġdirectly Ġfor ĠN ER Ġdecoding . ĠCR F Ġtagging Ġis Ġone Ġglobally Ġnormalized Ġmodel , Ġaiming Ġto Ġfind Ġthe Ġbest Ġoutput Ġsequence Ġconsidering Ġthe Ġdependencies Ġbetween Ġsuccessive Ġlabels . ĠIn Ġthe Ġsequence Ġlabeling Ġsetting Ġfor ĠN ER , Ġthe Ġoutput Ġlabel Ġof Ġone Ġposition Ġhas Ġa Ġstrong Ġdependency Ġon Ġthe Ġlabel Ġof Ġthe Ġprevious Ġposition . ĠFor Ġexample , Ġthe Ġlabel Ġbefore ĠâĢ ľ I - XX âĢ Ŀ Ġmust Ġbe Ġeither ĠâĢ ľ B - XX âĢ Ŀ Ġor ĠâĢ ľ I - XX âĢ Ŀ , Ġwhere ĠâĢ ľ XX âĢ Ŀ Ġshould Ġbe Ġexactly Ġthe Ġsame . </s> CR F Ġinvolves Ġtwo Ġparts Ġfor Ġprediction . ĠFirst Ġwe Ġshould Ġcompute Ġthe Ġscores Ġfor Ġeach Ġlabel Ġbased Ġ$\ math bf Ġ{ h } _ t ^ {\ text { ner }} $ Ġ, Ġresulting Ġin Ġ$\ math bf Ġ{ o } _ t ^ {\ text { ner }} $ Ġ, Ġwhose Ġdimension Ġis Ġthe Ġnumber Ġof Ġoutput Ġlabels . ĠThe Ġother Ġpart Ġis Ġa Ġtransition Ġmatrix Ġ$\ math bf Ġ{ T } $ Ġwhich Ġdefines Ġthe Ġscores Ġof Ġtwo Ġsuccessive Ġlabels . Ġ$\ math bf Ġ{ T } $ Ġis Ġalso Ġa Ġmodel Ġparameter . ĠBased Ġon Ġ$\ math bf Ġ{ o } _ t ^ {\ text { ner }} $ Ġand Ġ$\ math bf Ġ{ T } $ Ġ, Ġwe Ġuse Ġthe ĠV iter bi Ġalgorithm Ġto Ġfind Ġthe Ġbest - scoring Ġlabel Ġsequence . </s> We Ġcan Ġformal ize Ġthe ĠCR F Ġtagging Ġprocess Ġas Ġfollows : </s> $$ \ begin { split } Ġ& Ġ\ math bf Ġ{ o } _ t ^ {\ text { ner }} Ġ= Ġ\ math bf Ġ{ W } ^ {\ text { ner }} Ġ\ math bf Ġ{ h } _ t ^ {\ text { ner } }, Ġ\ text { ~~~~ } Ġt Ġ\ in Ġ[ 1 , n ] Ġ\\ Ġ& Ġ\ text { score } (\ math bf Ġ{ X }, Ġ\ math bf Ġ{ y }) Ġ= Ġ\ sum Ġ_ { t Ġ= Ġ1 } ^{ n } (\ math bf Ġ{ o } _{ t , y _ t } Ġ+ ĠT _{ y _{ t - 1 }, y _ t }) Ġ\\ Ġ& Ġ\ math bf Ġ{ y } ^ {\ text { ner }} Ġ= Ġ\ math op Ġ{ arg ~ max } _{ \ math bf Ġ{ y }} \ big Ġ( \ text { score } (\ math bf Ġ{ X }, Ġ\ math bf Ġ{ y } )) \ big Ġ), Ġ\\ Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> where Ġ$\ text { score } (\ cd ot Ġ) $ Ġis Ġthe Ġscoring Ġfunction Ġfor Ġa Ġgiven Ġoutput Ġlabel Ġsequence Ġ$\ math bf Ġ{ y } Ġ= Ġy _ 1 y _ 2 Ġ\ cd ots Ġy _ n $ Ġbased Ġon Ġinput Ġ$\ math bf Ġ{ X } $ Ġ, Ġ$\ math bf Ġ{ y } ^ {\ text { ner }} $ Ġis Ġthe Ġresulting Ġlabel Ġsequence , Ġ$\ math bf Ġ{ W } ^ {\ text { ner }} $ Ġis Ġa Ġmodel Ġparameter . </s> Training </s> To Ġtrain Ġmodel Ġparameters , Ġwe Ġexploit Ġa Ġnegative Ġlog - like lihood Ġobjective Ġas Ġthe Ġloss Ġfunction . ĠWe Ġapply Ġsoft max Ġover Ġall Ġcandidate Ġoutput Ġlabel Ġsequences , Ġthus Ġthe Ġprobability Ġof Ġthe Ġcrowd - annot ated Ġlabel Ġsequence Ġis Ġcomputed Ġby : </s> $$ p (\ math bf Ġ{\ bar { y }} | \ math bf Ġ{ X }) Ġ= Ġ\ frac {\ exp Ġ\ big Ġ( \ text { score } (\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y }} )\ big Ġ) }{ \ sum Ġ_ {\ math bf Ġ{ y } Ġ\ in Ġ\ math bf Ġ{ Y } _{ \ math bf Ġ{ X }}} Ġ\ exp Ġ\ big Ġ( \ text { score } (\ math bf Ġ{ X }, Ġ\ math bf Ġ{ y } )\ big Ġ) }, $$ Ġ Ġ Ġ( E q . Ġ10 ) </s> where Ġ$\ math bf Ġ{\ bar { y }} $ Ġis Ġthe Ġcrowd - annot ated Ġlabel Ġsequences Ġand Ġ$\ math bf Ġ{ Y } _{ \ math bf Ġ{ X }} $ Ġis Ġall Ġcandidate Ġlabel Ġsequence Ġof Ġinput Ġ$\ math bf Ġ{ X } $ Ġ. </s> Based Ġon Ġthe Ġabove Ġformula , Ġthe Ġloss Ġfunction Ġof Ġour Ġbaseline Ġmodel Ġis : </s> $$ \ text { loss } (\ The ta Ġ, Ġ\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y }} ) Ġ= Ġ- \ log Ġp (\ math bf Ġ{\ bar { y }} | \ math bf Ġ{ X } ), $$ Ġ Ġ Ġ( E q . Ġ11 ) </s> where Ġ$\ The ta Ġ$ Ġis Ġthe Ġset Ġof Ġall Ġmodel Ġparameters . ĠWe Ġuse Ġstandard Ġback - prop ag ation Ġmethod Ġto Ġminimize Ġthe Ġloss Ġfunction Ġof Ġthe Ġbaseline ĠCR F Ġmodel . </s> Work er ĠAd vers arial </s> Ad vers arial Ġlearning Ġhas Ġbeen Ġan Ġeffective Ġmechanism Ġto Ġresolve Ġthe Ġproblem Ġof Ġthe Ġinput Ġfeatures Ġbetween Ġthe Ġtraining Ġand Ġtest Ġexamples Ġhaving Ġlarge Ġdiver g ences ĠB IB REF 27 Ġ, ĠB IB REF 13 Ġ. ĠIt Ġhas Ġbeen Ġsuccessfully Ġapplied Ġon Ġdomain Ġad a ption ĠB IB REF 7 Ġ, Ġcross - ling ual Ġlearning ĠB IB REF 15 Ġand Ġmulti - task Ġlearning ĠB IB REF 17 Ġ. ĠAll Ġsettings Ġinvolve Ġfeature Ġshifting Ġbetween Ġthe Ġtraining Ġand Ġtesting . </s> In Ġthis Ġpaper , Ġour Ġsetting Ġis Ġdifferent . ĠWe Ġare Ġusing Ġthe Ġannotations Ġfrom Ġnon - exper ts , Ġwhich Ġare Ġnoise Ġand Ġcan Ġinfluence Ġthe Ġfinal Ġperformances Ġif Ġthey Ġare Ġnot Ġproperly Ġprocessed . ĠDirect ly Ġlearning Ġbased Ġon Ġthe Ġresulting Ġcorpus Ġmay Ġadapt Ġthe Ġneural Ġfeature Ġextraction Ġinto Ġthe Ġbiased Ġannotations . ĠIn Ġthis Ġwork , Ġwe Ġassume Ġthat Ġindividual Ġworkers Ġhave Ġtheir Ġown Ġguidelines Ġin Ġmind Ġafter Ġshort Ġtraining . ĠFor Ġexample , Ġa Ġperfect Ġworker Ġcan Ġannot ate Ġhighly Ġconsistently Ġwith Ġan Ġexpert , Ġwhile Ġcommon Ġcrowds ourcing Ġworkers Ġmay Ġbe Ġconfused Ġand Ġhave Ġdifferent Ġunderstand ings Ġon Ġcertain Ġcontexts . ĠBased Ġon Ġthe Ġassumption , Ġwe Ġmake Ġan Ġad a ption Ġfor Ġthe Ġoriginal Ġadvers arial Ġneural Ġnetwork Ġto Ġour Ġsetting . </s> Our Ġad a ption Ġis Ġvery Ġsimple . ĠBrief ly Ġspeaking , Ġthe Ġoriginal Ġadvers arial Ġlearning Ġadds Ġan Ġadditional Ġdiscrim inator Ġto Ġclassify Ġthe Ġtype Ġof Ġsource Ġinputs , Ġfor Ġexample , Ġthe Ġdomain Ġcategory Ġin Ġthe Ġdomain Ġad a ption Ġsetting , Ġwhile Ġwe Ġadd Ġa Ġdiscrim inator Ġto Ġclassify Ġthe Ġannotation Ġworkers . ĠSole ly Ġthe Ġfeatures Ġfrom Ġthe Ġinput Ġsentence Ġis Ġnot Ġenough Ġfor Ġworker Ġclassification . ĠThe Ġannotation Ġresult Ġof Ġthe Ġworker Ġis Ġalso Ġrequired . ĠThus Ġthe Ġinputs Ġof Ġour Ġdiscrim inator Ġare Ġdifferent . ĠHere Ġwe Ġexploit Ġboth Ġthe Ġsource Ġsentences Ġand Ġthe Ġcrowd - annot ated ĠNE Ġlabels Ġas Ġbasic Ġinputs Ġfor Ġthe Ġworker Ġdiscrimination . </s> In Ġthe Ġfollowing , Ġwe Ġdescribe Ġthe Ġproposed Ġadvers arial Ġlearning Ġmodule , Ġincluding Ġboth Ġthe Ġsub models Ġand Ġthe Ġtraining Ġmethod . ĠAs Ġshown Ġby Ġthe Ġleft Ġpart Ġof ĠFigure Ġ1 Ġ, Ġthe Ġsub model Ġconsists Ġof Ġfour Ġparts : Ġ( 1 ) Ġa Ġcommon ĠBi - L ST M Ġover Ġinput Ġcharacters ; Ġ( 2 ) Ġan Ġadditional ĠBi - L ST M Ġto Ġencode Ġcrowd - annot ated ĠNE Ġlabel Ġsequence ; Ġ( 3 ) Ġa Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġto Ġextract Ġfeatures Ġfor Ġworker Ġdiscrim inator ; Ġ( 4 ) Ġoutput Ġand Ġprediction . </s> Common ĠBi - L ST M Ġover ĠCharacters </s> To Ġbuild Ġthe Ġadvers arial Ġpart , Ġfirst Ġwe Ġcreate Ġa Ġnew Ġbi - direction al ĠL ST M , Ġnamed Ġby Ġthe Ġcommon ĠBi - L ST M : </s> $$ \ math bf Ġ{ h } _ 1 ^ {\ text {\ tiny Ġcommon }} Ġ\ math bf Ġ{ h } _ 2 ^ {\ text {\ tiny Ġcommon }} Ġ\ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text {\ tiny Ġcommon }} Ġ= Ġ\ text { Bi - L ST M } (\ math bf Ġ{ x } _ 1 \ math bf Ġ{ x } _ 2 \ cd ots Ġ\ math bf Ġ{ x } _ n ). $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> As Ġshown Ġin ĠFigure Ġ1 Ġ, Ġthis ĠBi - L ST M Ġis Ġconstructed Ġover Ġthe Ġsame Ġinput Ġcharacter Ġrepresentations Ġof Ġthe Ġprivate ĠBi - L ST M , Ġin Ġorder Ġto Ġextract Ġworker Ġindependent Ġfeatures . </s> The Ġresulting Ġfeatures Ġof Ġthe Ġcommon ĠBi - L ST M Ġare Ġused Ġfor Ġboth ĠN ER Ġand Ġthe Ġworker Ġdiscrim inator , Ġdifferent Ġwith Ġthe Ġfeatures Ġof Ġprivate ĠBi - L ST M Ġwhich Ġare Ġused Ġfor ĠN ER Ġonly . ĠAs Ġshown Ġin ĠFigure Ġ1 Ġ, Ġwe Ġconc aten ate Ġthe Ġoutputs Ġof Ġthe Ġcommon Ġand Ġprivate ĠBi - L ST Ms Ġtogether , Ġand Ġthen Ġfeed Ġthe Ġresults Ġinto Ġthe Ġfeed - forward Ġcombination Ġlayer Ġof Ġthe ĠN ER Ġpart . ĠThus ĠFormula Ġ6 Ġcan Ġbe Ġrewritten Ġas : </s> $$ \ math bf Ġ{ h } _ t ^ {\ text { ner }} Ġ= Ġ\ math bf Ġ{ W } Ġ( \ math bf Ġ{ h } _ t ^ {\ text { common }} Ġ\ opl us Ġ\ math bf Ġ{ h } _ t ^ {\ text { private }} ) Ġ+ Ġ\ math bf Ġ{ b }, $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$\ math bf Ġ{ W } $ Ġis Ġwider Ġthan Ġthe Ġoriginal Ġcombination Ġbecause Ġthe Ġnewly - added Ġ$\ math bf Ġ{ h } _ t ^ {\ text { common }} $ Ġ. </s> Notice ably , Ġalthough Ġthe Ġresulting Ġcommon Ġfeatures Ġare Ġused Ġfor Ġthe Ġworker Ġdiscrim inator , Ġthey Ġactually Ġhave Ġno Ġcapability Ġto Ġdistinguish Ġthe Ġworkers . ĠBecause Ġthis Ġpart Ġis Ġexploited Ġto Ġmaximize Ġthe Ġloss Ġof Ġthe Ġworker Ġdiscrim inator , Ġit Ġwill Ġbe Ġinterpreted Ġin Ġthe Ġlater Ġtraining Ġsubsection . ĠThese Ġfeatures Ġare Ġinvari ant Ġamong Ġdifferent Ġworkers , Ġthus Ġthey Ġcan Ġhave Ġless Ġnoises Ġfor ĠN ER . ĠThis Ġis Ġthe Ġgoal Ġof Ġadvers arial Ġlearning , Ġand Ġwe Ġhope Ġthe ĠN ER Ġbeing Ġable Ġto Ġfind Ġuseful Ġfeatures Ġfrom Ġthese Ġworker Ġindependent Ġfeatures . </s> Additional ĠBi - L ST M Ġover ĠAn not ated ĠN ER ĠLab els </s> In Ġorder Ġto Ġincorporate Ġthe Ġannot ated ĠNE Ġlabels Ġto Ġpredict Ġthe Ġexact Ġworker , Ġwe Ġbuild Ġanother Ġbi - direction al ĠL ST M Ġ( named Ġby Ġlabel ĠBi - L ST M ) Ġbased Ġon Ġthe Ġcrowd - annot ated ĠNE Ġlabel Ġsequence . ĠThis ĠBi - L ST M Ġis Ġused Ġfor Ġworker Ġdiscrim inator Ġonly . ĠDuring Ġthe Ġdecoding Ġof Ġthe Ġtesting Ġphase , Ġwe Ġwill Ġnever Ġhave Ġthis ĠBi - L ST M , Ġbecause Ġthe Ġworker Ġdiscrim inator Ġis Ġno Ġlonger Ġrequired . </s> Assuming Ġthe Ġcrowd - annot ated ĠNE Ġlabel Ġsequence Ġannot ated Ġby Ġone Ġworker Ġis Ġ$\ math bf Ġ{\ bar { y }} Ġ= Ġ\ bar { y } _ 1 \ bar { y } _ 2 Ġ\ cd ots Ġ\ bar { y } _ n $ Ġ, Ġwe Ġexploit Ġa Ġlooking - up Ġtable Ġ$\ math bf Ġ{ E } ^{ L } $ Ġto Ġobtain Ġthe Ġcorresponding Ġsequence Ġof Ġtheir Ġvector Ġrepresentations Ġ$\ math bf Ġ{ x ^ {\ prime Ġ}} _ 1 \ math bf Ġ{ x ^ {\ prime Ġ}} _ 2 \ cd ots Ġ\ math bf Ġ{ x ^ {\ prime Ġ}} _ n $ Ġ, Ġsimilar Ġto Ġthe Ġmethod Ġthat Ġmaps Ġcharacters Ġinto Ġtheir Ġneural Ġrepresentations . ĠCon crete ly , Ġfor Ġone ĠNE Ġlabel Ġ$\ bar { y } _ t $ Ġ( Ġ$ t Ġ\ in Ġ[ 1 , Ġn ] $ Ġ), Ġwe Ġobtain Ġits Ġneural Ġvector Ġby : Ġ$\ math bf Ġ{ x ^ {\ prime Ġ}} _ t Ġ= Ġ\ text { look - up } (\ bar { y } _ t , Ġ\ math bf Ġ{ E } ^ L ) $ Ġ. </s> Next Ġstep Ġwe Ġapply Ġbi - direction al ĠL ST M Ġover Ġthe Ġsequence Ġ$\ math bf Ġ{ x ^ {\ prime Ġ}} _ 1 \ math bf Ġ{ x ^ {\ prime Ġ}} _ 2 \ cd ots Ġ\ math bf Ġ{ x ^ {\ prime Ġ}} _ n $ Ġ, Ġwhich Ġcan Ġbe Ġformal ized Ġas : </s> $$ \ math bf Ġ{ h } _ 1 ^ {\ text { label }} Ġ\ math bf Ġ{ h } _ 2 ^ {\ text { label }} Ġ\ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { label }} Ġ= Ġ\ text { Bi - L ST M } (\ math bf Ġ{ x ^ {\ prime Ġ}} _ 1 \ math bf Ġ{ x ^ {\ prime Ġ}} _ 2 \ cd ots Ġ\ math bf Ġ{ x ^ {\ prime Ġ}} _ n ). $$ Ġ Ġ Ġ( E q . Ġ16 ) </s> The Ġresulting Ġfeature Ġsequence Ġis Ġconc aten ated Ġwith Ġthe Ġoutputs Ġof Ġthe Ġcommon ĠBi - L ST M , Ġand Ġfurther Ġbe Ġused Ġfor Ġworker Ġclassification . </s> CNN </s> Following , Ġwe Ġadd Ġa Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġmodule Ġbased Ġon Ġthe Ġconc aten ated Ġoutputs Ġof Ġthe Ġcommon ĠBi - L ST M Ġand Ġthe Ġlabel ĠBi - L ST M , Ġto Ġproduce Ġthe Ġfinal Ġfeatures Ġfor Ġworker Ġdiscrim inator . ĠA Ġconv olution al Ġoperator Ġwith Ġwindow Ġsize Ġ5 Ġis Ġused , Ġand Ġthen Ġmax Ġpool ing Ġstrategy Ġis Ġapplied Ġover Ġthe Ġconv olution Ġsequence Ġto Ġobtain Ġthe Ġfinal Ġfixed - dimensional Ġfeature Ġvector . ĠThe Ġwhole Ġprocess Ġcan Ġbe Ġdescribed Ġby Ġthe Ġfollowing Ġequations : </s> $$ \ begin { split } Ġ& \ math bf Ġ{ h } _ t ^ {\ text { worker }} Ġ= Ġ\ math bf Ġ{ h } _ t ^ {\ text { common }} Ġ\ opl us Ġ\ math bf Ġ{ h } _ t ^ {\ text { label }} Ġ\\ Ġ& \ math bf Ġ{\ t ilde { h }} _ t ^ {\ text { worker }} Ġ= Ġ\ tan h Ġ( \ math bf Ġ{ W } ^ {\ text { c nn }} [ \ math bf Ġ{ h } _{ t - 2 } ^ {\ text { worker } }, Ġ\ math bf Ġ{ h } _{ t - 1 } ^ {\ text { worker } }, Ġ\ cd ots Ġ, Ġ\ math bf Ġ{ h } _{ t + 2 } ^ {\ text { worker }} ]) Ġ\\ Ġ& \ math bf Ġ{ h } ^ {\ text { worker }} Ġ= Ġ\ text { max - pool ing } (\ math bf Ġ{\ t ilde { h }} _ 1 ^ {\ text { worker }} \ math bf Ġ{\ t ilde { h }} _ 2 ^ {\ text { worker }} Ġ\ cd ots Ġ\ math bf Ġ{\ t ilde { h }} _ n ^ {\ text { worker }} ) Ġ\\ Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ18 ) </s> where Ġ$ t Ġ\ in Ġ[ 1 , n ] $ Ġand Ġ$\ math bf Ġ{ W } ^ {\ text { c nn }} $ Ġis Ġone Ġmodel Ġparameter . ĠWe Ġexploit Ġzero Ġvector Ġto Ġpaddle Ġthe Ġout - of - index Ġvectors . </s> Output Ġand ĠPrediction </s> After Ġobtaining Ġthe Ġfinal Ġfeature Ġvector Ġfor Ġthe Ġworker Ġdiscrim inator , Ġwe Ġuse Ġit Ġto Ġcompute Ġthe Ġoutput Ġvector , Ġwhich Ġscores Ġall Ġthe Ġannotation Ġworkers . ĠThe Ġscore Ġfunction Ġis Ġdefined Ġby : </s> $$ \ math bf Ġ{ o } ^ {\ text { worker }} Ġ= Ġ\ math bf Ġ{ W } ^ {\ text { worker }} Ġ\ math bf Ġ{ h } ^ {\ text { worker } }, $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> where Ġ$\ math bf Ġ{ W } ^ {\ text { worker }} $ Ġis Ġone Ġmodel Ġparameter Ġand Ġthe Ġoutput Ġdimension Ġequals Ġthe Ġnumber Ġof Ġtotal Ġnon - ex pert Ġannot ators . ĠThe Ġprediction Ġis Ġto Ġfind Ġthe Ġworker Ġwhich Ġis Ġresponsible Ġfor Ġthis Ġannotation . </s> Ad vers arial ĠTraining </s> The Ġtraining Ġobjective Ġwith Ġadvers arial Ġneural Ġnetwork Ġis Ġdifferent Ġfrom Ġthe Ġbaseline Ġmodel , Ġas Ġit Ġincludes Ġthe Ġextra Ġworker Ġdiscrim inator . ĠThus Ġthe Ġnew Ġobjective Ġincludes Ġtwo Ġparts , Ġone Ġbeing Ġthe Ġnegative Ġlog - like lihood Ġfrom ĠN ER Ġwhich Ġis Ġthe Ġsame Ġas Ġthe Ġbaseline , Ġand Ġthe Ġother Ġbeing Ġthe Ġnegative Ġthe Ġlog - like lihood Ġfrom Ġthe Ġworker Ġdiscrim inator . </s> In Ġorder Ġto Ġobtain Ġthe Ġnegative Ġlog - like lihood Ġof Ġthe Ġworker Ġdiscrim inator , Ġwe Ġuse Ġsoft max Ġto Ġcompute Ġthe Ġprobability Ġof Ġthe Ġactual Ġworker Ġ$\ bar { z } $ Ġas Ġwell , Ġwhich Ġis Ġdefined Ġby : </s> $$ p (\ bar { z } | \ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y }} ) Ġ= Ġ\ frac {\ exp Ġ( \ math bf Ġ{ o } ^ {\ text { worker }} _{ \ bar { z }} ) }{ \ sum Ġ_ { z } Ġ\ exp Ġ( \ math bf Ġ{ o } ^ {\ text { worker }} _ z ) }, $$ Ġ Ġ Ġ( E q . Ġ22 ) </s> where Ġ$ z $ Ġshould Ġenumer ate Ġall Ġworkers . </s> Based Ġon Ġthe Ġabove Ġdefinition Ġof Ġprobability , Ġour Ġnew Ġobjective Ġis Ġdefined Ġas Ġfollows : </s> $$ \ begin { split } Ġ\ text { R } (\ The ta Ġ, Ġ\ The ta Ġ^ {\ prime Ġ}, Ġ\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y } }, Ġ\ bar { z }) Ġ& = Ġ\ text { loss } (\ The ta Ġ, Ġ\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y }} ) Ġ- Ġ\ text { loss } (\ The ta Ġ, Ġ\ The ta Ġ^ {\ prime Ġ}, Ġ\ math bf Ġ{ X }) Ġ\\ Ġ\ text { ~~~~ ~~ } Ġ& = Ġ- \ log Ġp (\ math bf Ġ{\ bar { y }} | \ math bf Ġ{ X }) Ġ+ Ġ\ log Ġp (\ bar { z } | \ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y }} ), Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġ$\ The ta Ġ$ Ġis Ġthe Ġset Ġof Ġall Ġmodel Ġparameters Ġrelated Ġto ĠN ER , Ġand Ġ$\ The ta Ġ^ {\ prime Ġ} $ Ġis Ġthe Ġset Ġof Ġthe Ġremaining Ġparameters Ġwhich Ġare Ġonly Ġrelated Ġto Ġthe Ġworker Ġdiscrim inator , Ġ$\ math bf Ġ{ X } $ Ġ, Ġ$\ math bf Ġ{\ bar { y }} $ Ġand Ġ$\ bar { z } $ Ġare Ġthe Ġinput Ġsentence , Ġthe Ġcrowd - annot ated ĠNE Ġlabels Ġand Ġthe Ġcorresponding Ġannot ator Ġfor Ġthis Ġannotation , Ġrespectively . ĠIt Ġis Ġworth Ġnoting Ġthat Ġthe Ġparameters Ġof Ġthe Ġcommon ĠBi - L ST M Ġare Ġincluded Ġin Ġthe Ġset Ġof Ġ$\ The ta Ġ$ Ġby Ġdefinition . </s> In Ġparticular , Ġour Ġgoal Ġis Ġnot Ġto Ġsimply Ġminimize Ġthe Ġnew Ġobjective . ĠActually , Ġwe Ġaim Ġfor Ġa Ġsaddle Ġpoint , Ġfinding Ġthe Ġparameters Ġ$\ The ta Ġ$ Ġand Ġ$\ The ta Ġ^ {\ prime Ġ} $ Ġsatisfying Ġthe Ġfollowing Ġconditions : </s> $$ \ begin { split } Ġ\ hat {\ The ta Ġ} Ġ& = Ġ\ math op Ġ{ arg ~ min } _{ \ The ta Ġ} \ text { R } (\ The ta Ġ, Ġ\ The ta Ġ^ {\ prime Ġ}, Ġ\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y } }, Ġ\ bar { z }) Ġ\\ Ġ\ hat {\ The ta Ġ} ^ {\ prime Ġ} Ġ& = Ġ\ math op Ġ{ arg ~ max } _{ \ The ta Ġ^ {\ prime Ġ} }\ text { R } (\ hat {\ The ta Ġ}, Ġ\ The ta Ġ^ {\ prime Ġ}, Ġ\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y } }, Ġ\ bar { z }) Ġ\\ Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ24 ) </s> where Ġthe Ġfirst Ġequation Ġaims Ġto Ġfind Ġone Ġ$\ The ta Ġ$ Ġthat Ġminim izes Ġour Ġnew Ġobjective Ġ$\ text { R } (\ cd ot Ġ) $ Ġ, Ġand Ġthe Ġsecond Ġequation Ġaims Ġto Ġfind Ġone Ġ$\ The ta Ġ^ {\ prime Ġ} $ Ġmaximizing Ġthe Ġsame Ġobjective . </s> Int uitive ly , Ġthe Ġfirst Ġequation Ġof ĠFormula Ġ24 Ġtries Ġto Ġminimize Ġthe ĠN ER Ġloss , Ġbut Ġat Ġthe Ġsame Ġtime Ġmaximize Ġthe Ġworker Ġdiscrim inator Ġloss Ġby Ġthe Ġshared Ġparameters Ġof Ġthe Ġcommon ĠBi - L ST M . ĠThus Ġthe Ġresulting Ġfeatures Ġof Ġcommon ĠBi - L ST M Ġactually Ġattempt Ġto Ġhurt Ġthe Ġworker Ġdiscrim inator , Ġwhich Ġmakes Ġthese Ġfeatures Ġworker Ġindependent Ġsince Ġthey Ġare Ġunable Ġto Ġdistinguish Ġdifferent Ġworkers . ĠThe Ġsecond Ġequation Ġtries Ġto Ġminimize Ġthe Ġworker Ġdiscrim inator Ġloss Ġby Ġits Ġown Ġparameter Ġ$\ The ta Ġ^ {\ prime Ġ} $ Ġ. </s> We Ġuse Ġthe Ġstandard Ġback - prop ag ation Ġmethod Ġto Ġtrain Ġthe Ġmodel Ġparameters , Ġthe Ġsame Ġas Ġthe Ġbaseline Ġmodel . ĠIn Ġorder Ġto Ġincorporate Ġthe Ġterm Ġof Ġthe Ġarg max Ġpart Ġof ĠFormula Ġ24 Ġ, Ġwe Ġfollow Ġthe Ġprevious Ġwork Ġof Ġadvers arial Ġtraining ĠB IB REF 13 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 17 Ġ, Ġby Ġintroducing Ġa Ġgradient Ġreverse Ġlayer Ġbetween Ġthe Ġcommon ĠBi - L ST M Ġand Ġthe ĠCNN Ġmodule , Ġwhose Ġforward Ġdoes Ġnothing Ġbut Ġthe Ġbackward Ġsimply Ġneg ates Ġthe Ġgrad ients . </s> Data ĠSets </s> With Ġthe Ġpurpose Ġof Ġobtaining Ġevaluation Ġdatasets Ġfrom Ġcrowd Ġannot ators , Ġwe Ġcollect Ġthe Ġsentences Ġfrom Ġtwo Ġdomains : ĠDial og Ġand ĠE - commerce Ġdomain . ĠWe Ġhire Ġundergraduate Ġstudents Ġto Ġannot ate Ġthe Ġsentences . ĠThey Ġare Ġrequired Ġto Ġidentify Ġthe Ġpred efined Ġtypes Ġof Ġentities Ġin Ġthe Ġsentences . ĠTogether Ġwith Ġthe Ġguideline Ġdocument , Ġthe Ġannot ators Ġare Ġeducated Ġsome Ġtips Ġin Ġfifteen Ġminutes Ġand Ġalso Ġprovided Ġwith Ġ20 Ġexempl ifying Ġsentences . </s> Label ed ĠData : ĠDL - PS . ĠIn ĠDial og Ġdomain Ġ( DL ), Ġwe Ġcollect Ġraw Ġsentences Ġfrom Ġa Ġchat bot Ġapplication . ĠAnd Ġthen Ġwe Ġrandomly Ġselect Ġ20 K Ġsentences Ġas Ġour Ġpool Ġand Ġhire Ġ43 Ġstudents Ġto Ġannot ate Ġthe Ġsentences . ĠWe Ġask Ġthe Ġannot ators Ġto Ġlabel Ġtwo Ġtypes Ġof Ġentities : ĠPerson - Name Ġand ĠSong - Name . ĠThe Ġannot ators Ġlabel Ġthe Ġsentences Ġindependently . ĠIn Ġparticular , Ġeach Ġsentence Ġis Ġassigned Ġto Ġthree Ġannot ators Ġfor Ġthis Ġdata . ĠAlthough Ġthe Ġsetting Ġcan Ġbe Ġwasteful Ġof Ġlabor , Ġwe Ġcan Ġuse Ġthe Ġresulting Ġdataset Ġto Ġtest Ġseveral Ġwell - known Ġbas elines Ġsuch Ġas Ġmajority Ġvoting . </s> After Ġannotation , Ġwe Ġremove Ġsome Ġillegal Ġsentences Ġreported Ġby Ġthe Ġannot ators . ĠFinally , Ġwe Ġhave Ġ16 , 9 48 Ġsentences Ġannot ated Ġby Ġthe Ġstudents . ĠTable Ġ1 Ġshows Ġthe Ġinformation Ġof Ġannot ated Ġdata . ĠThe Ġaverage ĠKappa Ġvalue Ġamong Ġthe Ġannot ators Ġis Ġ0 . 60 33 , Ġindicating Ġthat Ġthe Ġcrowd Ġannot ators Ġhave Ġmoderate Ġagreement Ġon Ġidentifying Ġentities Ġon Ġthis Ġdata . </s> In Ġorder Ġto Ġevaluate Ġthe Ġsystem Ġperformances , Ġwe Ġcreate Ġa Ġset Ġof Ġcorpus Ġwith Ġgold Ġannotations . ĠCon crete ly , Ġwe Ġrandomly Ġselect Ġ1 , 000 Ġsentences Ġfrom Ġthe Ġfinal Ġdataset Ġand Ġlet Ġtwo Ġexperts Ġgenerate Ġthe Ġgold Ġannotations . ĠAmong Ġthem , Ġwe Ġuse Ġ300 Ġsentences Ġas Ġthe Ġdevelopment Ġset Ġand Ġthe Ġremaining Ġ700 Ġas Ġthe Ġtest Ġset . ĠThe Ġrest Ġsentences Ġwith Ġonly Ġstudent Ġannotations Ġare Ġused Ġas Ġthe Ġtraining Ġset . </s> Label ed Ġdata : ĠEC - MT Ġand ĠEC - U Q . ĠIn ĠE - commerce Ġdomain Ġ( EC ), Ġwe Ġcollect Ġraw Ġsentences Ġfrom Ġtwo Ġtypes Ġof Ġtexts : Ġone Ġis Ġtitles Ġof Ġmerchandise Ġentries Ġ( EC - MT ) Ġand Ġanother Ġis Ġuser Ġqueries Ġ( EC - U Q ). ĠThe Ġannot ators Ġlabel Ġfive Ġtypes Ġof Ġentities : ĠBrand , ĠProduct , ĠModel , ĠMaterial , Ġand ĠSpec ification . ĠThese Ġfive Ġtypes Ġof Ġentities Ġare Ġvery Ġimportant Ġfor ĠE - commerce Ġplatform , Ġfor Ġexample Ġbuilding Ġknowledge Ġgraph Ġof Ġmerch and ises . ĠFive Ġstudents Ġparticipate Ġthe Ġannotations Ġfor Ġthis Ġdomain Ġsince Ġthe Ġnumber Ġof Ġsentences Ġis Ġsmall . ĠWe Ġuse Ġthe Ġsimilar Ġstrategy Ġas ĠDL - PS Ġto Ġannot ate Ġthe Ġsentences , Ġexcept Ġthat Ġonly Ġtwo Ġannot ators Ġare Ġassigned Ġfor Ġeach Ġsentence , Ġbecause Ġwe Ġaim Ġto Ġtest Ġthe Ġsystem Ġperformances Ġunder Ġvery Ġsmall Ġdupl icated Ġannotations . </s> Finally , Ġwe Ġobtain Ġ2 , 337 Ġsentences Ġfor ĠEC - MT Ġand Ġ2 , 300 Ġfor ĠEC - U Q . ĠTable Ġ1 Ġshows Ġthe Ġinformation Ġof Ġannot ated Ġresults . ĠSimilarly , Ġwe Ġproduce Ġthe Ġdevelopment Ġand Ġtest Ġdatasets Ġfor Ġsystem Ġevaluation , Ġby Ġrandomly Ġselecting Ġ400 Ġsentences Ġand Ġletting Ġtwo Ġexperts Ġto Ġgenerate Ġthe Ġground truth Ġannotations . ĠAmong Ġthem , Ġwe Ġuse Ġ100 Ġsentences Ġas Ġthe Ġdevelopment Ġset Ġand Ġthe Ġremaining Ġ300 Ġas Ġthe Ġtest Ġset . ĠThe Ġrest Ġsentences Ġwith Ġonly Ġcrowds ourcing Ġannotations Ġare Ġused Ġas Ġthe Ġtraining Ġset . </s> Un label ed Ġdata . ĠThe Ġvector Ġrepresentations Ġof Ġcharacters Ġare Ġbasic Ġinputs Ġof Ġour Ġbaseline Ġand Ġproposed Ġmodels , Ġwhich Ġare Ġobtained Ġby Ġthe Ġlooking - up Ġtable Ġ$\ math bf Ġ{ E } ^ W $ Ġ. ĠAs Ġintroduced Ġbefore , Ġwe Ġcan Ġuse Ġpret rained Ġembed d ings Ġfrom Ġlarge - scale Ġraw Ġcorpus Ġto Ġinitialize Ġthe Ġtable . ĠIn Ġorder Ġto Ġpret rain Ġthe Ġcharacter Ġembed d ings , Ġwe Ġuse Ġone Ġlarge - scale Ġunl abel ed Ġdata Ġfrom Ġthe Ġuser - generated Ġcontent Ġin ĠInternet . ĠTotally , Ġwe Ġobtain Ġa Ġnumber Ġof Ġ5 M Ġsentences . ĠFinally , Ġwe Ġuse Ġthe Ġtool Ġword 2 vec Ġto Ġpret rain Ġthe Ġcharacter Ġembed d ings Ġbased Ġon Ġthe Ġunl abel ed Ġdataset Ġin Ġour Ġexperiments . </s> Settings </s> For Ġevaluation , Ġwe Ġuse Ġthe Ġentity - level Ġmetrics Ġof ĠPrecision Ġ( P ), ĠRecall Ġ( R ), Ġand Ġtheir ĠF 1 Ġvalue Ġin Ġour Ġexperiments , Ġtreating Ġone Ġtagged Ġentity Ġas Ġcorrect Ġonly Ġwhen Ġit Ġmatches Ġthe Ġgold Ġentity Ġexactly . </s> There Ġare Ġseveral Ġhyper - param eters Ġin Ġthe Ġbaseline ĠL ST M - CR F Ġand Ġour Ġfinal Ġmodels . ĠWe Ġset Ġthem Ġempir ically Ġby Ġthe Ġdevelopment Ġperformances . ĠCon crete ly , Ġwe Ġset Ġthe Ġdimension Ġsize Ġof Ġthe Ġcharacter Ġembed d ings Ġby Ġ100 , Ġthe Ġdimension Ġsize Ġof Ġthe ĠNE Ġlabel Ġembed d ings Ġby Ġ50 , Ġand Ġthe Ġdimension Ġsizes Ġof Ġall Ġthe Ġother Ġhidden Ġfeatures Ġby Ġ200 . </s> We Ġexploit Ġonline Ġtraining Ġwith Ġa Ġmini - batch Ġsize Ġ128 Ġto Ġlearn Ġmodel Ġparameters . ĠThe Ġmax - ep och Ġiteration Ġis Ġset Ġby Ġ200 , Ġand Ġthe Ġbest - ep och Ġmodel Ġis Ġchosen Ġaccording Ġto Ġthe Ġdevelopment Ġperformances . ĠWe Ġuse ĠR MS prop ĠB IB REF 28 Ġwith Ġa Ġlearning Ġrate Ġ$ 10 ^{ - 3 } $ Ġto Ġupdate Ġmodel Ġparameters , Ġand Ġuse Ġ$ l _ 2 $ Ġ- regular ization Ġby Ġa Ġparameter Ġ$ 10 ^{ - 5 } $ Ġ. ĠWe Ġadopt Ġthe Ġdrop out Ġtechnique Ġto Ġavoid Ġover fitting Ġby Ġa Ġdrop Ġvalue Ġof Ġ$ 0 . 2 $ Ġ. </s> Compar ison ĠSystems </s> The Ġproposed Ġapproach Ġ( hen ce forward Ġreferred Ġto Ġas ĠâĢ ľ AL C rowd âĢ Ŀ ) Ġis Ġcompared Ġwith Ġthe Ġfollowing Ġsystems : </s> CR F : ĠWe Ġuse Ġthe ĠCr f su ite Ġtool Ġto Ġtrain Ġa Ġmodel Ġon Ġthe Ġcrowds ourcing Ġlabeled Ġdata . ĠAs Ġfor Ġthe Ġfeature Ġsettings , Ġwe Ġuse Ġthe Ġsupervised Ġversion Ġof ĠB IB REF 0 Ġz hao 2008 un super vised . </s> CR F - VT : ĠWe Ġuse Ġthe Ġsame Ġsettings Ġof Ġthe ĠCR F Ġsystem , Ġexcept Ġthat Ġthe Ġtraining Ġdata Ġis Ġthe Ġvoted Ġversion , Ġwhose Ġground truth s Ġare Ġproduced Ġby Ġmajority Ġvoting Ġat Ġthe Ġcharacter Ġlevel Ġfor Ġeach Ġannot ated Ġsentence . </s> CR F - MA : ĠThe ĠCR F Ġmodel Ġproposed Ġby ĠB IB REF 3 Ġrod rig ues 2014 sequence , Ġwhich Ġuses Ġa Ġprior Ġdistribut ation Ġto Ġmodel Ġmultiple Ġcrowds ourcing Ġannot ators . ĠWe Ġuse Ġthe Ġsource Ġcode Ġprovided Ġby Ġthe Ġauthors . </s> L ST M - CR F : ĠOur Ġbaseline Ġsystem Ġtrained Ġon Ġthe Ġcrowds ourcing Ġlabeled Ġdata . </s> L ST M - CR F - VT : ĠOur Ġbaseline Ġsystem Ġtrained Ġon Ġthe Ġvoted Ġcorpus , Ġwhich Ġis Ġthe Ġsame Ġas ĠCR F - VT . </s> L ST M - C rowd : ĠThe ĠL ST M - CR F Ġmodel Ġwith Ġcrowd Ġannotation Ġlearning Ġproposed Ġby ĠB IB REF 4 Ġn guyen 2017 agg reg ating . ĠWe Ġuse Ġthe Ġsource Ġcode Ġprovided Ġby Ġthe Ġauthors . </s> The Ġfirst Ġthree Ġsystems Ġare Ġbased Ġon Ġthe ĠCR F Ġmodel Ġusing Ġtraditional Ġhand crafted Ġfeatures , Ġand Ġthe Ġlast Ġthree Ġsystems Ġare Ġbased Ġon Ġthe Ġneural ĠL ST M - CR F Ġmodel . ĠAmong Ġthem , ĠCR F - MA , ĠL ST M - C rowd Ġand Ġour Ġsystem Ġwith Ġadvers arial Ġlearning Ġ( AL C rowd ) Ġare Ġbased Ġon Ġcrowd Ġannotation Ġlearning Ġthat Ġdirectly Ġtrains Ġthe Ġmodel Ġon Ġthe Ġcrowd - annot ations . ĠFive Ġsystems , Ġincluding ĠCR F , ĠCR F - MA , ĠL ST M - CR F , ĠL ST M - C rowd , Ġand ĠA LC rowd , Ġare Ġtrained Ġon Ġthe Ġoriginal Ġversion Ġof Ġlabeled Ġdata , Ġwhile ĠCR F - VT Ġand ĠL ST M - CR F - VT Ġare Ġtrained Ġon Ġthe Ġvoted Ġversion . ĠSince ĠCR F - VT , ĠCR F - MA Ġand ĠL ST M - CR F - VT Ġall Ġrequire Ġground - truth Ġanswers Ġfor Ġeach Ġtraining Ġsentence , Ġwhich Ġare Ġdifficult Ġto Ġbe Ġproduced Ġwith Ġonly Ġtwo Ġannotations , Ġwe Ġdo Ġnot Ġapply Ġthe Ġthree Ġmodels Ġon Ġthe Ġtwo ĠEC Ġdatasets . </s> Main ĠResults </s> In Ġthis Ġsection , Ġwe Ġshow Ġthe Ġmodel Ġperformances Ġof Ġour Ġproposed Ġcrowds ourcing Ġlearning Ġsystem Ġ( AL C rowd ), Ġand Ġmeanwhile Ġcompare Ġit Ġwith Ġthe Ġother Ġsystems Ġmentioned Ġabove . ĠTable Ġ2 Ġshows Ġthe Ġexperimental Ġresults Ġon Ġthe ĠDL - PS Ġdatasets Ġand ĠTable Ġ3 Ġshows Ġthe Ġexperiment Ġresults Ġon Ġthe ĠEC - MT Ġand ĠEC - U Q Ġdatasets , Ġrespectively . </s> The Ġresults Ġof ĠCR F Ġand ĠL ST M - CR F Ġmean Ġthat Ġthe Ġcrowd Ġannotation Ġis Ġan Ġalternative Ġsolution Ġwith Ġlow Ġcost Ġfor Ġlabeling Ġdata Ġthat Ġcould Ġbe Ġused Ġfor Ġtraining Ġa ĠN ER Ġsystem Ġeven Ġthere Ġare Ġsome Ġinconsistencies . ĠCompared Ġwith ĠCR F , ĠL ST M - CR F Ġachieves Ġmuch Ġbetter Ġperformances Ġon Ġall Ġthe Ġthree Ġdata , Ġshowing Ġ+ 6 . 12 ĠF 1 Ġimprovement Ġon ĠDL - PS , Ġ+ 4 . 51 Ġon ĠEC - MT , Ġand Ġ+ 9 . 19 Ġon ĠEC - U Q . ĠThis Ġindicates Ġthat ĠL ST M - CR F Ġis Ġa Ġvery Ġstrong Ġbaseline Ġsystem , Ġdemonstrating Ġthe Ġeffectiveness Ġof Ġneural Ġnetwork . </s> Interestingly , Ġwhen Ġcompared Ġwith ĠCR F Ġand ĠL ST M - CR F , ĠCR F - VT Ġand ĠL ST M - CR F - VT Ġtrained Ġon Ġthe Ġvoted Ġversion Ġperform Ġworse Ġin Ġthe ĠDL - PS Ġdataset . ĠThis Ġtrend Ġis Ġalso Ġmentioned Ġin ĠB IB REF 4 Ġn guyen 2017 agg reg ating . ĠThis Ġfact Ġshows Ġthat Ġthe Ġmajority Ġvoting Ġmethod Ġmight Ġbe Ġunsu itable Ġfor Ġour Ġtask . ĠThere Ġare Ġtwo Ġpossible Ġreasons Ġaccounting Ġfor Ġthe Ġobservation . ĠOn Ġthe Ġone Ġhand , Ġsimple Ġcharacter - level Ġvoting Ġbased Ġon Ġthree Ġannotations Ġfor Ġeach Ġsentence Ġmay Ġbe Ġstill Ġnot Ġenough . ĠIn Ġthe ĠDL - PS Ġdataset , Ġeven Ġwith Ġonly Ġtwo Ġpred efined Ġentity Ġtypes , Ġone Ġcharacter Ġcan Ġhave Ġnine ĠNE Ġlabels . ĠThus Ġthe Ġmajority - v oting Ġmay Ġbe Ġincapable Ġof Ġhandling Ġsome Ġcases . ĠWhile Ġthe Ġcost Ġby Ġadding Ġmore Ġannotations Ġfor Ġeach Ġsentence Ġwould Ġbe Ġgreatly Ġincreased . ĠOn Ġthe Ġother Ġhand , Ġthe Ġlost Ġinformation Ġproduced Ġby Ġmajority - v oting Ġmay Ġbe Ġimportant , Ġat Ġleast Ġthe Ġambiguous Ġannotations Ġdenote Ġthat Ġthe Ġinput Ġsentence Ġis Ġdifficult Ġfor ĠN ER . ĠThe Ġnormal ĠCR F Ġand ĠL ST M - CR F Ġmodels Ġwithout Ġdiscard Ġany Ġannotations Ġcan Ġdifferentiate Ġthese Ġdifficult Ġcontexts Ġthrough Ġlearning . </s> Three Ġcrowd - ann otation Ġlearning Ġsystems Ġprovide Ġbetter Ġperformances Ġthan Ġtheir Ġcounterpart Ġsystems , Ġ( CR F - MA ĠVS ĠCR F ) Ġand Ġ( L ST M - C rowd / AL C rowd ĠVS ĠL ST M - CR F ). ĠCompared Ġwith Ġthe Ġstrong Ġbaseline ĠL ST M - CR F , ĠA LC rowd Ġshows Ġits Ġadvantage Ġwith Ġ+ 1 . 08 ĠF 1 Ġimprovements Ġon ĠDL - PS , Ġ+ 1 . 24 Ġon ĠEC - MT , Ġand Ġ+ 2 . 38 Ġon ĠEC - U Q , Ġrespectively . ĠThis Ġindicates Ġthat Ġadding Ġthe Ġcrowd - ann otation Ġlearning Ġis Ġquite Ġuseful Ġfor Ġbuilding ĠN ER Ġsystems . ĠIn Ġaddition , ĠA LC rowd Ġalso Ġoutper forms ĠL ST M - C rowd Ġon Ġall Ġthe Ġdatasets Ġconsistently , Ġdemonstrating Ġthe Ġhigh Ġeffectiveness Ġof ĠA LC rowd Ġin Ġextracting Ġworker Ġindependent Ġfeatures . ĠAmong Ġall Ġthe Ġsystems , ĠA LC rowd Ġperforms Ġthe Ġbest , Ġand Ġsignificantly Ġbetter Ġthan Ġall Ġthe Ġother Ġmodels Ġ( the Ġp - value Ġis Ġbelow Ġ$ 10 ^{ - 5 } $ Ġby Ġusing Ġt - test ). ĠThe Ġresults Ġindicate Ġthat Ġwith Ġthe Ġhelp Ġof Ġadvers arial Ġtraining , Ġour Ġsystem Ġcan Ġlearn Ġa Ġbetter Ġfeature Ġrepresentation Ġfrom Ġcrowd Ġannotation . </s> Discussion </s> Imp act Ġof ĠCharacter ĠEmb edd ings . ĠFirst , Ġwe Ġinvestigate Ġthe Ġeffect Ġof Ġthe Ġpret rained Ġcharacter Ġembed d ings Ġin Ġour Ġproposed Ġcrowds ourcing Ġlearning Ġmodel . ĠThe Ġcomparison Ġresults Ġare Ġshown Ġin ĠFigure Ġ2 Ġ, Ġwhere ĠRandom Ġrefers Ġto Ġthe Ġrandom Ġinitialized Ġcharacter Ġembed d ings , Ġand ĠPret rained Ġrefers Ġto Ġthe Ġembed d ings Ġpret rained Ġon Ġthe Ġunl abel ed Ġdata . ĠAccording Ġto Ġthe Ġresults , Ġwe Ġfind Ġthat Ġour Ġmodel Ġwith Ġthe Ġpret rained Ġembed d ings Ġsignificantly Ġoutper forms Ġthat Ġusing Ġthe Ġrandom Ġembed d ings , Ġdemonstrating Ġthat Ġthe Ġpret rained Ġembed d ings Ġsuccessfully Ġprovide Ġuseful Ġinformation . </s> Case ĠStudies . ĠSecond , Ġwe Ġpresent Ġseveral Ġcase Ġstudies Ġin Ġorder Ġto Ġstudy Ġthe Ġdifferences Ġbetween Ġour Ġbaseline Ġand Ġthe Ġworker Ġadvers arial Ġmodels . ĠWe Ġconduct Ġa Ġclosed Ġtest Ġon Ġthe Ġtraining Ġset , Ġthe Ġresults Ġof Ġwhich Ġcan Ġbe Ġregarded Ġas Ġmodifications Ġof Ġthe Ġtraining Ġcorpus , Ġsince Ġthere Ġexist Ġinconsistent Ġannotations Ġfor Ġeach Ġtraining Ġsentence Ġamong Ġthe Ġdifferent Ġworkers . ĠFigure Ġ3 Ġshows Ġthe Ġtwo Ġexamples Ġfrom Ġthe ĠDL - PS Ġdataset , Ġwhich Ġcompares Ġthe Ġoutputs Ġof Ġthe Ġbaseline Ġand Ġour Ġfinal Ġmodels , Ġas Ġwell Ġas Ġthe Ġmajority - v oting Ġstrategy . </s> In Ġthe Ġfirst Ġcase , Ġnone Ġof Ġthe Ġannotations Ġget Ġthe Ġcorrect ĠN ER Ġresult , Ġbut Ġour Ġproposed Ġmodel Ġcan Ġcapture Ġit . ĠThe Ġresult Ġof ĠL ST M - CR F Ġis Ġthe Ġsame Ġas Ġmajority - v oting . ĠIn Ġthe Ġsecond Ġexample , Ġthe Ġoutput Ġof Ġmajority - v oting Ġis Ġthe Ġworst , Ġwhich Ġcan Ġaccount Ġfor Ġthe Ġreason Ġwhy Ġthe Ġsame Ġmodel Ġtrained Ġon Ġthe Ġvoted Ġcorpus Ġperforms Ġso Ġbadly , Ġas Ġshown Ġin ĠTable Ġ2 Ġ. ĠThe Ġmodel Ġof ĠL ST M - CR F Ġfails Ġto Ġrecognize Ġthe Ġnamed Ġentity ĠâĢ ľ X ie x ie âĢ Ŀ Ġbecause Ġof Ġnot Ġtrusting Ġthe Ġsecond Ġannotation , Ġtreating Ġit Ġas Ġone Ġnoise Ġannotation . ĠOur Ġproposed Ġmodel Ġis Ġable Ġto Ġrecognize Ġit , Ġbecause Ġof Ġits Ġability Ġof Ġextracting Ġworker Ġindependent Ġfeatures . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġan Ġapproach Ġto Ġperforming Ġcrowd Ġannotation Ġlearning Ġbased Ġon Ġthe Ġidea Ġof Ġadvers arial Ġtraining Ġfor ĠChinese ĠNamed ĠEntity ĠRecogn ition Ġ( NER ). ĠIn Ġour Ġapproach , Ġwe Ġuse Ġa Ġcommon Ġand Ġprivate ĠBi - L ST Ms Ġfor Ġrepresenting Ġannot ator - generic Ġand Ġ- specific Ġinformation , Ġand Ġlearn Ġa Ġlabel ĠBi - L ST M Ġfrom Ġthe Ġcrowd - annot ated ĠNE Ġlabel Ġsequences . ĠFinally , Ġthe Ġproposed Ġapproach Ġadop ts Ġa ĠL ST M - CR F Ġmodel Ġto Ġperform Ġtagging . ĠIn Ġour Ġexperiments , Ġwe Ġcreate Ġtwo Ġdata Ġsets Ġfor ĠChinese ĠN ER Ġtasks Ġin Ġthe Ġdialog Ġand Ġe - commerce Ġdomains . ĠThe Ġexperimental Ġresults Ġshow Ġthat Ġthe Ġproposed Ġapproach Ġoutper forms Ġstrong Ġbaseline Ġsystems . </s> ĠA cknowled gments </s> This Ġwork Ġis Ġsupported Ġby Ġthe ĠNational ĠNatural ĠScience ĠFoundation Ġof ĠChina Ġ( Grant ĠNo . Ġ6 157 23 38 , Ġ6 15 25 205 , Ġand Ġ6 16 02 160 ). ĠThis Ġwork Ġis Ġalso Ġpartially Ġsupported Ġby Ġthe Ġjoint Ġresearch Ġproject Ġof ĠAlibaba Ġand ĠSo och ow ĠUniversity . ĠWen li ang Ġis Ġalso Ġpartially Ġsupported Ġby ĠCollabor ative ĠInnovation ĠCenter Ġof ĠNovel ĠSoftware ĠTechnology Ġand ĠIndustrial ization .
Introduction There has been significant progress on Named Entity Recognition (NER) in recent years using models based on machine learning algorithms BIBREF0 , BIBREF1 , BIBREF2 . As with other Natural Language Processing (NLP) tasks, building NER systems typically requires a massive amount of labeled training data which are annotated by experts. In real applications, we often need to consider new types of entities in new domains where we do not have existing annotated data. For such new types of entities, however, it is very hard to find experts to annotate the data within short time limits and hiring experts is costly and non-scalable, both in terms of time and money. In order to quickly obtain new training data, we can use crowdsourcing as one alternative way at lower cost in a short time. But as an exchange, crowd annotations from non-experts may be of lower quality than those from experts. It is one biggest challenge to build a powerful NER system on such a low quality annotated data. Although we can obtain high quality annotations for each input sentence by majority voting, it can be a waste of human labors to achieve such a goal, especially for some ambiguous sentences which may require a number of annotations to reach an agreement. Thus majority work directly build models on crowd annotations, trying to model the differences among annotators, for example, some of the annotators may be more trustful BIBREF3 , BIBREF4 . Here we focus mainly on the Chinese NER, which is more difficult than NER for other languages such as English for the lack of morphological variations such as capitalization and in particular the uncertainty in word segmentation. The Chinese NE taggers trained on news domain often perform poor in other domains. Although we can alleviate the problem by using character-level tagging to resolve the problem of poor word segmentation performances BIBREF5 , still there exists a large gap when the target domain changes, especially for the texts of social media. Thus, in order to get a good tagger for new domains and also for the conditions of new entity types, we require large amounts of labeled data. Therefore, crowdsourcing is a reasonable solution for these situations. In this paper, we propose an approach to training a Chinese NER system on the crowd-annotated data. Our goal is to extract additional annotator independent features by adversarial training, alleviating the annotation noises of non-experts. The idea of adversarial training in neural networks has been used successfully in several NLP tasks, such as cross-lingual POS tagging BIBREF6 and cross-domain POS tagging BIBREF7 . They use it to reduce the negative influences of the input divergences among different domains or languages, while we use adversarial training to reduce the negative influences brought by different crowd annotators. To our best knowledge, we are the first to apply adversarial training for crowd annotation learning. In the learning framework, we perform adversarial training between the basic NER and an additional worker discriminator. We have a common Bi-LSTM for representing annotator-generic information and a private Bi-LSTM for representing annotator-specific information. We build another label Bi-LSTM by the crowd-annotated NE label sequence which reflects the mind of the crowd annotators who learn entity definitions by reading the annotation guidebook. The common and private Bi-LSTMs are used for NER, while the common and label Bi-LSTMs are used as inputs for the worker discriminator. The parameters of the common Bi-LSTM are learned by adversarial training, maximizing the worker discriminator loss and meanwhile minimizing the NER loss. Thus the resulting features of the common Bi-LSTM are worker invariant and NER sensitive. For evaluation, we create two Chinese NER datasets in two domains: dialog and e-commerce. We require the crowd annotators to label the types of entities, including person, song, brand, product, and so on. Identifying these entities is useful for chatbot and e-commerce platforms BIBREF8 . Then we conduct experiments on the newly created datasets to verify the effectiveness of the proposed adversarial neural network model. The results show that our system outperforms very strong baseline systems. In summary, we make the following contributions: Related Work Our work is related to three lines of research: Sequence labeling, Adversarial training, and Crowdsourcing. Sequence labeling. NER is widely treated as a sequence labeling problem, by assigning a unique label over each sentential word BIBREF9 . Early studies on sequence labeling often use the models of HMM, MEMM, and CRF BIBREF10 based on manually-crafted discrete features, which can suffer the feature sparsity problem and require heavy feature engineering. Recently, neural network models have been successfully applied to sequence labeling BIBREF1 , BIBREF11 , BIBREF2 . Among these work, the model which uses Bi-LSTM for feature extraction and CRF for decoding has achieved state-of-the-art performances BIBREF11 , BIBREF2 , which is exploited as the baseline model in our work. Adversarial Training. Adversarial Networks have achieved great success in computer vision such as image generation BIBREF12 , BIBREF13 . In the NLP community, the method is mainly exploited under the settings of domain adaption BIBREF14 , BIBREF7 , cross-lingual BIBREF15 , BIBREF6 and multi-task learning BIBREF16 , BIBREF17 . All these settings involve the feature divergences between the training and test examples, and aim to learn invariant features across the divergences by an additional adversarial discriminator, such as domain discriminator. Our work is similar to these work but is applies on crowdsourcing learning, aiming to find invariant features among different crowdsourcing workers. Crowdsourcing. Most NLP tasks require a massive amount of labeled training data which are annotated by experts. However, hiring experts is costly and non-scalable, both in terms of time and money. Instead, crowdsourcing is another solution to obtain labeled data at a lower cost but with relative lower quality than those from experts. BIBREF18 snow2008cheap collected labeled results for several NLP tasks from Amazon Mechanical Turk and demonstrated that non-experts annotations were quite useful for training new systems. In recent years, a series of work have focused on how to use crowdsourcing data efficiently in tasks such as classification BIBREF19 , BIBREF20 , and compare quality of crowd and expert labels BIBREF21 . In sequence labeling tasks, BIBREF22 dredze2009sequence viewed this task as a multi-label problem while BIBREF3 rodrigues2014sequence took workers identities into account by assuming that each sentential word was tagged correctly by one of the crowdsourcing workers and proposed a CRF-based model with multiple annotators. BIBREF4 nguyen2017aggregating introduced a crowd representation in which the crowd vectors were added into the LSTM-CRF model at train time, but ignored them at test time. In this paper, we apply adversarial training on crowd annotations on Chinese NER in new domains, and achieve better performances than previous studies on crowdsourcing learning. Baseline: LSTM-CRF We use a neural CRF model as the baseline system BIBREF9 , treating NER as a sequence labeling problem over Chinese characters, which has achieved state-of-the-art performances BIBREF5 . To this end, we explore the BIEO schema to convert NER into sequence labeling, following BIBREF2 lample-EtAl:2016:N16-1, where sentential character is assigned with one unique tag. Concretely, we tag the non-entity character by label “O”, the beginning character of an entity by “B-XX”, the ending character of an entity by “E-XX” and the other character of an entity by “I-XX”, where “XX” denotes the entity type. We build high-level neural features from the input character sequence by a bi-directional LSTM BIBREF2 . The resulting features are combined and then are fed into an output CRF layer for decoding. In summary, the baseline model has three main components. First, we make vector representations for sentential characters $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ , transforming the discrete inputs into low-dimensional neural inputs. Second, feature extraction is performed to obtain high-level features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ , by using a bi-directional LSTM (Bi-LSTM) structure together with a linear transformation over $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ . Third, we apply a CRF tagging module over $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ , obtaining the final output NE labels. The overall framework of the baseline model is shown by the right part of Figure 1 . Vector Representation of Characters To represent Chinese characters, we simply exploit a neural embedding layer to map discrete characters into the low-dimensional vector representations. The goal is achieved by a looking-up table $\mathbf {E}^W$ , which is a model parameter and will be fine-tuned during training. The looking-up table can be initialized either by random or by using a pretrained embeddings from large scale raw corpus. For a given Chinese character sequence $c_1c_2\cdots c_n$ , we obtain the vector representation of each sentential character by: $ \mathbf {x}_t = \text{look-up}(c_t, \mathbf {E}^W), \text{~~~} t \in [1, n]$ . Feature Extraction Based on the vector sequence $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ , we extract higher-level features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ by using a bidirectional LSTM module and a simple feed-forward neural layer, which are then used for CRF tagging at the next step. LSTM is a type of recurrent neural network (RNN), which is designed for solving the exploding and diminishing gradients of basic RNNs BIBREF23 . It has been widely used in a number of NLP tasks, including POS-tagging BIBREF11 , BIBREF24 , parsing BIBREF25 and machine translation BIBREF26 , because of its strong capabilities of modeling natural language sentences. By traversing $\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n$ by order and reversely, we obtain the output features $\mathbf {h}_1^{\text{private}}\mathbf {h}_2^{\text{private}}\cdots \mathbf {h}_n^{\text{private}}$ of the bi-LSTM, where $\mathbf {h}_t^{\text{private}} = \overrightarrow{\mathbf {h}}_t \oplus \overleftarrow{\mathbf {h}}_t $ . Here we refer this Bi-LSTM as private in order to differentiate it with the common Bi-LSTM over the same character inputs which will be introduced in the next section. Further we make an integration of the output vectors of bi-directional LSTM by a linear feed-forward neural layer, resulting in the features $\mathbf {h}_1^{\text{ner}}\mathbf {h}_2^{\text{ner}}\cdots \mathbf {h}_n^{\text{ner}}$ by equation: $$\mathbf {h}_t^{\text{ner}} = \mathbf {W} \mathbf {h}_t^{\text{private}} + \mathbf {b},$$   (Eq. 6) where $\mathbf {W}$ and $\mathbf {b}$ are both model parameters. CRF Tagging Finally we feed the resulting features $\mathbf {h}_t^{\text{ner}}, t\in [1, n]$ into a CRF layer directly for NER decoding. CRF tagging is one globally normalized model, aiming to find the best output sequence considering the dependencies between successive labels. In the sequence labeling setting for NER, the output label of one position has a strong dependency on the label of the previous position. For example, the label before “I-XX” must be either “B-XX” or “I-XX”, where “XX” should be exactly the same. CRF involves two parts for prediction. First we should compute the scores for each label based $\mathbf {h}_t^{\text{ner}}$ , resulting in $\mathbf {o}_t^{\text{ner}}$ , whose dimension is the number of output labels. The other part is a transition matrix $\mathbf {T}$ which defines the scores of two successive labels. $\mathbf {T}$ is also a model parameter. Based on $\mathbf {o}_t^{\text{ner}}$ and $\mathbf {T}$ , we use the Viterbi algorithm to find the best-scoring label sequence. We can formalize the CRF tagging process as follows: $$\begin{split} & \mathbf {o}_t^{\text{ner}} = \mathbf {W}^{\text{ner}} \mathbf {h}_t^{\text{ner}}, \text{~~~~} t \in [1,n] \\ & \text{score}(\mathbf {X}, \mathbf {y}) = \sum _{t = 1}^{n}(\mathbf {o}_{t,y_t} + T_{y_{t-1},y_t}) \\ & \mathbf {y}^{\text{ner}} = \mathop {arg~max}_{\mathbf {y}}\big (\text{score}(\mathbf {X}, \mathbf {y}))\big ), \\ \end{split}$$   (Eq. 8) where $\text{score}(\cdot )$ is the scoring function for a given output label sequence $\mathbf {y} = y_1y_2 \cdots y_n$ based on input $\mathbf {X}$ , $\mathbf {y}^{\text{ner}}$ is the resulting label sequence, $\mathbf {W}^{\text{ner}}$ is a model parameter. Training To train model parameters, we exploit a negative log-likelihood objective as the loss function. We apply softmax over all candidate output label sequences, thus the probability of the crowd-annotated label sequence is computed by: $$p(\mathbf {\bar{y}}|\mathbf {X}) = \frac{\exp \big (\text{score}(\mathbf {X}, \mathbf {\bar{y}})\big )}{\sum _{\mathbf {y} \in \mathbf {Y}_{\mathbf {X}}} \exp \big (\text{score}(\mathbf {X}, \mathbf {y})\big )},$$   (Eq. 10) where $\mathbf {\bar{y}}$ is the crowd-annotated label sequences and $\mathbf {Y}_{\mathbf {X}}$ is all candidate label sequence of input $\mathbf {X}$ . Based on the above formula, the loss function of our baseline model is: $$\text{loss}(\Theta , \mathbf {X}, \mathbf {\bar{y}}) = -\log p(\mathbf {\bar{y}}|\mathbf {X}),$$   (Eq. 11) where $\Theta $ is the set of all model parameters. We use standard back-propagation method to minimize the loss function of the baseline CRF model. Worker Adversarial Adversarial learning has been an effective mechanism to resolve the problem of the input features between the training and test examples having large divergences BIBREF27 , BIBREF13 . It has been successfully applied on domain adaption BIBREF7 , cross-lingual learning BIBREF15 and multi-task learning BIBREF17 . All settings involve feature shifting between the training and testing. In this paper, our setting is different. We are using the annotations from non-experts, which are noise and can influence the final performances if they are not properly processed. Directly learning based on the resulting corpus may adapt the neural feature extraction into the biased annotations. In this work, we assume that individual workers have their own guidelines in mind after short training. For example, a perfect worker can annotate highly consistently with an expert, while common crowdsourcing workers may be confused and have different understandings on certain contexts. Based on the assumption, we make an adaption for the original adversarial neural network to our setting. Our adaption is very simple. Briefly speaking, the original adversarial learning adds an additional discriminator to classify the type of source inputs, for example, the domain category in the domain adaption setting, while we add a discriminator to classify the annotation workers. Solely the features from the input sentence is not enough for worker classification. The annotation result of the worker is also required. Thus the inputs of our discriminator are different. Here we exploit both the source sentences and the crowd-annotated NE labels as basic inputs for the worker discrimination. In the following, we describe the proposed adversarial learning module, including both the submodels and the training method. As shown by the left part of Figure 1 , the submodel consists of four parts: (1) a common Bi-LSTM over input characters; (2) an additional Bi-LSTM to encode crowd-annotated NE label sequence; (3) a convolutional neural network (CNN) to extract features for worker discriminator; (4) output and prediction. Common Bi-LSTM over Characters To build the adversarial part, first we create a new bi-directional LSTM, named by the common Bi-LSTM: $$\mathbf {h}_1^{\text{\tiny common}} \mathbf {h}_2^{\text{\tiny common}} \cdots \mathbf {h}_n^{\text{\tiny common}} = \text{Bi-LSTM}(\mathbf {x}_1\mathbf {x}_2\cdots \mathbf {x}_n).$$   (Eq. 13) As shown in Figure 1 , this Bi-LSTM is constructed over the same input character representations of the private Bi-LSTM, in order to extract worker independent features. The resulting features of the common Bi-LSTM are used for both NER and the worker discriminator, different with the features of private Bi-LSTM which are used for NER only. As shown in Figure 1 , we concatenate the outputs of the common and private Bi-LSTMs together, and then feed the results into the feed-forward combination layer of the NER part. Thus Formula 6 can be rewritten as: $$\mathbf {h}_t^{\text{ner}} = \mathbf {W} (\mathbf {h}_t^{\text{common}} \oplus \mathbf {h}_t^{\text{private}}) + \mathbf {b},$$   (Eq. 14) where $\mathbf {W}$ is wider than the original combination because the newly-added $\mathbf {h}_t^{\text{common}}$ . Noticeably, although the resulting common features are used for the worker discriminator, they actually have no capability to distinguish the workers. Because this part is exploited to maximize the loss of the worker discriminator, it will be interpreted in the later training subsection. These features are invariant among different workers, thus they can have less noises for NER. This is the goal of adversarial learning, and we hope the NER being able to find useful features from these worker independent features. Additional Bi-LSTM over Annotated NER Labels In order to incorporate the annotated NE labels to predict the exact worker, we build another bi-directional LSTM (named by label Bi-LSTM) based on the crowd-annotated NE label sequence. This Bi-LSTM is used for worker discriminator only. During the decoding of the testing phase, we will never have this Bi-LSTM, because the worker discriminator is no longer required. Assuming the crowd-annotated NE label sequence annotated by one worker is $\mathbf {\bar{y}} = \bar{y}_1\bar{y}_2 \cdots \bar{y}_n$ , we exploit a looking-up table $\mathbf {E}^{L}$ to obtain the corresponding sequence of their vector representations $\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n$ , similar to the method that maps characters into their neural representations. Concretely, for one NE label $\bar{y}_t$ ( $t \in [1, n]$ ), we obtain its neural vector by: $\mathbf {x^{\prime }}_t = \text{look-up}(\bar{y}_t, \mathbf {E}^L)$ . Next step we apply bi-directional LSTM over the sequence $\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n$ , which can be formalized as: $$\mathbf {h}_1^{\text{label}} \mathbf {h}_2^{\text{label}} \cdots \mathbf {h}_n^{\text{label}} = \text{Bi-LSTM}(\mathbf {x^{\prime }}_1\mathbf {x^{\prime }}_2\cdots \mathbf {x^{\prime }}_n).$$   (Eq. 16) The resulting feature sequence is concatenated with the outputs of the common Bi-LSTM, and further be used for worker classification. CNN Following, we add a convolutional neural network (CNN) module based on the concatenated outputs of the common Bi-LSTM and the label Bi-LSTM, to produce the final features for worker discriminator. A convolutional operator with window size 5 is used, and then max pooling strategy is applied over the convolution sequence to obtain the final fixed-dimensional feature vector. The whole process can be described by the following equations: $$\begin{split} &\mathbf {h}_t^{\text{worker}} = \mathbf {h}_t^{\text{common}} \oplus \mathbf {h}_t^{\text{label}} \\ &\mathbf {\tilde{h}}_t^{\text{worker}} = \tanh (\mathbf {W}^{\text{cnn}}[\mathbf {h}_{t-2}^{\text{worker}}, \mathbf {h}_{t-1}^{\text{worker}}, \cdots , \mathbf {h}_{t+2}^{\text{worker}}]) \\ &\mathbf {h}^{\text{worker}} = \text{max-pooling}(\mathbf {\tilde{h}}_1^{\text{worker}}\mathbf {\tilde{h}}_2^{\text{worker}} \cdots \mathbf {\tilde{h}}_n^{\text{worker}}) \\ \end{split}$$   (Eq. 18) where $t \in [1,n]$ and $\mathbf {W}^{\text{cnn}}$ is one model parameter. We exploit zero vector to paddle the out-of-index vectors. Output and Prediction After obtaining the final feature vector for the worker discriminator, we use it to compute the output vector, which scores all the annotation workers. The score function is defined by: $$\mathbf {o}^{\text{worker}} = \mathbf {W}^{\text{worker}} \mathbf {h}^{\text{worker}},$$   (Eq. 20) where $\mathbf {W}^{\text{worker}}$ is one model parameter and the output dimension equals the number of total non-expert annotators. The prediction is to find the worker which is responsible for this annotation. Adversarial Training The training objective with adversarial neural network is different from the baseline model, as it includes the extra worker discriminator. Thus the new objective includes two parts, one being the negative log-likelihood from NER which is the same as the baseline, and the other being the negative the log-likelihood from the worker discriminator. In order to obtain the negative log-likelihood of the worker discriminator, we use softmax to compute the probability of the actual worker $\bar{z}$ as well, which is defined by: $$p(\bar{z}|\mathbf {X}, \mathbf {\bar{y}}) = \frac{\exp (\mathbf {o}^{\text{worker}}_{\bar{z}})}{\sum _{z} \exp (\mathbf {o}^{\text{worker}}_z)},$$   (Eq. 22) where $z$ should enumerate all workers. Based on the above definition of probability, our new objective is defined as follows: $$\begin{split} \text{R}(\Theta , \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) &= \text{loss}(\Theta , \mathbf {X}, \mathbf {\bar{y}}) - \text{loss}(\Theta , \Theta ^{\prime }, \mathbf {X}) \\ \text{~~~~~~} &= -\log p(\mathbf {\bar{y}}|\mathbf {X}) + \log p(\bar{z}|\mathbf {X}, \mathbf {\bar{y}}), \end{split}$$   (Eq. 23) where $\Theta $ is the set of all model parameters related to NER, and $\Theta ^{\prime }$ is the set of the remaining parameters which are only related to the worker discriminator, $\mathbf {X}$ , $\mathbf {\bar{y}}$ and $\bar{z}$ are the input sentence, the crowd-annotated NE labels and the corresponding annotator for this annotation, respectively. It is worth noting that the parameters of the common Bi-LSTM are included in the set of $\Theta $ by definition. In particular, our goal is not to simply minimize the new objective. Actually, we aim for a saddle point, finding the parameters $\Theta $ and $\Theta ^{\prime }$ satisfying the following conditions: $$\begin{split} \hat{\Theta } &= \mathop {arg~min}_{\Theta }\text{R}(\Theta , \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) \\ \hat{\Theta }^{\prime } &= \mathop {arg~max}_{\Theta ^{\prime }}\text{R}(\hat{\Theta }, \Theta ^{\prime }, \mathbf {X}, \mathbf {\bar{y}}, \bar{z}) \\ \end{split}$$   (Eq. 24) where the first equation aims to find one $\Theta $ that minimizes our new objective $\text{R}(\cdot )$ , and the second equation aims to find one $\Theta ^{\prime }$ maximizing the same objective. Intuitively, the first equation of Formula 24 tries to minimize the NER loss, but at the same time maximize the worker discriminator loss by the shared parameters of the common Bi-LSTM. Thus the resulting features of common Bi-LSTM actually attempt to hurt the worker discriminator, which makes these features worker independent since they are unable to distinguish different workers. The second equation tries to minimize the worker discriminator loss by its own parameter $\Theta ^{\prime }$ . We use the standard back-propagation method to train the model parameters, the same as the baseline model. In order to incorporate the term of the argmax part of Formula 24 , we follow the previous work of adversarial training BIBREF13 , BIBREF15 , BIBREF17 , by introducing a gradient reverse layer between the common Bi-LSTM and the CNN module, whose forward does nothing but the backward simply negates the gradients. Data Sets With the purpose of obtaining evaluation datasets from crowd annotators, we collect the sentences from two domains: Dialog and E-commerce domain. We hire undergraduate students to annotate the sentences. They are required to identify the predefined types of entities in the sentences. Together with the guideline document, the annotators are educated some tips in fifteen minutes and also provided with 20 exemplifying sentences. Labeled Data: DL-PS. In Dialog domain (DL), we collect raw sentences from a chatbot application. And then we randomly select 20K sentences as our pool and hire 43 students to annotate the sentences. We ask the annotators to label two types of entities: Person-Name and Song-Name. The annotators label the sentences independently. In particular, each sentence is assigned to three annotators for this data. Although the setting can be wasteful of labor, we can use the resulting dataset to test several well-known baselines such as majority voting. After annotation, we remove some illegal sentences reported by the annotators. Finally, we have 16,948 sentences annotated by the students. Table 1 shows the information of annotated data. The average Kappa value among the annotators is 0.6033, indicating that the crowd annotators have moderate agreement on identifying entities on this data. In order to evaluate the system performances, we create a set of corpus with gold annotations. Concretely, we randomly select 1,000 sentences from the final dataset and let two experts generate the gold annotations. Among them, we use 300 sentences as the development set and the remaining 700 as the test set. The rest sentences with only student annotations are used as the training set. Labeled data: EC-MT and EC-UQ. In E-commerce domain (EC), we collect raw sentences from two types of texts: one is titles of merchandise entries (EC-MT) and another is user queries (EC-UQ). The annotators label five types of entities: Brand, Product, Model, Material, and Specification. These five types of entities are very important for E-commerce platform, for example building knowledge graph of merchandises. Five students participate the annotations for this domain since the number of sentences is small. We use the similar strategy as DL-PS to annotate the sentences, except that only two annotators are assigned for each sentence, because we aim to test the system performances under very small duplicated annotations. Finally, we obtain 2,337 sentences for EC-MT and 2,300 for EC-UQ. Table 1 shows the information of annotated results. Similarly, we produce the development and test datasets for system evaluation, by randomly selecting 400 sentences and letting two experts to generate the groundtruth annotations. Among them, we use 100 sentences as the development set and the remaining 300 as the test set. The rest sentences with only crowdsourcing annotations are used as the training set. Unlabeled data. The vector representations of characters are basic inputs of our baseline and proposed models, which are obtained by the looking-up table $\mathbf {E}^W$ . As introduced before, we can use pretrained embeddings from large-scale raw corpus to initialize the table. In order to pretrain the character embeddings, we use one large-scale unlabeled data from the user-generated content in Internet. Totally, we obtain a number of 5M sentences. Finally, we use the tool word2vec to pretrain the character embeddings based on the unlabeled dataset in our experiments. Settings For evaluation, we use the entity-level metrics of Precision (P), Recall (R), and their F1 value in our experiments, treating one tagged entity as correct only when it matches the gold entity exactly. There are several hyper-parameters in the baseline LSTM-CRF and our final models. We set them empirically by the development performances. Concretely, we set the dimension size of the character embeddings by 100, the dimension size of the NE label embeddings by 50, and the dimension sizes of all the other hidden features by 200. We exploit online training with a mini-batch size 128 to learn model parameters. The max-epoch iteration is set by 200, and the best-epoch model is chosen according to the development performances. We use RMSprop BIBREF28 with a learning rate $10^{-3}$ to update model parameters, and use $l_2$ -regularization by a parameter $10^{-5}$ . We adopt the dropout technique to avoid overfitting by a drop value of $0.2$ . Comparison Systems The proposed approach (henceforward referred to as “ALCrowd”) is compared with the following systems: CRF: We use the Crfsuite tool to train a model on the crowdsourcing labeled data. As for the feature settings, we use the supervised version of BIBREF0 zhao2008unsupervised. CRF-VT: We use the same settings of the CRF system, except that the training data is the voted version, whose groundtruths are produced by majority voting at the character level for each annotated sentence. CRF-MA: The CRF model proposed by BIBREF3 rodrigues2014sequence, which uses a prior distributation to model multiple crowdsourcing annotators. We use the source code provided by the authors. LSTM-CRF: Our baseline system trained on the crowdsourcing labeled data. LSTM-CRF-VT: Our baseline system trained on the voted corpus, which is the same as CRF-VT. LSTM-Crowd: The LSTM-CRF model with crowd annotation learning proposed by BIBREF4 nguyen2017aggregating. We use the source code provided by the authors. The first three systems are based on the CRF model using traditional handcrafted features, and the last three systems are based on the neural LSTM-CRF model. Among them, CRF-MA, LSTM-Crowd and our system with adversarial learning (ALCrowd) are based on crowd annotation learning that directly trains the model on the crowd-annotations. Five systems, including CRF, CRF-MA, LSTM-CRF, LSTM-Crowd, and ALCrowd, are trained on the original version of labeled data, while CRF-VT and LSTM-CRF-VT are trained on the voted version. Since CRF-VT, CRF-MA and LSTM-CRF-VT all require ground-truth answers for each training sentence, which are difficult to be produced with only two annotations, we do not apply the three models on the two EC datasets. Main Results In this section, we show the model performances of our proposed crowdsourcing learning system (ALCrowd), and meanwhile compare it with the other systems mentioned above. Table 2 shows the experimental results on the DL-PS datasets and Table 3 shows the experiment results on the EC-MT and EC-UQ datasets, respectively. The results of CRF and LSTM-CRF mean that the crowd annotation is an alternative solution with low cost for labeling data that could be used for training a NER system even there are some inconsistencies. Compared with CRF, LSTM-CRF achieves much better performances on all the three data, showing +6.12 F1 improvement on DL-PS, +4.51 on EC-MT, and +9.19 on EC-UQ. This indicates that LSTM-CRF is a very strong baseline system, demonstrating the effectiveness of neural network. Interestingly, when compared with CRF and LSTM-CRF, CRF-VT and LSTM-CRF-VT trained on the voted version perform worse in the DL-PS dataset. This trend is also mentioned in BIBREF4 nguyen2017aggregating. This fact shows that the majority voting method might be unsuitable for our task. There are two possible reasons accounting for the observation. On the one hand, simple character-level voting based on three annotations for each sentence may be still not enough. In the DL-PS dataset, even with only two predefined entity types, one character can have nine NE labels. Thus the majority-voting may be incapable of handling some cases. While the cost by adding more annotations for each sentence would be greatly increased. On the other hand, the lost information produced by majority-voting may be important, at least the ambiguous annotations denote that the input sentence is difficult for NER. The normal CRF and LSTM-CRF models without discard any annotations can differentiate these difficult contexts through learning. Three crowd-annotation learning systems provide better performances than their counterpart systems, (CRF-MA VS CRF) and (LSTM-Crowd/ALCrowd VS LSTM-CRF). Compared with the strong baseline LSTM-CRF, ALCrowd shows its advantage with +1.08 F1 improvements on DL-PS, +1.24 on EC-MT, and +2.38 on EC-UQ, respectively. This indicates that adding the crowd-annotation learning is quite useful for building NER systems. In addition, ALCrowd also outperforms LSTM-Crowd on all the datasets consistently, demonstrating the high effectiveness of ALCrowd in extracting worker independent features. Among all the systems, ALCrowd performs the best, and significantly better than all the other models (the p-value is below $10^{-5}$ by using t-test). The results indicate that with the help of adversarial training, our system can learn a better feature representation from crowd annotation. Discussion Impact of Character Embeddings. First, we investigate the effect of the pretrained character embeddings in our proposed crowdsourcing learning model. The comparison results are shown in Figure 2 , where Random refers to the random initialized character embeddings, and Pretrained refers to the embeddings pretrained on the unlabeled data. According to the results, we find that our model with the pretrained embeddings significantly outperforms that using the random embeddings, demonstrating that the pretrained embeddings successfully provide useful information. Case Studies. Second, we present several case studies in order to study the differences between our baseline and the worker adversarial models. We conduct a closed test on the training set, the results of which can be regarded as modifications of the training corpus, since there exist inconsistent annotations for each training sentence among the different workers. Figure 3 shows the two examples from the DL-PS dataset, which compares the outputs of the baseline and our final models, as well as the majority-voting strategy. In the first case, none of the annotations get the correct NER result, but our proposed model can capture it. The result of LSTM-CRF is the same as majority-voting. In the second example, the output of majority-voting is the worst, which can account for the reason why the same model trained on the voted corpus performs so badly, as shown in Table 2 . The model of LSTM-CRF fails to recognize the named entity “Xiexie” because of not trusting the second annotation, treating it as one noise annotation. Our proposed model is able to recognize it, because of its ability of extracting worker independent features. Conclusions In this paper, we presented an approach to performing crowd annotation learning based on the idea of adversarial training for Chinese Named Entity Recognition (NER). In our approach, we use a common and private Bi-LSTMs for representing annotator-generic and -specific information, and learn a label Bi-LSTM from the crowd-annotated NE label sequences. Finally, the proposed approach adopts a LSTM-CRF model to perform tagging. In our experiments, we create two data sets for Chinese NER tasks in the dialog and e-commerce domains. The experimental results show that the proposed approach outperforms strong baseline systems.  Acknowledgments This work is supported by the National Natural Science Foundation of China (Grant No. 61572338, 61525205, and 61602160). This work is also partially supported by the joint research project of Alibaba and Soochow University. Wenliang is also partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization.
What crowdsourcing platform is used?
2df4a045a9cd7b44874340b6fdf9308d3c55327a
What Ġcrowds ourcing Ġplatform Ġis Ġused ?
Un answer able
<s> What Ġcrowds ourcing Ġplatform Ġis Ġused ? </s> Introduction </s> There Ġhas Ġbeen Ġsignificant Ġprogress Ġon ĠNamed ĠEntity ĠRecogn ition Ġ( NER ) Ġin Ġrecent Ġyears Ġusing Ġmodels Ġbased Ġon Ġmachine Ġlearning Ġalgorithms ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠAs Ġwith Ġother ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġtasks , Ġbuilding ĠN ER Ġsystems Ġtypically Ġrequires Ġa Ġmassive Ġamount Ġof Ġlabeled Ġtraining Ġdata Ġwhich Ġare Ġannot ated Ġby Ġexperts . ĠIn Ġreal Ġapplications , Ġwe Ġoften Ġneed Ġto Ġconsider Ġnew Ġtypes Ġof Ġentities Ġin Ġnew Ġdomains Ġwhere Ġwe Ġdo Ġnot Ġhave Ġexisting Ġannot ated Ġdata . ĠFor Ġsuch Ġnew Ġtypes Ġof Ġentities , Ġhowever , Ġit Ġis Ġvery Ġhard Ġto Ġfind Ġexperts Ġto Ġannot ate Ġthe Ġdata Ġwithin Ġshort Ġtime Ġlimits Ġand Ġhiring Ġexperts Ġis Ġcostly Ġand Ġnon - sc al able , Ġboth Ġin Ġterms Ġof Ġtime Ġand Ġmoney . </s> In Ġorder Ġto Ġquickly Ġobtain Ġnew Ġtraining Ġdata , Ġwe Ġcan Ġuse Ġcrowds ourcing Ġas Ġone Ġalternative Ġway Ġat Ġlower Ġcost Ġin Ġa Ġshort Ġtime . ĠBut Ġas Ġan Ġexchange , Ġcrowd Ġannotations Ġfrom Ġnon - exper ts Ġmay Ġbe Ġof Ġlower Ġquality Ġthan Ġthose Ġfrom Ġexperts . ĠIt Ġis Ġone Ġbiggest Ġchallenge Ġto Ġbuild Ġa Ġpowerful ĠN ER Ġsystem Ġon Ġsuch Ġa Ġlow Ġquality Ġannot ated Ġdata . ĠAlthough Ġwe Ġcan Ġobtain Ġhigh Ġquality Ġannotations Ġfor Ġeach Ġinput Ġsentence Ġby Ġmajority Ġvoting , Ġit Ġcan Ġbe Ġa Ġwaste Ġof Ġhuman Ġlab ors Ġto Ġachieve Ġsuch Ġa Ġgoal , Ġespecially Ġfor Ġsome Ġambiguous Ġsentences Ġwhich Ġmay Ġrequire Ġa Ġnumber Ġof Ġannotations Ġto Ġreach Ġan Ġagreement . ĠThus Ġmajority Ġwork Ġdirectly Ġbuild Ġmodels Ġon Ġcrowd Ġannotations , Ġtrying Ġto Ġmodel Ġthe Ġdifferences Ġamong Ġannot ators , Ġfor Ġexample , Ġsome Ġof Ġthe Ġannot ators Ġmay Ġbe Ġmore Ġtrust ful ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. </s> Here Ġwe Ġfocus Ġmainly Ġon Ġthe ĠChinese ĠN ER , Ġwhich Ġis Ġmore Ġdifficult Ġthan ĠN ER Ġfor Ġother Ġlanguages Ġsuch Ġas ĠEnglish Ġfor Ġthe Ġlack Ġof Ġmorph ological Ġvariations Ġsuch Ġas Ġcapital ization Ġand Ġin Ġparticular Ġthe Ġuncertainty Ġin Ġword Ġsegment ation . ĠThe ĠChinese ĠNE Ġtag gers Ġtrained Ġon Ġnews Ġdomain Ġoften Ġperform Ġpoor Ġin Ġother Ġdomains . ĠAlthough Ġwe Ġcan Ġalleviate Ġthe Ġproblem Ġby Ġusing Ġcharacter - level Ġtagging Ġto Ġresolve Ġthe Ġproblem Ġof Ġpoor Ġword Ġsegment ation Ġperformances ĠB IB REF 5 Ġ, Ġstill Ġthere Ġexists Ġa Ġlarge Ġgap Ġwhen Ġthe Ġtarget Ġdomain Ġchanges , Ġespecially Ġfor Ġthe Ġtexts Ġof Ġsocial Ġmedia . ĠThus , Ġin Ġorder Ġto Ġget Ġa Ġgood Ġtag ger Ġfor Ġnew Ġdomains Ġand Ġalso Ġfor Ġthe Ġconditions Ġof Ġnew Ġentity Ġtypes , Ġwe Ġrequire Ġlarge Ġamounts Ġof Ġlabeled Ġdata . ĠTherefore , Ġcrowds ourcing Ġis Ġa Ġreasonable Ġsolution Ġfor Ġthese Ġsituations . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġan Ġapproach Ġto Ġtraining Ġa ĠChinese ĠN ER Ġsystem Ġon Ġthe Ġcrowd - annot ated Ġdata . ĠOur Ġgoal Ġis Ġto Ġextract Ġadditional Ġannot ator Ġindependent Ġfeatures Ġby Ġadvers arial Ġtraining , Ġallev iating Ġthe Ġannotation Ġnoises Ġof Ġnon - exper ts . ĠThe Ġidea Ġof Ġadvers arial Ġtraining Ġin Ġneural Ġnetworks Ġhas Ġbeen Ġused Ġsuccessfully Ġin Ġseveral ĠN LP Ġtasks , Ġsuch Ġas Ġcross - ling ual ĠPOS Ġtagging ĠB IB REF 6 Ġand Ġcross - domain ĠPOS Ġtagging ĠB IB REF 7 Ġ. ĠThey Ġuse Ġit Ġto Ġreduce Ġthe Ġnegative Ġinfluences Ġof Ġthe Ġinput Ġdiver g ences Ġamong Ġdifferent Ġdomains Ġor Ġlanguages , Ġwhile Ġwe Ġuse Ġadvers arial Ġtraining Ġto Ġreduce Ġthe Ġnegative Ġinfluences Ġbrought Ġby Ġdifferent Ġcrowd Ġannot ators . ĠTo Ġour Ġbest Ġknowledge , Ġwe Ġare Ġthe Ġfirst Ġto Ġapply Ġadvers arial Ġtraining Ġfor Ġcrowd Ġannotation Ġlearning . </s> In Ġthe Ġlearning Ġframework , Ġwe Ġperform Ġadvers arial Ġtraining Ġbetween Ġthe Ġbasic ĠN ER Ġand Ġan Ġadditional Ġworker Ġdiscrim inator . ĠWe Ġhave Ġa Ġcommon ĠBi - L ST M Ġfor Ġrepresenting Ġannot ator - generic Ġinformation Ġand Ġa Ġprivate ĠBi - L ST M Ġfor Ġrepresenting Ġannot ator - specific Ġinformation . ĠWe Ġbuild Ġanother Ġlabel ĠBi - L ST M Ġby Ġthe Ġcrowd - annot ated ĠNE Ġlabel Ġsequence Ġwhich Ġreflects Ġthe Ġmind Ġof Ġthe Ġcrowd Ġannot ators Ġwho Ġlearn Ġentity Ġdefinitions Ġby Ġreading Ġthe Ġannotation Ġguide book . ĠThe Ġcommon Ġand Ġprivate ĠBi - L ST Ms Ġare Ġused Ġfor ĠN ER , Ġwhile Ġthe Ġcommon Ġand Ġlabel ĠBi - L ST Ms Ġare Ġused Ġas Ġinputs Ġfor Ġthe Ġworker Ġdiscrim inator . ĠThe Ġparameters Ġof Ġthe Ġcommon ĠBi - L ST M Ġare Ġlearned Ġby Ġadvers arial Ġtraining , Ġmaximizing Ġthe Ġworker Ġdiscrim inator Ġloss Ġand Ġmeanwhile Ġminimizing Ġthe ĠN ER Ġloss . ĠThus Ġthe Ġresulting Ġfeatures Ġof Ġthe Ġcommon ĠBi - L ST M Ġare Ġworker Ġinvari ant Ġand ĠN ER Ġsensitive . </s> For Ġevaluation , Ġwe Ġcreate Ġtwo ĠChinese ĠN ER Ġdatasets Ġin Ġtwo Ġdomains : Ġdialog Ġand Ġe - commerce . ĠWe Ġrequire Ġthe Ġcrowd Ġannot ators Ġto Ġlabel Ġthe Ġtypes Ġof Ġentities , Ġincluding Ġperson , Ġsong , Ġbrand , Ġproduct , Ġand Ġso Ġon . ĠIdent ifying Ġthese Ġentities Ġis Ġuseful Ġfor Ġchat bot Ġand Ġe - commerce Ġplatforms ĠB IB REF 8 Ġ. ĠThen Ġwe Ġconduct Ġexperiments Ġon Ġthe Ġnewly Ġcreated Ġdatasets Ġto Ġverify Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġadvers arial Ġneural Ġnetwork Ġmodel . ĠThe Ġresults Ġshow Ġthat Ġour Ġsystem Ġoutper forms Ġvery Ġstrong Ġbaseline Ġsystems . ĠIn Ġsummary , Ġwe Ġmake Ġthe Ġfollowing Ġcontributions : </s> Related ĠWork </s> Our Ġwork Ġis Ġrelated Ġto Ġthree Ġlines Ġof Ġresearch : ĠSequence Ġlabeling , ĠAd vers arial Ġtraining , Ġand ĠCrowd s ourcing . </s> Sequ ence Ġlabeling . ĠN ER Ġis Ġwidely Ġtreated Ġas Ġa Ġsequence Ġlabeling Ġproblem , Ġby Ġassigning Ġa Ġunique Ġlabel Ġover Ġeach Ġsent ential Ġword ĠB IB REF 9 Ġ. ĠEarly Ġstudies Ġon Ġsequence Ġlabeling Ġoften Ġuse Ġthe Ġmodels Ġof ĠH MM , ĠMEM M , Ġand ĠCR F ĠB IB REF 10 Ġbased Ġon Ġmanually - crafted Ġdiscrete Ġfeatures , Ġwhich Ġcan Ġsuffer Ġthe Ġfeature Ġsp arsity Ġproblem Ġand Ġrequire Ġheavy Ġfeature Ġengineering . ĠRecently , Ġneural Ġnetwork Ġmodels Ġhave Ġbeen Ġsuccessfully Ġapplied Ġto Ġsequence Ġlabeling ĠB IB REF 1 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 2 Ġ. ĠAmong Ġthese Ġwork , Ġthe Ġmodel Ġwhich Ġuses ĠBi - L ST M Ġfor Ġfeature Ġextraction Ġand ĠCR F Ġfor Ġdecoding Ġhas Ġachieved Ġstate - of - the - art Ġperformances ĠB IB REF 11 Ġ, ĠB IB REF 2 Ġ, Ġwhich Ġis Ġexploited Ġas Ġthe Ġbaseline Ġmodel Ġin Ġour Ġwork . </s> Ad vers arial ĠTraining . ĠAd vers arial ĠNetworks Ġhave Ġachieved Ġgreat Ġsuccess Ġin Ġcomputer Ġvision Ġsuch Ġas Ġimage Ġgeneration ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠIn Ġthe ĠN LP Ġcommunity , Ġthe Ġmethod Ġis Ġmainly Ġexploited Ġunder Ġthe Ġsettings Ġof Ġdomain Ġad a ption ĠB IB REF 14 Ġ, ĠB IB REF 7 Ġ, Ġcross - ling ual ĠB IB REF 15 Ġ, ĠB IB REF 6 Ġand Ġmulti - task Ġlearning ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. ĠAll Ġthese Ġsettings Ġinvolve Ġthe Ġfeature Ġdiver g ences Ġbetween Ġthe Ġtraining Ġand Ġtest Ġexamples , Ġand Ġaim Ġto Ġlearn Ġinvari ant Ġfeatures Ġacross Ġthe Ġdiver g ences Ġby Ġan Ġadditional Ġadvers arial Ġdiscrim inator , Ġsuch Ġas Ġdomain Ġdiscrim inator . ĠOur Ġwork Ġis Ġsimilar Ġto Ġthese Ġwork Ġbut Ġis Ġapplies Ġon Ġcrowds ourcing Ġlearning , Ġaiming Ġto Ġfind Ġinvari ant Ġfeatures Ġamong Ġdifferent Ġcrowds ourcing Ġworkers . </s> C rowd s ourcing . ĠMost ĠN LP Ġtasks Ġrequire Ġa Ġmassive Ġamount Ġof Ġlabeled Ġtraining Ġdata Ġwhich Ġare Ġannot ated Ġby Ġexperts . ĠHowever , Ġhiring Ġexperts Ġis Ġcostly Ġand Ġnon - sc al able , Ġboth Ġin Ġterms Ġof Ġtime Ġand Ġmoney . ĠInstead , Ġcrowds ourcing Ġis Ġanother Ġsolution Ġto Ġobtain Ġlabeled Ġdata Ġat Ġa Ġlower Ġcost Ġbut Ġwith Ġrelative Ġlower Ġquality Ġthan Ġthose Ġfrom Ġexperts . ĠB IB REF 18 Ġsnow 2008 che ap Ġcollected Ġlabeled Ġresults Ġfor Ġseveral ĠN LP Ġtasks Ġfrom ĠAmazon ĠMechanical ĠTurk Ġand Ġdemonstrated Ġthat Ġnon - exper ts Ġannotations Ġwere Ġquite Ġuseful Ġfor Ġtraining Ġnew Ġsystems . ĠIn Ġrecent Ġyears , Ġa Ġseries Ġof Ġwork Ġhave Ġfocused Ġon Ġhow Ġto Ġuse Ġcrowds ourcing Ġdata Ġefficiently Ġin Ġtasks Ġsuch Ġas Ġclassification ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, Ġand Ġcompare Ġquality Ġof Ġcrowd Ġand Ġexpert Ġlabels ĠB IB REF 21 Ġ. </s> In Ġsequence Ġlabeling Ġtasks , ĠB IB REF 22 Ġdred ze 2009 sequence Ġviewed Ġthis Ġtask Ġas Ġa Ġmulti - label Ġproblem Ġwhile ĠB IB REF 3 Ġrod rig ues 2014 sequence Ġtook Ġworkers Ġidentities Ġinto Ġaccount Ġby Ġassuming Ġthat Ġeach Ġsent ential Ġword Ġwas Ġtagged Ġcorrectly Ġby Ġone Ġof Ġthe Ġcrowds ourcing Ġworkers Ġand Ġproposed Ġa ĠCR F - based Ġmodel Ġwith Ġmultiple Ġannot ators . ĠB IB REF 4 Ġn guyen 2017 agg reg ating Ġintroduced Ġa Ġcrowd Ġrepresentation Ġin Ġwhich Ġthe Ġcrowd Ġvectors Ġwere Ġadded Ġinto Ġthe ĠL ST M - CR F Ġmodel Ġat Ġtrain Ġtime , Ġbut Ġignored Ġthem Ġat Ġtest Ġtime . ĠIn Ġthis Ġpaper , Ġwe Ġapply Ġadvers arial Ġtraining Ġon Ġcrowd Ġannotations Ġon ĠChinese ĠN ER Ġin Ġnew Ġdomains , Ġand Ġachieve Ġbetter Ġperformances Ġthan Ġprevious Ġstudies Ġon Ġcrowds ourcing Ġlearning . </s> Bas eline : ĠL ST M - CR F </s> We Ġuse Ġa Ġneural ĠCR F Ġmodel Ġas Ġthe Ġbaseline Ġsystem ĠB IB REF 9 Ġ, Ġtreating ĠN ER Ġas Ġa Ġsequence Ġlabeling Ġproblem Ġover ĠChinese Ġcharacters , Ġwhich Ġhas Ġachieved Ġstate - of - the - art Ġperformances ĠB IB REF 5 Ġ. ĠTo Ġthis Ġend , Ġwe Ġexplore Ġthe ĠBI EO Ġschema Ġto Ġconvert ĠN ER Ġinto Ġsequence Ġlabeling , Ġfollowing ĠB IB REF 2 Ġl ample - E t Al : 2016 : N 16 - 1 , Ġwhere Ġsent ential Ġcharacter Ġis Ġassigned Ġwith Ġone Ġunique Ġtag . ĠCon crete ly , Ġwe Ġtag Ġthe Ġnon - entity Ġcharacter Ġby Ġlabel ĠâĢ ľ O âĢ Ŀ , Ġthe Ġbeginning Ġcharacter Ġof Ġan Ġentity Ġby ĠâĢ ľ B - XX âĢ Ŀ , Ġthe Ġending Ġcharacter Ġof Ġan Ġentity Ġby ĠâĢ ľ E - XX âĢ Ŀ Ġand Ġthe Ġother Ġcharacter Ġof Ġan Ġentity Ġby ĠâĢ ľ I - XX âĢ Ŀ , Ġwhere ĠâĢ ľ XX âĢ Ŀ Ġdenotes Ġthe Ġentity Ġtype . </s> We Ġbuild Ġhigh - level Ġneural Ġfeatures Ġfrom Ġthe Ġinput Ġcharacter Ġsequence Ġby Ġa Ġbi - direction al ĠL ST M ĠB IB REF 2 Ġ. ĠThe Ġresulting Ġfeatures Ġare Ġcombined Ġand Ġthen Ġare Ġfed Ġinto Ġan Ġoutput ĠCR F Ġlayer Ġfor Ġdecoding . ĠIn Ġsummary , Ġthe Ġbaseline Ġmodel Ġhas Ġthree Ġmain Ġcomponents . ĠFirst , Ġwe Ġmake Ġvector Ġrepresentations Ġfor Ġsent ential Ġcharacters Ġ$\ math bf Ġ{ x } _ 1 \ math bf Ġ{ x } _ 2 \ cd ots Ġ\ math bf Ġ{ x } _ n $ Ġ, Ġtransforming Ġthe Ġdiscrete Ġinputs Ġinto Ġlow - dimensional Ġneural Ġinputs . ĠSecond , Ġfeature Ġextraction Ġis Ġperformed Ġto Ġobtain Ġhigh - level Ġfeatures Ġ$\ math bf Ġ{ h } _ 1 ^ {\ text { ner }} \ math bf Ġ{ h } _ 2 ^ {\ text { ner }} \ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { ner }} $ Ġ, Ġby Ġusing Ġa Ġbi - direction al ĠL ST M Ġ( Bi - L ST M ) Ġstructure Ġtogether Ġwith Ġa Ġlinear Ġtransformation Ġover Ġ$\ math bf Ġ{ x } _ 1 \ math bf Ġ{ x } _ 2 \ cd ots Ġ\ math bf Ġ{ x } _ n $ Ġ. ĠThird , Ġwe Ġapply Ġa ĠCR F Ġtagging Ġmodule Ġover Ġ$\ math bf Ġ{ h } _ 1 ^ {\ text { ner }} \ math bf Ġ{ h } _ 2 ^ {\ text { ner }} \ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { ner }} $ Ġ, Ġobtaining Ġthe Ġfinal Ġoutput ĠNE Ġlabels . ĠThe Ġoverall Ġframework Ġof Ġthe Ġbaseline Ġmodel Ġis Ġshown Ġby Ġthe Ġright Ġpart Ġof ĠFigure Ġ1 Ġ. </s> Vector ĠRepresent ation Ġof ĠCharacters </s> To Ġrepresent ĠChinese Ġcharacters , Ġwe Ġsimply Ġexploit Ġa Ġneural Ġembed ding Ġlayer Ġto Ġmap Ġdiscrete Ġcharacters Ġinto Ġthe Ġlow - dimensional Ġvector Ġrepresentations . ĠThe Ġgoal Ġis Ġachieved Ġby Ġa Ġlooking - up Ġtable Ġ$\ math bf Ġ{ E } ^ W $ Ġ, Ġwhich Ġis Ġa Ġmodel Ġparameter Ġand Ġwill Ġbe Ġfine - tun ed Ġduring Ġtraining . ĠThe Ġlooking - up Ġtable Ġcan Ġbe Ġinitialized Ġeither Ġby Ġrandom Ġor Ġby Ġusing Ġa Ġpret rained Ġembed d ings Ġfrom Ġlarge Ġscale Ġraw Ġcorpus . ĠFor Ġa Ġgiven ĠChinese Ġcharacter Ġsequence Ġ$ c _ 1 c _ 2 \ cd ots Ġc _ n $ Ġ, Ġwe Ġobtain Ġthe Ġvector Ġrepresentation Ġof Ġeach Ġsent ential Ġcharacter Ġby : Ġ$ Ġ\ math bf Ġ{ x } _ t Ġ= Ġ\ text { look - up } ( c _ t , Ġ\ math bf Ġ{ E } ^ W ), Ġ\ text { ~~ ~ } Ġt Ġ\ in Ġ[ 1 , Ġn ] $ Ġ. </s> Feature ĠExt raction </s> Based Ġon Ġthe Ġvector Ġsequence Ġ$\ math bf Ġ{ x } _ 1 \ math bf Ġ{ x } _ 2 \ cd ots Ġ\ math bf Ġ{ x } _ n $ Ġ, Ġwe Ġextract Ġhigher - level Ġfeatures Ġ$\ math bf Ġ{ h } _ 1 ^ {\ text { ner }} \ math bf Ġ{ h } _ 2 ^ {\ text { ner }} \ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { ner }} $ Ġby Ġusing Ġa Ġbid irection al ĠL ST M Ġmodule Ġand Ġa Ġsimple Ġfeed - forward Ġneural Ġlayer , Ġwhich Ġare Ġthen Ġused Ġfor ĠCR F Ġtagging Ġat Ġthe Ġnext Ġstep . </s> L ST M Ġis Ġa Ġtype Ġof Ġrecurrent Ġneural Ġnetwork Ġ( R NN ), Ġwhich Ġis Ġdesigned Ġfor Ġsolving Ġthe Ġexploding Ġand Ġdiminishing Ġgrad ients Ġof Ġbasic ĠR NN s ĠB IB REF 23 Ġ. ĠIt Ġhas Ġbeen Ġwidely Ġused Ġin Ġa Ġnumber Ġof ĠN LP Ġtasks , Ġincluding ĠPOS - tag ging ĠB IB REF 11 Ġ, ĠB IB REF 24 Ġ, Ġparsing ĠB IB REF 25 Ġand Ġmachine Ġtranslation ĠB IB REF 26 Ġ, Ġbecause Ġof Ġits Ġstrong Ġcapabilities Ġof Ġmodeling Ġnatural Ġlanguage Ġsentences . </s> By Ġtravers ing Ġ$\ math bf Ġ{ x } _ 1 \ math bf Ġ{ x } _ 2 \ cd ots Ġ\ math bf Ġ{ x } _ n $ Ġby Ġorder Ġand Ġreverse ly , Ġwe Ġobtain Ġthe Ġoutput Ġfeatures Ġ$\ math bf Ġ{ h } _ 1 ^ {\ text { private }} \ math bf Ġ{ h } _ 2 ^ {\ text { private }} \ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { private }} $ Ġof Ġthe Ġbi - L ST M , Ġwhere Ġ$\ math bf Ġ{ h } _ t ^ {\ text { private }} Ġ= Ġ\ over right arrow {\ math bf Ġ{ h }} _ t Ġ\ opl us Ġ\ over left arrow {\ math bf Ġ{ h }} _ t Ġ$ Ġ. ĠHere Ġwe Ġrefer Ġthis ĠBi - L ST M Ġas Ġprivate Ġin Ġorder Ġto Ġdifferentiate Ġit Ġwith Ġthe Ġcommon ĠBi - L ST M Ġover Ġthe Ġsame Ġcharacter Ġinputs Ġwhich Ġwill Ġbe Ġintroduced Ġin Ġthe Ġnext Ġsection . </s> Further Ġwe Ġmake Ġan Ġintegration Ġof Ġthe Ġoutput Ġvectors Ġof Ġbi - direction al ĠL ST M Ġby Ġa Ġlinear Ġfeed - forward Ġneural Ġlayer , Ġresulting Ġin Ġthe Ġfeatures Ġ$\ math bf Ġ{ h } _ 1 ^ {\ text { ner }} \ math bf Ġ{ h } _ 2 ^ {\ text { ner }} \ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { ner }} $ Ġby Ġequation : </s> $$ \ math bf Ġ{ h } _ t ^ {\ text { ner }} Ġ= Ġ\ math bf Ġ{ W } Ġ\ math bf Ġ{ h } _ t ^ {\ text { private }} Ġ+ Ġ\ math bf Ġ{ b }, $$ Ġ Ġ Ġ( E q . Ġ6 ) </s> where Ġ$\ math bf Ġ{ W } $ Ġand Ġ$\ math bf Ġ{ b } $ Ġare Ġboth Ġmodel Ġparameters . </s> CR F ĠT agging </s> Finally Ġwe Ġfeed Ġthe Ġresulting Ġfeatures Ġ$\ math bf Ġ{ h } _ t ^ {\ text { ner } }, Ġt \ in Ġ[ 1 , Ġn ] $ Ġinto Ġa ĠCR F Ġlayer Ġdirectly Ġfor ĠN ER Ġdecoding . ĠCR F Ġtagging Ġis Ġone Ġglobally Ġnormalized Ġmodel , Ġaiming Ġto Ġfind Ġthe Ġbest Ġoutput Ġsequence Ġconsidering Ġthe Ġdependencies Ġbetween Ġsuccessive Ġlabels . ĠIn Ġthe Ġsequence Ġlabeling Ġsetting Ġfor ĠN ER , Ġthe Ġoutput Ġlabel Ġof Ġone Ġposition Ġhas Ġa Ġstrong Ġdependency Ġon Ġthe Ġlabel Ġof Ġthe Ġprevious Ġposition . ĠFor Ġexample , Ġthe Ġlabel Ġbefore ĠâĢ ľ I - XX âĢ Ŀ Ġmust Ġbe Ġeither ĠâĢ ľ B - XX âĢ Ŀ Ġor ĠâĢ ľ I - XX âĢ Ŀ , Ġwhere ĠâĢ ľ XX âĢ Ŀ Ġshould Ġbe Ġexactly Ġthe Ġsame . </s> CR F Ġinvolves Ġtwo Ġparts Ġfor Ġprediction . ĠFirst Ġwe Ġshould Ġcompute Ġthe Ġscores Ġfor Ġeach Ġlabel Ġbased Ġ$\ math bf Ġ{ h } _ t ^ {\ text { ner }} $ Ġ, Ġresulting Ġin Ġ$\ math bf Ġ{ o } _ t ^ {\ text { ner }} $ Ġ, Ġwhose Ġdimension Ġis Ġthe Ġnumber Ġof Ġoutput Ġlabels . ĠThe Ġother Ġpart Ġis Ġa Ġtransition Ġmatrix Ġ$\ math bf Ġ{ T } $ Ġwhich Ġdefines Ġthe Ġscores Ġof Ġtwo Ġsuccessive Ġlabels . Ġ$\ math bf Ġ{ T } $ Ġis Ġalso Ġa Ġmodel Ġparameter . ĠBased Ġon Ġ$\ math bf Ġ{ o } _ t ^ {\ text { ner }} $ Ġand Ġ$\ math bf Ġ{ T } $ Ġ, Ġwe Ġuse Ġthe ĠV iter bi Ġalgorithm Ġto Ġfind Ġthe Ġbest - scoring Ġlabel Ġsequence . </s> We Ġcan Ġformal ize Ġthe ĠCR F Ġtagging Ġprocess Ġas Ġfollows : </s> $$ \ begin { split } Ġ& Ġ\ math bf Ġ{ o } _ t ^ {\ text { ner }} Ġ= Ġ\ math bf Ġ{ W } ^ {\ text { ner }} Ġ\ math bf Ġ{ h } _ t ^ {\ text { ner } }, Ġ\ text { ~~~~ } Ġt Ġ\ in Ġ[ 1 , n ] Ġ\\ Ġ& Ġ\ text { score } (\ math bf Ġ{ X }, Ġ\ math bf Ġ{ y }) Ġ= Ġ\ sum Ġ_ { t Ġ= Ġ1 } ^{ n } (\ math bf Ġ{ o } _{ t , y _ t } Ġ+ ĠT _{ y _{ t - 1 }, y _ t }) Ġ\\ Ġ& Ġ\ math bf Ġ{ y } ^ {\ text { ner }} Ġ= Ġ\ math op Ġ{ arg ~ max } _{ \ math bf Ġ{ y }} \ big Ġ( \ text { score } (\ math bf Ġ{ X }, Ġ\ math bf Ġ{ y } )) \ big Ġ), Ġ\\ Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> where Ġ$\ text { score } (\ cd ot Ġ) $ Ġis Ġthe Ġscoring Ġfunction Ġfor Ġa Ġgiven Ġoutput Ġlabel Ġsequence Ġ$\ math bf Ġ{ y } Ġ= Ġy _ 1 y _ 2 Ġ\ cd ots Ġy _ n $ Ġbased Ġon Ġinput Ġ$\ math bf Ġ{ X } $ Ġ, Ġ$\ math bf Ġ{ y } ^ {\ text { ner }} $ Ġis Ġthe Ġresulting Ġlabel Ġsequence , Ġ$\ math bf Ġ{ W } ^ {\ text { ner }} $ Ġis Ġa Ġmodel Ġparameter . </s> Training </s> To Ġtrain Ġmodel Ġparameters , Ġwe Ġexploit Ġa Ġnegative Ġlog - like lihood Ġobjective Ġas Ġthe Ġloss Ġfunction . ĠWe Ġapply Ġsoft max Ġover Ġall Ġcandidate Ġoutput Ġlabel Ġsequences , Ġthus Ġthe Ġprobability Ġof Ġthe Ġcrowd - annot ated Ġlabel Ġsequence Ġis Ġcomputed Ġby : </s> $$ p (\ math bf Ġ{\ bar { y }} | \ math bf Ġ{ X }) Ġ= Ġ\ frac {\ exp Ġ\ big Ġ( \ text { score } (\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y }} )\ big Ġ) }{ \ sum Ġ_ {\ math bf Ġ{ y } Ġ\ in Ġ\ math bf Ġ{ Y } _{ \ math bf Ġ{ X }}} Ġ\ exp Ġ\ big Ġ( \ text { score } (\ math bf Ġ{ X }, Ġ\ math bf Ġ{ y } )\ big Ġ) }, $$ Ġ Ġ Ġ( E q . Ġ10 ) </s> where Ġ$\ math bf Ġ{\ bar { y }} $ Ġis Ġthe Ġcrowd - annot ated Ġlabel Ġsequences Ġand Ġ$\ math bf Ġ{ Y } _{ \ math bf Ġ{ X }} $ Ġis Ġall Ġcandidate Ġlabel Ġsequence Ġof Ġinput Ġ$\ math bf Ġ{ X } $ Ġ. </s> Based Ġon Ġthe Ġabove Ġformula , Ġthe Ġloss Ġfunction Ġof Ġour Ġbaseline Ġmodel Ġis : </s> $$ \ text { loss } (\ The ta Ġ, Ġ\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y }} ) Ġ= Ġ- \ log Ġp (\ math bf Ġ{\ bar { y }} | \ math bf Ġ{ X } ), $$ Ġ Ġ Ġ( E q . Ġ11 ) </s> where Ġ$\ The ta Ġ$ Ġis Ġthe Ġset Ġof Ġall Ġmodel Ġparameters . ĠWe Ġuse Ġstandard Ġback - prop ag ation Ġmethod Ġto Ġminimize Ġthe Ġloss Ġfunction Ġof Ġthe Ġbaseline ĠCR F Ġmodel . </s> Work er ĠAd vers arial </s> Ad vers arial Ġlearning Ġhas Ġbeen Ġan Ġeffective Ġmechanism Ġto Ġresolve Ġthe Ġproblem Ġof Ġthe Ġinput Ġfeatures Ġbetween Ġthe Ġtraining Ġand Ġtest Ġexamples Ġhaving Ġlarge Ġdiver g ences ĠB IB REF 27 Ġ, ĠB IB REF 13 Ġ. ĠIt Ġhas Ġbeen Ġsuccessfully Ġapplied Ġon Ġdomain Ġad a ption ĠB IB REF 7 Ġ, Ġcross - ling ual Ġlearning ĠB IB REF 15 Ġand Ġmulti - task Ġlearning ĠB IB REF 17 Ġ. ĠAll Ġsettings Ġinvolve Ġfeature Ġshifting Ġbetween Ġthe Ġtraining Ġand Ġtesting . </s> In Ġthis Ġpaper , Ġour Ġsetting Ġis Ġdifferent . ĠWe Ġare Ġusing Ġthe Ġannotations Ġfrom Ġnon - exper ts , Ġwhich Ġare Ġnoise Ġand Ġcan Ġinfluence Ġthe Ġfinal Ġperformances Ġif Ġthey Ġare Ġnot Ġproperly Ġprocessed . ĠDirect ly Ġlearning Ġbased Ġon Ġthe Ġresulting Ġcorpus Ġmay Ġadapt Ġthe Ġneural Ġfeature Ġextraction Ġinto Ġthe Ġbiased Ġannotations . ĠIn Ġthis Ġwork , Ġwe Ġassume Ġthat Ġindividual Ġworkers Ġhave Ġtheir Ġown Ġguidelines Ġin Ġmind Ġafter Ġshort Ġtraining . ĠFor Ġexample , Ġa Ġperfect Ġworker Ġcan Ġannot ate Ġhighly Ġconsistently Ġwith Ġan Ġexpert , Ġwhile Ġcommon Ġcrowds ourcing Ġworkers Ġmay Ġbe Ġconfused Ġand Ġhave Ġdifferent Ġunderstand ings Ġon Ġcertain Ġcontexts . ĠBased Ġon Ġthe Ġassumption , Ġwe Ġmake Ġan Ġad a ption Ġfor Ġthe Ġoriginal Ġadvers arial Ġneural Ġnetwork Ġto Ġour Ġsetting . </s> Our Ġad a ption Ġis Ġvery Ġsimple . ĠBrief ly Ġspeaking , Ġthe Ġoriginal Ġadvers arial Ġlearning Ġadds Ġan Ġadditional Ġdiscrim inator Ġto Ġclassify Ġthe Ġtype Ġof Ġsource Ġinputs , Ġfor Ġexample , Ġthe Ġdomain Ġcategory Ġin Ġthe Ġdomain Ġad a ption Ġsetting , Ġwhile Ġwe Ġadd Ġa Ġdiscrim inator Ġto Ġclassify Ġthe Ġannotation Ġworkers . ĠSole ly Ġthe Ġfeatures Ġfrom Ġthe Ġinput Ġsentence Ġis Ġnot Ġenough Ġfor Ġworker Ġclassification . ĠThe Ġannotation Ġresult Ġof Ġthe Ġworker Ġis Ġalso Ġrequired . ĠThus Ġthe Ġinputs Ġof Ġour Ġdiscrim inator Ġare Ġdifferent . ĠHere Ġwe Ġexploit Ġboth Ġthe Ġsource Ġsentences Ġand Ġthe Ġcrowd - annot ated ĠNE Ġlabels Ġas Ġbasic Ġinputs Ġfor Ġthe Ġworker Ġdiscrimination . </s> In Ġthe Ġfollowing , Ġwe Ġdescribe Ġthe Ġproposed Ġadvers arial Ġlearning Ġmodule , Ġincluding Ġboth Ġthe Ġsub models Ġand Ġthe Ġtraining Ġmethod . ĠAs Ġshown Ġby Ġthe Ġleft Ġpart Ġof ĠFigure Ġ1 Ġ, Ġthe Ġsub model Ġconsists Ġof Ġfour Ġparts : Ġ( 1 ) Ġa Ġcommon ĠBi - L ST M Ġover Ġinput Ġcharacters ; Ġ( 2 ) Ġan Ġadditional ĠBi - L ST M Ġto Ġencode Ġcrowd - annot ated ĠNE Ġlabel Ġsequence ; Ġ( 3 ) Ġa Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġto Ġextract Ġfeatures Ġfor Ġworker Ġdiscrim inator ; Ġ( 4 ) Ġoutput Ġand Ġprediction . </s> Common ĠBi - L ST M Ġover ĠCharacters </s> To Ġbuild Ġthe Ġadvers arial Ġpart , Ġfirst Ġwe Ġcreate Ġa Ġnew Ġbi - direction al ĠL ST M , Ġnamed Ġby Ġthe Ġcommon ĠBi - L ST M : </s> $$ \ math bf Ġ{ h } _ 1 ^ {\ text {\ tiny Ġcommon }} Ġ\ math bf Ġ{ h } _ 2 ^ {\ text {\ tiny Ġcommon }} Ġ\ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text {\ tiny Ġcommon }} Ġ= Ġ\ text { Bi - L ST M } (\ math bf Ġ{ x } _ 1 \ math bf Ġ{ x } _ 2 \ cd ots Ġ\ math bf Ġ{ x } _ n ). $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> As Ġshown Ġin ĠFigure Ġ1 Ġ, Ġthis ĠBi - L ST M Ġis Ġconstructed Ġover Ġthe Ġsame Ġinput Ġcharacter Ġrepresentations Ġof Ġthe Ġprivate ĠBi - L ST M , Ġin Ġorder Ġto Ġextract Ġworker Ġindependent Ġfeatures . </s> The Ġresulting Ġfeatures Ġof Ġthe Ġcommon ĠBi - L ST M Ġare Ġused Ġfor Ġboth ĠN ER Ġand Ġthe Ġworker Ġdiscrim inator , Ġdifferent Ġwith Ġthe Ġfeatures Ġof Ġprivate ĠBi - L ST M Ġwhich Ġare Ġused Ġfor ĠN ER Ġonly . ĠAs Ġshown Ġin ĠFigure Ġ1 Ġ, Ġwe Ġconc aten ate Ġthe Ġoutputs Ġof Ġthe Ġcommon Ġand Ġprivate ĠBi - L ST Ms Ġtogether , Ġand Ġthen Ġfeed Ġthe Ġresults Ġinto Ġthe Ġfeed - forward Ġcombination Ġlayer Ġof Ġthe ĠN ER Ġpart . ĠThus ĠFormula Ġ6 Ġcan Ġbe Ġrewritten Ġas : </s> $$ \ math bf Ġ{ h } _ t ^ {\ text { ner }} Ġ= Ġ\ math bf Ġ{ W } Ġ( \ math bf Ġ{ h } _ t ^ {\ text { common }} Ġ\ opl us Ġ\ math bf Ġ{ h } _ t ^ {\ text { private }} ) Ġ+ Ġ\ math bf Ġ{ b }, $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$\ math bf Ġ{ W } $ Ġis Ġwider Ġthan Ġthe Ġoriginal Ġcombination Ġbecause Ġthe Ġnewly - added Ġ$\ math bf Ġ{ h } _ t ^ {\ text { common }} $ Ġ. </s> Notice ably , Ġalthough Ġthe Ġresulting Ġcommon Ġfeatures Ġare Ġused Ġfor Ġthe Ġworker Ġdiscrim inator , Ġthey Ġactually Ġhave Ġno Ġcapability Ġto Ġdistinguish Ġthe Ġworkers . ĠBecause Ġthis Ġpart Ġis Ġexploited Ġto Ġmaximize Ġthe Ġloss Ġof Ġthe Ġworker Ġdiscrim inator , Ġit Ġwill Ġbe Ġinterpreted Ġin Ġthe Ġlater Ġtraining Ġsubsection . ĠThese Ġfeatures Ġare Ġinvari ant Ġamong Ġdifferent Ġworkers , Ġthus Ġthey Ġcan Ġhave Ġless Ġnoises Ġfor ĠN ER . ĠThis Ġis Ġthe Ġgoal Ġof Ġadvers arial Ġlearning , Ġand Ġwe Ġhope Ġthe ĠN ER Ġbeing Ġable Ġto Ġfind Ġuseful Ġfeatures Ġfrom Ġthese Ġworker Ġindependent Ġfeatures . </s> Additional ĠBi - L ST M Ġover ĠAn not ated ĠN ER ĠLab els </s> In Ġorder Ġto Ġincorporate Ġthe Ġannot ated ĠNE Ġlabels Ġto Ġpredict Ġthe Ġexact Ġworker , Ġwe Ġbuild Ġanother Ġbi - direction al ĠL ST M Ġ( named Ġby Ġlabel ĠBi - L ST M ) Ġbased Ġon Ġthe Ġcrowd - annot ated ĠNE Ġlabel Ġsequence . ĠThis ĠBi - L ST M Ġis Ġused Ġfor Ġworker Ġdiscrim inator Ġonly . ĠDuring Ġthe Ġdecoding Ġof Ġthe Ġtesting Ġphase , Ġwe Ġwill Ġnever Ġhave Ġthis ĠBi - L ST M , Ġbecause Ġthe Ġworker Ġdiscrim inator Ġis Ġno Ġlonger Ġrequired . </s> Assuming Ġthe Ġcrowd - annot ated ĠNE Ġlabel Ġsequence Ġannot ated Ġby Ġone Ġworker Ġis Ġ$\ math bf Ġ{\ bar { y }} Ġ= Ġ\ bar { y } _ 1 \ bar { y } _ 2 Ġ\ cd ots Ġ\ bar { y } _ n $ Ġ, Ġwe Ġexploit Ġa Ġlooking - up Ġtable Ġ$\ math bf Ġ{ E } ^{ L } $ Ġto Ġobtain Ġthe Ġcorresponding Ġsequence Ġof Ġtheir Ġvector Ġrepresentations Ġ$\ math bf Ġ{ x ^ {\ prime Ġ}} _ 1 \ math bf Ġ{ x ^ {\ prime Ġ}} _ 2 \ cd ots Ġ\ math bf Ġ{ x ^ {\ prime Ġ}} _ n $ Ġ, Ġsimilar Ġto Ġthe Ġmethod Ġthat Ġmaps Ġcharacters Ġinto Ġtheir Ġneural Ġrepresentations . ĠCon crete ly , Ġfor Ġone ĠNE Ġlabel Ġ$\ bar { y } _ t $ Ġ( Ġ$ t Ġ\ in Ġ[ 1 , Ġn ] $ Ġ), Ġwe Ġobtain Ġits Ġneural Ġvector Ġby : Ġ$\ math bf Ġ{ x ^ {\ prime Ġ}} _ t Ġ= Ġ\ text { look - up } (\ bar { y } _ t , Ġ\ math bf Ġ{ E } ^ L ) $ Ġ. </s> Next Ġstep Ġwe Ġapply Ġbi - direction al ĠL ST M Ġover Ġthe Ġsequence Ġ$\ math bf Ġ{ x ^ {\ prime Ġ}} _ 1 \ math bf Ġ{ x ^ {\ prime Ġ}} _ 2 \ cd ots Ġ\ math bf Ġ{ x ^ {\ prime Ġ}} _ n $ Ġ, Ġwhich Ġcan Ġbe Ġformal ized Ġas : </s> $$ \ math bf Ġ{ h } _ 1 ^ {\ text { label }} Ġ\ math bf Ġ{ h } _ 2 ^ {\ text { label }} Ġ\ cd ots Ġ\ math bf Ġ{ h } _ n ^ {\ text { label }} Ġ= Ġ\ text { Bi - L ST M } (\ math bf Ġ{ x ^ {\ prime Ġ}} _ 1 \ math bf Ġ{ x ^ {\ prime Ġ}} _ 2 \ cd ots Ġ\ math bf Ġ{ x ^ {\ prime Ġ}} _ n ). $$ Ġ Ġ Ġ( E q . Ġ16 ) </s> The Ġresulting Ġfeature Ġsequence Ġis Ġconc aten ated Ġwith Ġthe Ġoutputs Ġof Ġthe Ġcommon ĠBi - L ST M , Ġand Ġfurther Ġbe Ġused Ġfor Ġworker Ġclassification . </s> CNN </s> Following , Ġwe Ġadd Ġa Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġmodule Ġbased Ġon Ġthe Ġconc aten ated Ġoutputs Ġof Ġthe Ġcommon ĠBi - L ST M Ġand Ġthe Ġlabel ĠBi - L ST M , Ġto Ġproduce Ġthe Ġfinal Ġfeatures Ġfor Ġworker Ġdiscrim inator . ĠA Ġconv olution al Ġoperator Ġwith Ġwindow Ġsize Ġ5 Ġis Ġused , Ġand Ġthen Ġmax Ġpool ing Ġstrategy Ġis Ġapplied Ġover Ġthe Ġconv olution Ġsequence Ġto Ġobtain Ġthe Ġfinal Ġfixed - dimensional Ġfeature Ġvector . ĠThe Ġwhole Ġprocess Ġcan Ġbe Ġdescribed Ġby Ġthe Ġfollowing Ġequations : </s> $$ \ begin { split } Ġ& \ math bf Ġ{ h } _ t ^ {\ text { worker }} Ġ= Ġ\ math bf Ġ{ h } _ t ^ {\ text { common }} Ġ\ opl us Ġ\ math bf Ġ{ h } _ t ^ {\ text { label }} Ġ\\ Ġ& \ math bf Ġ{\ t ilde { h }} _ t ^ {\ text { worker }} Ġ= Ġ\ tan h Ġ( \ math bf Ġ{ W } ^ {\ text { c nn }} [ \ math bf Ġ{ h } _{ t - 2 } ^ {\ text { worker } }, Ġ\ math bf Ġ{ h } _{ t - 1 } ^ {\ text { worker } }, Ġ\ cd ots Ġ, Ġ\ math bf Ġ{ h } _{ t + 2 } ^ {\ text { worker }} ]) Ġ\\ Ġ& \ math bf Ġ{ h } ^ {\ text { worker }} Ġ= Ġ\ text { max - pool ing } (\ math bf Ġ{\ t ilde { h }} _ 1 ^ {\ text { worker }} \ math bf Ġ{\ t ilde { h }} _ 2 ^ {\ text { worker }} Ġ\ cd ots Ġ\ math bf Ġ{\ t ilde { h }} _ n ^ {\ text { worker }} ) Ġ\\ Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ18 ) </s> where Ġ$ t Ġ\ in Ġ[ 1 , n ] $ Ġand Ġ$\ math bf Ġ{ W } ^ {\ text { c nn }} $ Ġis Ġone Ġmodel Ġparameter . ĠWe Ġexploit Ġzero Ġvector Ġto Ġpaddle Ġthe Ġout - of - index Ġvectors . </s> Output Ġand ĠPrediction </s> After Ġobtaining Ġthe Ġfinal Ġfeature Ġvector Ġfor Ġthe Ġworker Ġdiscrim inator , Ġwe Ġuse Ġit Ġto Ġcompute Ġthe Ġoutput Ġvector , Ġwhich Ġscores Ġall Ġthe Ġannotation Ġworkers . ĠThe Ġscore Ġfunction Ġis Ġdefined Ġby : </s> $$ \ math bf Ġ{ o } ^ {\ text { worker }} Ġ= Ġ\ math bf Ġ{ W } ^ {\ text { worker }} Ġ\ math bf Ġ{ h } ^ {\ text { worker } }, $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> where Ġ$\ math bf Ġ{ W } ^ {\ text { worker }} $ Ġis Ġone Ġmodel Ġparameter Ġand Ġthe Ġoutput Ġdimension Ġequals Ġthe Ġnumber Ġof Ġtotal Ġnon - ex pert Ġannot ators . ĠThe Ġprediction Ġis Ġto Ġfind Ġthe Ġworker Ġwhich Ġis Ġresponsible Ġfor Ġthis Ġannotation . </s> Ad vers arial ĠTraining </s> The Ġtraining Ġobjective Ġwith Ġadvers arial Ġneural Ġnetwork Ġis Ġdifferent Ġfrom Ġthe Ġbaseline Ġmodel , Ġas Ġit Ġincludes Ġthe Ġextra Ġworker Ġdiscrim inator . ĠThus Ġthe Ġnew Ġobjective Ġincludes Ġtwo Ġparts , Ġone Ġbeing Ġthe Ġnegative Ġlog - like lihood Ġfrom ĠN ER Ġwhich Ġis Ġthe Ġsame Ġas Ġthe Ġbaseline , Ġand Ġthe Ġother Ġbeing Ġthe Ġnegative Ġthe Ġlog - like lihood Ġfrom Ġthe Ġworker Ġdiscrim inator . </s> In Ġorder Ġto Ġobtain Ġthe Ġnegative Ġlog - like lihood Ġof Ġthe Ġworker Ġdiscrim inator , Ġwe Ġuse Ġsoft max Ġto Ġcompute Ġthe Ġprobability Ġof Ġthe Ġactual Ġworker Ġ$\ bar { z } $ Ġas Ġwell , Ġwhich Ġis Ġdefined Ġby : </s> $$ p (\ bar { z } | \ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y }} ) Ġ= Ġ\ frac {\ exp Ġ( \ math bf Ġ{ o } ^ {\ text { worker }} _{ \ bar { z }} ) }{ \ sum Ġ_ { z } Ġ\ exp Ġ( \ math bf Ġ{ o } ^ {\ text { worker }} _ z ) }, $$ Ġ Ġ Ġ( E q . Ġ22 ) </s> where Ġ$ z $ Ġshould Ġenumer ate Ġall Ġworkers . </s> Based Ġon Ġthe Ġabove Ġdefinition Ġof Ġprobability , Ġour Ġnew Ġobjective Ġis Ġdefined Ġas Ġfollows : </s> $$ \ begin { split } Ġ\ text { R } (\ The ta Ġ, Ġ\ The ta Ġ^ {\ prime Ġ}, Ġ\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y } }, Ġ\ bar { z }) Ġ& = Ġ\ text { loss } (\ The ta Ġ, Ġ\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y }} ) Ġ- Ġ\ text { loss } (\ The ta Ġ, Ġ\ The ta Ġ^ {\ prime Ġ}, Ġ\ math bf Ġ{ X }) Ġ\\ Ġ\ text { ~~~~ ~~ } Ġ& = Ġ- \ log Ġp (\ math bf Ġ{\ bar { y }} | \ math bf Ġ{ X }) Ġ+ Ġ\ log Ġp (\ bar { z } | \ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y }} ), Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġ$\ The ta Ġ$ Ġis Ġthe Ġset Ġof Ġall Ġmodel Ġparameters Ġrelated Ġto ĠN ER , Ġand Ġ$\ The ta Ġ^ {\ prime Ġ} $ Ġis Ġthe Ġset Ġof Ġthe Ġremaining Ġparameters Ġwhich Ġare Ġonly Ġrelated Ġto Ġthe Ġworker Ġdiscrim inator , Ġ$\ math bf Ġ{ X } $ Ġ, Ġ$\ math bf Ġ{\ bar { y }} $ Ġand Ġ$\ bar { z } $ Ġare Ġthe Ġinput Ġsentence , Ġthe Ġcrowd - annot ated ĠNE Ġlabels Ġand Ġthe Ġcorresponding Ġannot ator Ġfor Ġthis Ġannotation , Ġrespectively . ĠIt Ġis Ġworth Ġnoting Ġthat Ġthe Ġparameters Ġof Ġthe Ġcommon ĠBi - L ST M Ġare Ġincluded Ġin Ġthe Ġset Ġof Ġ$\ The ta Ġ$ Ġby Ġdefinition . </s> In Ġparticular , Ġour Ġgoal Ġis Ġnot Ġto Ġsimply Ġminimize Ġthe Ġnew Ġobjective . ĠActually , Ġwe Ġaim Ġfor Ġa Ġsaddle Ġpoint , Ġfinding Ġthe Ġparameters Ġ$\ The ta Ġ$ Ġand Ġ$\ The ta Ġ^ {\ prime Ġ} $ Ġsatisfying Ġthe Ġfollowing Ġconditions : </s> $$ \ begin { split } Ġ\ hat {\ The ta Ġ} Ġ& = Ġ\ math op Ġ{ arg ~ min } _{ \ The ta Ġ} \ text { R } (\ The ta Ġ, Ġ\ The ta Ġ^ {\ prime Ġ}, Ġ\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y } }, Ġ\ bar { z }) Ġ\\ Ġ\ hat {\ The ta Ġ} ^ {\ prime Ġ} Ġ& = Ġ\ math op Ġ{ arg ~ max } _{ \ The ta Ġ^ {\ prime Ġ} }\ text { R } (\ hat {\ The ta Ġ}, Ġ\ The ta Ġ^ {\ prime Ġ}, Ġ\ math bf Ġ{ X }, Ġ\ math bf Ġ{\ bar { y } }, Ġ\ bar { z }) Ġ\\ Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ24 ) </s> where Ġthe Ġfirst Ġequation Ġaims Ġto Ġfind Ġone Ġ$\ The ta Ġ$ Ġthat Ġminim izes Ġour Ġnew Ġobjective Ġ$\ text { R } (\ cd ot Ġ) $ Ġ, Ġand Ġthe Ġsecond Ġequation Ġaims Ġto Ġfind Ġone Ġ$\ The ta Ġ^ {\ prime Ġ} $ Ġmaximizing Ġthe Ġsame Ġobjective . </s> Int uitive ly , Ġthe Ġfirst Ġequation Ġof ĠFormula Ġ24 Ġtries Ġto Ġminimize Ġthe ĠN ER Ġloss , Ġbut Ġat Ġthe Ġsame Ġtime Ġmaximize Ġthe Ġworker Ġdiscrim inator Ġloss Ġby Ġthe Ġshared Ġparameters Ġof Ġthe Ġcommon ĠBi - L ST M . ĠThus Ġthe Ġresulting Ġfeatures Ġof Ġcommon ĠBi - L ST M Ġactually Ġattempt Ġto Ġhurt Ġthe Ġworker Ġdiscrim inator , Ġwhich Ġmakes Ġthese Ġfeatures Ġworker Ġindependent Ġsince Ġthey Ġare Ġunable Ġto Ġdistinguish Ġdifferent Ġworkers . ĠThe Ġsecond Ġequation Ġtries Ġto Ġminimize Ġthe Ġworker Ġdiscrim inator Ġloss Ġby Ġits Ġown Ġparameter Ġ$\ The ta Ġ^ {\ prime Ġ} $ Ġ. </s> We Ġuse Ġthe Ġstandard Ġback - prop ag ation Ġmethod Ġto Ġtrain Ġthe Ġmodel Ġparameters , Ġthe Ġsame Ġas Ġthe Ġbaseline Ġmodel . ĠIn Ġorder Ġto Ġincorporate Ġthe Ġterm Ġof Ġthe Ġarg max Ġpart Ġof ĠFormula Ġ24 Ġ, Ġwe Ġfollow Ġthe Ġprevious Ġwork Ġof Ġadvers arial Ġtraining ĠB IB REF 13 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 17 Ġ, Ġby Ġintroducing Ġa Ġgradient Ġreverse Ġlayer Ġbetween Ġthe Ġcommon ĠBi - L ST M Ġand Ġthe ĠCNN Ġmodule , Ġwhose Ġforward Ġdoes Ġnothing Ġbut Ġthe Ġbackward Ġsimply Ġneg ates Ġthe Ġgrad ients . </s> Data ĠSets </s> With Ġthe Ġpurpose Ġof Ġobtaining Ġevaluation Ġdatasets Ġfrom Ġcrowd Ġannot ators , Ġwe Ġcollect Ġthe Ġsentences Ġfrom Ġtwo Ġdomains : ĠDial og Ġand ĠE - commerce Ġdomain . ĠWe Ġhire Ġundergraduate Ġstudents Ġto Ġannot ate Ġthe Ġsentences . ĠThey Ġare Ġrequired Ġto Ġidentify Ġthe Ġpred efined Ġtypes Ġof Ġentities Ġin Ġthe Ġsentences . ĠTogether Ġwith Ġthe Ġguideline Ġdocument , Ġthe Ġannot ators Ġare Ġeducated Ġsome Ġtips Ġin Ġfifteen Ġminutes Ġand Ġalso Ġprovided Ġwith Ġ20 Ġexempl ifying Ġsentences . </s> Label ed ĠData : ĠDL - PS . ĠIn ĠDial og Ġdomain Ġ( DL ), Ġwe Ġcollect Ġraw Ġsentences Ġfrom Ġa Ġchat bot Ġapplication . ĠAnd Ġthen Ġwe Ġrandomly Ġselect Ġ20 K Ġsentences Ġas Ġour Ġpool Ġand Ġhire Ġ43 Ġstudents Ġto Ġannot ate Ġthe Ġsentences . ĠWe Ġask Ġthe Ġannot ators Ġto Ġlabel Ġtwo Ġtypes Ġof Ġentities : ĠPerson - Name Ġand ĠSong - Name . ĠThe Ġannot ators Ġlabel Ġthe Ġsentences Ġindependently . ĠIn Ġparticular , Ġeach Ġsentence Ġis Ġassigned Ġto Ġthree Ġannot ators Ġfor Ġthis Ġdata . ĠAlthough Ġthe Ġsetting Ġcan Ġbe Ġwasteful Ġof Ġlabor , Ġwe Ġcan Ġuse Ġthe Ġresulting Ġdataset Ġto Ġtest Ġseveral Ġwell - known Ġbas elines Ġsuch Ġas Ġmajority Ġvoting . </s> After Ġannotation , Ġwe Ġremove Ġsome Ġillegal Ġsentences Ġreported Ġby Ġthe Ġannot ators . ĠFinally , Ġwe Ġhave Ġ16 , 9 48 Ġsentences Ġannot ated Ġby Ġthe Ġstudents . ĠTable Ġ1 Ġshows Ġthe Ġinformation Ġof Ġannot ated Ġdata . ĠThe Ġaverage ĠKappa Ġvalue Ġamong Ġthe Ġannot ators Ġis Ġ0 . 60 33 , Ġindicating Ġthat Ġthe Ġcrowd Ġannot ators Ġhave Ġmoderate Ġagreement Ġon Ġidentifying Ġentities Ġon Ġthis Ġdata . </s> In Ġorder Ġto Ġevaluate Ġthe Ġsystem Ġperformances , Ġwe Ġcreate Ġa Ġset Ġof Ġcorpus Ġwith Ġgold Ġannotations . ĠCon crete ly , Ġwe Ġrandomly Ġselect Ġ1 , 000 Ġsentences Ġfrom Ġthe Ġfinal Ġdataset Ġand Ġlet Ġtwo Ġexperts Ġgenerate Ġthe Ġgold Ġannotations . ĠAmong Ġthem , Ġwe Ġuse Ġ300 Ġsentences Ġas Ġthe Ġdevelopment Ġset Ġand Ġthe Ġremaining Ġ700 Ġas Ġthe Ġtest Ġset . ĠThe Ġrest Ġsentences Ġwith Ġonly Ġstudent Ġannotations Ġare Ġused Ġas Ġthe Ġtraining Ġset . </s> Label ed Ġdata : ĠEC - MT Ġand ĠEC - U Q . ĠIn ĠE - commerce Ġdomain Ġ( EC ), Ġwe Ġcollect Ġraw Ġsentences Ġfrom Ġtwo Ġtypes Ġof Ġtexts : Ġone Ġis Ġtitles Ġof Ġmerchandise Ġentries Ġ( EC - MT ) Ġand Ġanother Ġis Ġuser Ġqueries Ġ( EC - U Q ). ĠThe Ġannot ators Ġlabel Ġfive Ġtypes Ġof Ġentities : ĠBrand , ĠProduct , ĠModel , ĠMaterial , Ġand ĠSpec ification . ĠThese Ġfive Ġtypes Ġof Ġentities Ġare Ġvery Ġimportant Ġfor ĠE - commerce Ġplatform , Ġfor Ġexample Ġbuilding Ġknowledge Ġgraph Ġof Ġmerch and ises . ĠFive Ġstudents Ġparticipate Ġthe Ġannotations Ġfor Ġthis Ġdomain Ġsince Ġthe Ġnumber Ġof Ġsentences Ġis Ġsmall . ĠWe Ġuse Ġthe Ġsimilar Ġstrategy Ġas ĠDL - PS Ġto Ġannot ate Ġthe Ġsentences , Ġexcept Ġthat Ġonly Ġtwo Ġannot ators Ġare Ġassigned Ġfor Ġeach Ġsentence , Ġbecause Ġwe Ġaim Ġto Ġtest Ġthe Ġsystem Ġperformances Ġunder Ġvery Ġsmall Ġdupl icated Ġannotations . </s> Finally , Ġwe Ġobtain Ġ2 , 337 Ġsentences Ġfor ĠEC - MT Ġand Ġ2 , 300 Ġfor ĠEC - U Q . ĠTable Ġ1 Ġshows Ġthe Ġinformation Ġof Ġannot ated Ġresults . ĠSimilarly , Ġwe Ġproduce Ġthe Ġdevelopment Ġand Ġtest Ġdatasets Ġfor Ġsystem Ġevaluation , Ġby Ġrandomly Ġselecting Ġ400 Ġsentences Ġand Ġletting Ġtwo Ġexperts Ġto Ġgenerate Ġthe Ġground truth Ġannotations . ĠAmong Ġthem , Ġwe Ġuse Ġ100 Ġsentences Ġas Ġthe Ġdevelopment Ġset Ġand Ġthe Ġremaining Ġ300 Ġas Ġthe Ġtest Ġset . ĠThe Ġrest Ġsentences Ġwith Ġonly Ġcrowds ourcing Ġannotations Ġare Ġused Ġas Ġthe Ġtraining Ġset . </s> Un label ed Ġdata . ĠThe Ġvector Ġrepresentations Ġof Ġcharacters Ġare Ġbasic Ġinputs Ġof Ġour Ġbaseline Ġand Ġproposed Ġmodels , Ġwhich Ġare Ġobtained Ġby Ġthe Ġlooking - up Ġtable Ġ$\ math bf Ġ{ E } ^ W $ Ġ. ĠAs Ġintroduced Ġbefore , Ġwe Ġcan Ġuse Ġpret rained Ġembed d ings Ġfrom Ġlarge - scale Ġraw Ġcorpus Ġto Ġinitialize Ġthe Ġtable . ĠIn Ġorder Ġto Ġpret rain Ġthe Ġcharacter Ġembed d ings , Ġwe Ġuse Ġone Ġlarge - scale Ġunl abel ed Ġdata Ġfrom Ġthe Ġuser - generated Ġcontent Ġin ĠInternet . ĠTotally , Ġwe Ġobtain Ġa Ġnumber Ġof Ġ5 M Ġsentences . ĠFinally , Ġwe Ġuse Ġthe Ġtool Ġword 2 vec Ġto Ġpret rain Ġthe Ġcharacter Ġembed d ings Ġbased Ġon Ġthe Ġunl abel ed Ġdataset Ġin Ġour Ġexperiments . </s> Settings </s> For Ġevaluation , Ġwe Ġuse Ġthe Ġentity - level Ġmetrics Ġof ĠPrecision Ġ( P ), ĠRecall Ġ( R ), Ġand Ġtheir ĠF 1 Ġvalue Ġin Ġour Ġexperiments , Ġtreating Ġone Ġtagged Ġentity Ġas Ġcorrect Ġonly Ġwhen Ġit Ġmatches Ġthe Ġgold Ġentity Ġexactly . </s> There Ġare Ġseveral Ġhyper - param eters Ġin Ġthe Ġbaseline ĠL ST M - CR F Ġand Ġour Ġfinal Ġmodels . ĠWe Ġset Ġthem Ġempir ically Ġby Ġthe Ġdevelopment Ġperformances . ĠCon crete ly , Ġwe Ġset Ġthe Ġdimension Ġsize Ġof Ġthe Ġcharacter Ġembed d ings Ġby Ġ100 , Ġthe Ġdimension Ġsize Ġof Ġthe ĠNE Ġlabel Ġembed d ings Ġby Ġ50 , Ġand Ġthe Ġdimension Ġsizes Ġof Ġall Ġthe Ġother Ġhidden Ġfeatures Ġby Ġ200 . </s> We Ġexploit Ġonline Ġtraining Ġwith Ġa Ġmini - batch Ġsize Ġ128 Ġto Ġlearn Ġmodel Ġparameters . ĠThe Ġmax - ep och Ġiteration Ġis Ġset Ġby Ġ200 , Ġand Ġthe Ġbest - ep och Ġmodel Ġis Ġchosen Ġaccording Ġto Ġthe Ġdevelopment Ġperformances . ĠWe Ġuse ĠR MS prop ĠB IB REF 28 Ġwith Ġa Ġlearning Ġrate Ġ$ 10 ^{ - 3 } $ Ġto Ġupdate Ġmodel Ġparameters , Ġand Ġuse Ġ$ l _ 2 $ Ġ- regular ization Ġby Ġa Ġparameter Ġ$ 10 ^{ - 5 } $ Ġ. ĠWe Ġadopt Ġthe Ġdrop out Ġtechnique Ġto Ġavoid Ġover fitting Ġby Ġa Ġdrop Ġvalue Ġof Ġ$ 0 . 2 $ Ġ. </s> Compar ison ĠSystems </s> The Ġproposed Ġapproach Ġ( hen ce forward Ġreferred Ġto Ġas ĠâĢ ľ AL C rowd âĢ Ŀ ) Ġis Ġcompared Ġwith Ġthe Ġfollowing Ġsystems : </s> CR F : ĠWe Ġuse Ġthe ĠCr f su ite Ġtool Ġto Ġtrain Ġa Ġmodel Ġon Ġthe Ġcrowds ourcing Ġlabeled Ġdata . ĠAs Ġfor Ġthe Ġfeature Ġsettings , Ġwe Ġuse Ġthe Ġsupervised Ġversion Ġof ĠB IB REF 0 Ġz hao 2008 un super vised . </s> CR F - VT : ĠWe Ġuse Ġthe Ġsame Ġsettings Ġof Ġthe ĠCR F Ġsystem , Ġexcept Ġthat Ġthe Ġtraining Ġdata Ġis Ġthe Ġvoted Ġversion , Ġwhose Ġground truth s Ġare Ġproduced Ġby Ġmajority Ġvoting Ġat Ġthe Ġcharacter Ġlevel Ġfor Ġeach Ġannot ated Ġsentence . </s> CR F - MA : ĠThe ĠCR F Ġmodel Ġproposed Ġby ĠB IB REF 3 Ġrod rig ues 2014 sequence , Ġwhich Ġuses Ġa Ġprior Ġdistribut ation Ġto Ġmodel Ġmultiple Ġcrowds ourcing Ġannot ators . ĠWe Ġuse Ġthe Ġsource Ġcode Ġprovided Ġby Ġthe Ġauthors . </s> L ST M - CR F : ĠOur Ġbaseline Ġsystem Ġtrained Ġon Ġthe Ġcrowds ourcing Ġlabeled Ġdata . </s> L ST M - CR F - VT : ĠOur Ġbaseline Ġsystem Ġtrained Ġon Ġthe Ġvoted Ġcorpus , Ġwhich Ġis Ġthe Ġsame Ġas ĠCR F - VT . </s> L ST M - C rowd : ĠThe ĠL ST M - CR F Ġmodel Ġwith Ġcrowd Ġannotation Ġlearning Ġproposed Ġby ĠB IB REF 4 Ġn guyen 2017 agg reg ating . ĠWe Ġuse Ġthe Ġsource Ġcode Ġprovided Ġby Ġthe Ġauthors . </s> The Ġfirst Ġthree Ġsystems Ġare Ġbased Ġon Ġthe ĠCR F Ġmodel Ġusing Ġtraditional Ġhand crafted Ġfeatures , Ġand Ġthe Ġlast Ġthree Ġsystems Ġare Ġbased Ġon Ġthe Ġneural ĠL ST M - CR F Ġmodel . ĠAmong Ġthem , ĠCR F - MA , ĠL ST M - C rowd Ġand Ġour Ġsystem Ġwith Ġadvers arial Ġlearning Ġ( AL C rowd ) Ġare Ġbased Ġon Ġcrowd Ġannotation Ġlearning Ġthat Ġdirectly Ġtrains Ġthe Ġmodel Ġon Ġthe Ġcrowd - annot ations . ĠFive Ġsystems , Ġincluding ĠCR F , ĠCR F - MA , ĠL ST M - CR F , ĠL ST M - C rowd , Ġand ĠA LC rowd , Ġare Ġtrained Ġon Ġthe Ġoriginal Ġversion Ġof Ġlabeled Ġdata , Ġwhile ĠCR F - VT Ġand ĠL ST M - CR F - VT Ġare Ġtrained Ġon Ġthe Ġvoted Ġversion . ĠSince ĠCR F - VT , ĠCR F - MA Ġand ĠL ST M - CR F - VT Ġall Ġrequire Ġground - truth Ġanswers Ġfor Ġeach Ġtraining Ġsentence , Ġwhich Ġare Ġdifficult Ġto Ġbe Ġproduced Ġwith Ġonly Ġtwo Ġannotations , Ġwe Ġdo Ġnot Ġapply Ġthe Ġthree Ġmodels Ġon Ġthe Ġtwo ĠEC Ġdatasets . </s> Main ĠResults </s> In Ġthis Ġsection , Ġwe Ġshow Ġthe Ġmodel Ġperformances Ġof Ġour Ġproposed Ġcrowds ourcing Ġlearning Ġsystem Ġ( AL C rowd ), Ġand Ġmeanwhile Ġcompare Ġit Ġwith Ġthe Ġother Ġsystems Ġmentioned Ġabove . ĠTable Ġ2 Ġshows Ġthe Ġexperimental Ġresults Ġon Ġthe ĠDL - PS Ġdatasets Ġand ĠTable Ġ3 Ġshows Ġthe Ġexperiment Ġresults Ġon Ġthe ĠEC - MT Ġand ĠEC - U Q Ġdatasets , Ġrespectively . </s> The Ġresults Ġof ĠCR F Ġand ĠL ST M - CR F Ġmean Ġthat Ġthe Ġcrowd Ġannotation Ġis Ġan Ġalternative Ġsolution Ġwith Ġlow Ġcost Ġfor Ġlabeling Ġdata Ġthat Ġcould Ġbe Ġused Ġfor Ġtraining Ġa ĠN ER Ġsystem Ġeven Ġthere Ġare Ġsome Ġinconsistencies . ĠCompared Ġwith ĠCR F , ĠL ST M - CR F Ġachieves Ġmuch Ġbetter Ġperformances Ġon Ġall Ġthe Ġthree Ġdata , Ġshowing Ġ+ 6 . 12 ĠF 1 Ġimprovement Ġon ĠDL - PS , Ġ+ 4 . 51 Ġon ĠEC - MT , Ġand Ġ+ 9 . 19 Ġon ĠEC - U Q . ĠThis Ġindicates Ġthat ĠL ST M - CR F Ġis Ġa Ġvery Ġstrong Ġbaseline Ġsystem , Ġdemonstrating Ġthe Ġeffectiveness Ġof Ġneural Ġnetwork . </s> Interestingly , Ġwhen Ġcompared Ġwith ĠCR F Ġand ĠL ST M - CR F , ĠCR F - VT Ġand ĠL ST M - CR F - VT Ġtrained Ġon Ġthe Ġvoted Ġversion Ġperform Ġworse Ġin Ġthe ĠDL - PS Ġdataset . ĠThis Ġtrend Ġis Ġalso Ġmentioned Ġin ĠB IB REF 4 Ġn guyen 2017 agg reg ating . ĠThis Ġfact Ġshows Ġthat Ġthe Ġmajority Ġvoting Ġmethod Ġmight Ġbe Ġunsu itable Ġfor Ġour Ġtask . ĠThere Ġare Ġtwo Ġpossible Ġreasons Ġaccounting Ġfor Ġthe Ġobservation . ĠOn Ġthe Ġone Ġhand , Ġsimple Ġcharacter - level Ġvoting Ġbased Ġon Ġthree Ġannotations Ġfor Ġeach Ġsentence Ġmay Ġbe Ġstill Ġnot Ġenough . ĠIn Ġthe ĠDL - PS Ġdataset , Ġeven Ġwith Ġonly Ġtwo Ġpred efined Ġentity Ġtypes , Ġone Ġcharacter Ġcan Ġhave Ġnine ĠNE Ġlabels . ĠThus Ġthe Ġmajority - v oting Ġmay Ġbe Ġincapable Ġof Ġhandling Ġsome Ġcases . ĠWhile Ġthe Ġcost Ġby Ġadding Ġmore Ġannotations Ġfor Ġeach Ġsentence Ġwould Ġbe Ġgreatly Ġincreased . ĠOn Ġthe Ġother Ġhand , Ġthe Ġlost Ġinformation Ġproduced Ġby Ġmajority - v oting Ġmay Ġbe Ġimportant , Ġat Ġleast Ġthe Ġambiguous Ġannotations Ġdenote Ġthat Ġthe Ġinput Ġsentence Ġis Ġdifficult Ġfor ĠN ER . ĠThe Ġnormal ĠCR F Ġand ĠL ST M - CR F Ġmodels Ġwithout Ġdiscard Ġany Ġannotations Ġcan Ġdifferentiate Ġthese Ġdifficult Ġcontexts Ġthrough Ġlearning . </s> Three Ġcrowd - ann otation Ġlearning Ġsystems Ġprovide Ġbetter Ġperformances Ġthan Ġtheir Ġcounterpart Ġsystems , Ġ( CR F - MA ĠVS ĠCR F ) Ġand Ġ( L ST M - C rowd / AL C rowd ĠVS ĠL ST M - CR F ). ĠCompared Ġwith Ġthe Ġstrong Ġbaseline ĠL ST M - CR F , ĠA LC rowd Ġshows Ġits Ġadvantage Ġwith Ġ+ 1 . 08 ĠF 1 Ġimprovements Ġon ĠDL - PS , Ġ+ 1 . 24 Ġon ĠEC - MT , Ġand Ġ+ 2 . 38 Ġon ĠEC - U Q , Ġrespectively . ĠThis Ġindicates Ġthat Ġadding Ġthe Ġcrowd - ann otation Ġlearning Ġis Ġquite Ġuseful Ġfor Ġbuilding ĠN ER Ġsystems . ĠIn Ġaddition , ĠA LC rowd Ġalso Ġoutper forms ĠL ST M - C rowd Ġon Ġall Ġthe Ġdatasets Ġconsistently , Ġdemonstrating Ġthe Ġhigh Ġeffectiveness Ġof ĠA LC rowd Ġin Ġextracting Ġworker Ġindependent Ġfeatures . ĠAmong Ġall Ġthe Ġsystems , ĠA LC rowd Ġperforms Ġthe Ġbest , Ġand Ġsignificantly Ġbetter Ġthan Ġall Ġthe Ġother Ġmodels Ġ( the Ġp - value Ġis Ġbelow Ġ$ 10 ^{ - 5 } $ Ġby Ġusing Ġt - test ). ĠThe Ġresults Ġindicate Ġthat Ġwith Ġthe Ġhelp Ġof Ġadvers arial Ġtraining , Ġour Ġsystem Ġcan Ġlearn Ġa Ġbetter Ġfeature Ġrepresentation Ġfrom Ġcrowd Ġannotation . </s> Discussion </s> Imp act Ġof ĠCharacter ĠEmb edd ings . ĠFirst , Ġwe Ġinvestigate Ġthe Ġeffect Ġof Ġthe Ġpret rained Ġcharacter Ġembed d ings Ġin Ġour Ġproposed Ġcrowds ourcing Ġlearning Ġmodel . ĠThe Ġcomparison Ġresults Ġare Ġshown Ġin ĠFigure Ġ2 Ġ, Ġwhere ĠRandom Ġrefers Ġto Ġthe Ġrandom Ġinitialized Ġcharacter Ġembed d ings , Ġand ĠPret rained Ġrefers Ġto Ġthe Ġembed d ings Ġpret rained Ġon Ġthe Ġunl abel ed Ġdata . ĠAccording Ġto Ġthe Ġresults , Ġwe Ġfind Ġthat Ġour Ġmodel Ġwith Ġthe Ġpret rained Ġembed d ings Ġsignificantly Ġoutper forms Ġthat Ġusing Ġthe Ġrandom Ġembed d ings , Ġdemonstrating Ġthat Ġthe Ġpret rained Ġembed d ings Ġsuccessfully Ġprovide Ġuseful Ġinformation . </s> Case ĠStudies . ĠSecond , Ġwe Ġpresent Ġseveral Ġcase Ġstudies Ġin Ġorder Ġto Ġstudy Ġthe Ġdifferences Ġbetween Ġour Ġbaseline Ġand Ġthe Ġworker Ġadvers arial Ġmodels . ĠWe Ġconduct Ġa Ġclosed Ġtest Ġon Ġthe Ġtraining Ġset , Ġthe Ġresults Ġof Ġwhich Ġcan Ġbe Ġregarded Ġas Ġmodifications Ġof Ġthe Ġtraining Ġcorpus , Ġsince Ġthere Ġexist Ġinconsistent Ġannotations Ġfor Ġeach Ġtraining Ġsentence Ġamong Ġthe Ġdifferent Ġworkers . ĠFigure Ġ3 Ġshows Ġthe Ġtwo Ġexamples Ġfrom Ġthe ĠDL - PS Ġdataset , Ġwhich Ġcompares Ġthe Ġoutputs Ġof Ġthe Ġbaseline Ġand Ġour Ġfinal Ġmodels , Ġas Ġwell Ġas Ġthe Ġmajority - v oting Ġstrategy . </s> In Ġthe Ġfirst Ġcase , Ġnone Ġof Ġthe Ġannotations Ġget Ġthe Ġcorrect ĠN ER Ġresult , Ġbut Ġour Ġproposed Ġmodel Ġcan Ġcapture Ġit . ĠThe Ġresult Ġof ĠL ST M - CR F Ġis Ġthe Ġsame Ġas Ġmajority - v oting . ĠIn Ġthe Ġsecond Ġexample , Ġthe Ġoutput Ġof Ġmajority - v oting Ġis Ġthe Ġworst , Ġwhich Ġcan Ġaccount Ġfor Ġthe Ġreason Ġwhy Ġthe Ġsame Ġmodel Ġtrained Ġon Ġthe Ġvoted Ġcorpus Ġperforms Ġso Ġbadly , Ġas Ġshown Ġin ĠTable Ġ2 Ġ. ĠThe Ġmodel Ġof ĠL ST M - CR F Ġfails Ġto Ġrecognize Ġthe Ġnamed Ġentity ĠâĢ ľ X ie x ie âĢ Ŀ Ġbecause Ġof Ġnot Ġtrusting Ġthe Ġsecond Ġannotation , Ġtreating Ġit Ġas Ġone Ġnoise Ġannotation . ĠOur Ġproposed Ġmodel Ġis Ġable Ġto Ġrecognize Ġit , Ġbecause Ġof Ġits Ġability Ġof Ġextracting Ġworker Ġindependent Ġfeatures . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġan Ġapproach Ġto Ġperforming Ġcrowd Ġannotation Ġlearning Ġbased Ġon Ġthe Ġidea Ġof Ġadvers arial Ġtraining Ġfor ĠChinese ĠNamed ĠEntity ĠRecogn ition Ġ( NER ). ĠIn Ġour Ġapproach , Ġwe Ġuse Ġa Ġcommon Ġand Ġprivate ĠBi - L ST Ms Ġfor Ġrepresenting Ġannot ator - generic Ġand Ġ- specific Ġinformation , Ġand Ġlearn Ġa Ġlabel ĠBi - L ST M Ġfrom Ġthe Ġcrowd - annot ated ĠNE Ġlabel Ġsequences . ĠFinally , Ġthe Ġproposed Ġapproach Ġadop ts Ġa ĠL ST M - CR F Ġmodel Ġto Ġperform Ġtagging . ĠIn Ġour Ġexperiments , Ġwe Ġcreate Ġtwo Ġdata Ġsets Ġfor ĠChinese ĠN ER Ġtasks Ġin Ġthe Ġdialog Ġand Ġe - commerce Ġdomains . ĠThe Ġexperimental Ġresults Ġshow Ġthat Ġthe Ġproposed Ġapproach Ġoutper forms Ġstrong Ġbaseline Ġsystems . </s> ĠA cknowled gments </s> This Ġwork Ġis Ġsupported Ġby Ġthe ĠNational ĠNatural ĠScience ĠFoundation Ġof ĠChina Ġ( Grant ĠNo . Ġ6 157 23 38 , Ġ6 15 25 205 , Ġand Ġ6 16 02 160 ). ĠThis Ġwork Ġis Ġalso Ġpartially Ġsupported Ġby Ġthe Ġjoint Ġresearch Ġproject Ġof ĠAlibaba Ġand ĠSo och ow ĠUniversity . ĠWen li ang Ġis Ġalso Ġpartially Ġsupported Ġby ĠCollabor ative ĠInnovation ĠCenter Ġof ĠNovel ĠSoftware ĠTechnology Ġand ĠIndustrial ization .