The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun. The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun.
EXTRACTIVE
A homographic and heterographic benchmark datasets by BIBREF9.
ABSTRACTIVE
We evaluate our model on two benchmark datasets BIBREF9 . The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun. The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun. We notice there is no standard splitting information provided for both datasets. Thus we apply 10-fold cross validation. To make direct comparisons with prior studies, following BIBREF4 , we accumulated the predictions for all ten folds and calculate the scores in the end.
We evaluate our model on two benchmark datasets BIBREF9 . The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun. The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun. We notice there is no standard splitting information provided for both datasets. Thus we apply 10-fold cross validation. To make direct comparisons with prior studies, following BIBREF4 , we accumulated the predictions for all ten folds and calculate the scores in the end.
1909.00175
Introduction </s> There Ġexists Ġa Ġclass Ġof Ġlanguage Ġconstruction Ġknown Ġas Ġpun Ġin Ġnatural Ġlanguage Ġtexts Ġand Ġutter ances , Ġwhere Ġa Ġcertain Ġword Ġor Ġother Ġlex ical Ġitems Ġare Ġused Ġto Ġexploit Ġtwo Ġor Ġmore Ġseparate Ġmeanings . ĠIt Ġhas Ġbeen Ġshown Ġthat Ġunderstanding Ġof Ġpun s Ġis Ġan Ġimportant Ġresearch Ġquestion Ġwith Ġvarious Ġreal - world Ġapplications , Ġsuch Ġas Ġhuman - computer Ġinteraction ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġand Ġmachine Ġtranslation ĠB IB REF 2 Ġ. ĠRecently , Ġmany Ġresearchers Ġshow Ġtheir Ġinterests Ġin Ġstudying Ġpun s , Ġlike Ġdetecting Ġpun Ġsentences ĠB IB REF 3 Ġ, Ġlocating Ġpun s Ġin Ġthe Ġtext ĠB IB REF 4 Ġ, Ġinterpreting Ġpun Ġsentences ĠB IB REF 5 Ġand Ġgenerating Ġsentences Ġcontaining Ġpun s ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠA Ġpun Ġis Ġa Ġword play Ġin Ġwhich Ġa Ġcertain Ġword Ġsuggests Ġtwo Ġor Ġmore Ġmeanings Ġby Ġexploiting Ġpoly se my , Ġhom ony my , Ġor Ġphon ological Ġsimilarity Ġto Ġanother Ġsign , Ġfor Ġan Ġintended Ġhumorous Ġor Ġrhetorical Ġeffect . ĠP uns Ġcan Ġbe Ġgenerally Ġcategorized Ġinto Ġtwo Ġgroups , Ġnamely Ġheter ographic Ġpun s Ġ( where Ġthe Ġpun Ġand Ġits Ġlatent Ġtarget Ġare Ġphon ologically Ġsimilar ) Ġand Ġhom ographic Ġpun s Ġ( where Ġthe Ġtwo Ġmeanings Ġof Ġthe Ġpun Ġreflect Ġits Ġtwo Ġdistinct Ġsenses ) ĠB IB REF 9 Ġ. ĠConsider Ġthe Ġfollowing Ġtwo Ġexamples : </s> The Ġfirst Ġpun ning Ġjoke Ġexploits Ġthe Ġsound Ġsimilarity Ġbetween Ġthe Ġword ĠâĢ ľ prop ane " Ġand Ġthe Ġlatent Ġtarget ĠâĢ ľ prof ane ", Ġwhich Ġcan Ġbe Ġcategorized Ġinto Ġthe Ġgroup Ġof Ġheter ographic Ġpun s . ĠAnother Ġcategor ization Ġof ĠEnglish Ġpun s Ġis Ġhom ographic Ġpun , Ġexempl ified Ġby Ġthe Ġsecond Ġinstance Ġleveraging Ġdistinct Ġsenses Ġof Ġthe Ġword ĠâĢ ľ g ut ". </s> P un Ġdetection Ġis Ġthe Ġtask Ġof Ġdetecting Ġwhether Ġthere Ġis Ġa Ġpun Ġresiding Ġin Ġthe Ġgiven Ġtext . ĠThe Ġgoal Ġof Ġpun Ġlocation Ġis Ġto Ġfind Ġthe Ġexact Ġword Ġappearing Ġin Ġthe Ġtext Ġthat Ġimplies Ġmore Ġthan Ġone Ġmeanings . ĠMost Ġprevious Ġwork Ġaddresses Ġsuch Ġtwo Ġtasks Ġseparately Ġand Ġdevelop Ġseparate Ġsystems ĠB IB REF 10 Ġ, ĠB IB REF 5 Ġ. ĠTypically , Ġa Ġsystem Ġfor Ġpun Ġdetection Ġis Ġbuilt Ġto Ġmake Ġa Ġbinary Ġprediction Ġon Ġwhether Ġa Ġsentence Ġcontains Ġa Ġpun Ġor Ġnot , Ġwhere Ġall Ġinstances Ġ( with Ġor Ġwithout Ġpun s ) Ġare Ġtaken Ġinto Ġaccount Ġduring Ġtraining . ĠFor Ġthe Ġtask Ġof Ġpun Ġlocation , Ġa Ġseparate Ġsystem Ġis Ġused Ġto Ġmake Ġa Ġsingle Ġprediction Ġas Ġto Ġwhich Ġword Ġin Ġthe Ġgiven Ġsentence Ġin Ġthe Ġtext Ġthat Ġtrigger Ġmore Ġthan Ġone Ġsemantic Ġinterpretations Ġof Ġthe Ġtext , Ġwhere Ġthe Ġtraining Ġdata Ġinvolves Ġonly Ġsentences Ġthat Ġcontain Ġa Ġpun . ĠTherefore , Ġif Ġone Ġis Ġinterested Ġin Ġsolving Ġboth Ġproblems Ġat Ġthe Ġsame Ġtime , Ġa Ġpipeline Ġapproach Ġthat Ġperforms Ġpun Ġdetection Ġfollowed Ġby Ġpun Ġlocation Ġcan Ġbe Ġused . </s> Compared Ġto Ġthe Ġpipeline Ġmethods , Ġjoint Ġlearning Ġhas Ġbeen Ġshown Ġeffective ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġsince Ġit Ġis Ġable Ġto Ġreduce Ġerror Ġpropagation Ġand Ġallows Ġinformation Ġexchange Ġbetween Ġtasks Ġwhich Ġis Ġpotentially Ġbeneficial Ġto Ġall Ġthe Ġtasks . ĠIn Ġthis Ġwork , Ġwe Ġdemonstrate Ġthat Ġthe Ġdetection Ġand Ġlocation Ġof Ġpun s Ġcan Ġbe Ġjointly Ġaddressed Ġby Ġa Ġsingle Ġmodel . ĠThe Ġpun Ġdetection Ġand Ġlocation Ġtasks Ġcan Ġbe Ġcombined Ġas Ġa Ġsequence Ġlabeling Ġproblem , Ġwhich Ġallows Ġus Ġto Ġjointly Ġdetect Ġand Ġlocate Ġa Ġpun Ġin Ġa Ġsentence Ġby Ġassigning Ġeach Ġword Ġa Ġtag . ĠSince Ġeach Ġcontext Ġcontains Ġa Ġmaximum Ġof Ġone Ġpun ĠB IB REF 9 Ġ, Ġwe Ġdesign Ġa Ġnovel Ġtagging Ġscheme Ġto Ġcapture Ġthis Ġstructural Ġconstraint . ĠStatistics Ġon Ġthe Ġcorpor a Ġalso Ġshow Ġthat Ġa Ġpun Ġtends Ġto Ġappear Ġin Ġthe Ġsecond Ġhalf Ġof Ġa Ġcontext . ĠTo Ġcapture Ġsuch Ġa Ġstructural Ġproperty , Ġwe Ġalso Ġincorporate Ġword Ġposition Ġknowledge Ġinto Ġour Ġstructured Ġprediction Ġmodel . ĠExper iments Ġon Ġthe Ġbenchmark Ġdatasets Ġshow Ġthat Ġdetection Ġand Ġlocation Ġtasks Ġcan Ġreinforce Ġeach Ġother , Ġleading Ġto Ġnew Ġstate - of - the - art Ġperformance Ġon Ġthese Ġtwo Ġtasks . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġwork Ġthat Ġperforms Ġjoint Ġdetection Ġand Ġlocation Ġof ĠEnglish Ġpun s Ġby Ġusing Ġa Ġsequence Ġlabeling Ġapproach . </s> Problem ĠDefinition </s> We Ġfirst Ġdesign Ġa Ġsimple Ġtagging Ġscheme Ġconsisting Ġof Ġtwo Ġtags Ġ{ ĠIN LINE FORM 0 Ġ} : </s> IN LINE FORM 0 Ġtag Ġmeans Ġthe Ġcurrent Ġword Ġis Ġnot Ġa Ġpun . </s> IN LINE FORM 0 Ġtag Ġmeans Ġthe Ġcurrent Ġword Ġis Ġa Ġpun . </s> If Ġthe Ġtag Ġsequence Ġof Ġa Ġsentence Ġcontains Ġa ĠIN LINE FORM 0 Ġtag , Ġthen Ġthe Ġtext Ġcontains Ġa Ġpun Ġand Ġthe Ġword Ġcorresponding Ġto ĠIN LINE FORM 1 Ġis Ġthe Ġpun . </s> The Ġcontexts Ġhave Ġthe Ġcharacteristic Ġthat Ġeach Ġcontext Ġcontains Ġa Ġmaximum Ġof Ġone Ġpun ĠB IB REF 9 Ġ. ĠIn Ġother Ġwords , Ġthere Ġexists Ġonly Ġone Ġpun Ġif Ġthe Ġgiven Ġsentence Ġis Ġdetected Ġas Ġthe Ġone Ġcontaining Ġa Ġpun . ĠOtherwise , Ġthere Ġis Ġno Ġpun Ġresiding Ġin Ġthe Ġtext . ĠTo Ġcapture Ġthis Ġinteresting Ġproperty , Ġwe Ġpropose Ġa Ġnew Ġtagging Ġscheme Ġconsisting Ġof Ġthree Ġtags , Ġnamely Ġ{ ĠIN LINE FORM 0 Ġ} . </s> IN LINE FORM 0 Ġtag Ġindicates Ġthat Ġthe Ġcurrent Ġword Ġappears Ġbefore Ġthe Ġpun Ġin Ġthe Ġgiven Ġcontext . </s> IN LINE FORM 0 Ġtag Ġhighlights Ġthe Ġcurrent Ġword Ġis Ġa Ġpun . </s> IN LINE FORM 0 Ġtag Ġindicates Ġthat Ġthe Ġcurrent Ġword Ġappears Ġafter Ġthe Ġpun . </s> We Ġempir ically Ġshow Ġthat Ġthe ĠIN LINE FORM 0 Ġscheme Ġcan Ġguarantee Ġthe Ġcontext Ġproperty Ġthat Ġthere Ġexists Ġa Ġmaximum Ġof Ġone Ġpun Ġresiding Ġin Ġthe Ġtext . </s> Given Ġa Ġcontext Ġfrom Ġthe Ġtraining Ġset , Ġwe Ġwill Ġbe Ġable Ġto Ġgenerate Ġits Ġcorresponding Ġgold Ġtag Ġsequence Ġusing Ġa Ġdeter ministic Ġprocedure . ĠUnder Ġthe Ġtwo Ġschemes , Ġif Ġa Ġsentence Ġdoes Ġnot Ġcontain Ġany Ġpun s , Ġall Ġwords Ġwill Ġbe Ġtagged Ġwith ĠIN LINE FORM 0 Ġor ĠIN LINE FORM 1 Ġ, Ġrespectively . ĠEx empl ified Ġby Ġthe Ġsecond Ġsentence ĠâĢ ľ Some Ġdiets Ġcause Ġa Ġgut Ġreaction ," Ġthe Ġpun Ġis Ġgiven Ġas ĠâĢ ľ g ut ." ĠThus , Ġunder Ġthe ĠIN LINE FORM 2 Ġscheme , Ġit Ġshould Ġbe Ġtagged Ġwith ĠIN LINE FORM 3 Ġ, Ġwhile Ġthe Ġwords Ġbefore Ġit Ġare Ġassigned Ġwith Ġthe Ġtag ĠIN LINE FORM 4 Ġand Ġwords Ġafter Ġit Ġare Ġwith ĠIN LINE FORM 5 Ġ, Ġas Ġillustrated Ġin ĠFigure ĠFIG REF 8 Ġ. ĠLikewise , Ġthe ĠIN LINE FORM 6 Ġscheme Ġtags Ġthe Ġword ĠâĢ ľ g ut " Ġwith ĠIN LINE FORM 7 Ġ, Ġwhile Ġother Ġwords Ġare Ġtagged Ġwith ĠIN LINE FORM 8 Ġ. ĠTherefore , Ġwe Ġcan Ġcombine Ġthe Ġpun Ġdetection Ġand Ġlocation Ġtasks Ġinto Ġone Ġproblem Ġwhich Ġcan Ġbe Ġsolved Ġby Ġthe Ġsequence Ġlabeling Ġapproach . </s> Model </s> Ne ural Ġmodels Ġhave Ġshown Ġtheir Ġeffectiveness Ġon Ġsequence Ġlabeling Ġtasks ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġadopt Ġthe Ġbid irection al ĠLong ĠShort ĠTerm ĠMemory Ġ( Bi L ST M ) ĠB IB REF 16 Ġnetworks Ġon Ġtop Ġof Ġthe ĠCond itional ĠRandom ĠFields ĠB IB REF 17 Ġ( CR F ) Ġarchitecture Ġto Ġmake Ġlabeling Ġdecisions , Ġwhich Ġis Ġone Ġof Ġthe Ġclassical Ġmodels Ġfor Ġsequence Ġlabeling . ĠOur Ġmodel Ġarchitecture Ġis Ġillustrated Ġin ĠFigure ĠFIG REF 8 Ġwith Ġa Ġrunning Ġexample . ĠGiven Ġa Ġcontext / sent ence ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġlength Ġof Ġthe Ġcontext , Ġwe Ġgenerate Ġthe Ġcorresponding Ġtag Ġsequence ĠIN LINE FORM 2 Ġbased Ġon Ġour Ġdesigned Ġtagging Ġschemes Ġand Ġthe Ġoriginal Ġannotations Ġfor Ġpun Ġdetection Ġand Ġlocation Ġprovided Ġby Ġthe Ġcorpor a . ĠOur Ġmodel Ġis Ġthen Ġtrained Ġon Ġpairs Ġof ĠIN LINE FORM 3 Ġ. </s> Input . ĠThe Ġcontexts Ġin Ġthe Ġpun Ġcorpus Ġhold Ġthe Ġproperty Ġthat Ġeach Ġpun Ġcontains Ġexactly Ġone Ġcontent Ġword , Ġwhich Ġcan Ġbe Ġeither Ġa Ġnoun , Ġa Ġverb , Ġan Ġadjective , Ġor Ġan Ġad verb . ĠTo Ġcapture Ġthis Ġcharacteristic , Ġwe Ġconsider Ġlex ical Ġfeatures Ġat Ġthe Ġcharacter Ġlevel . ĠSimilar Ġto Ġthe Ġwork Ġof ĠB IB REF 15 Ġ, Ġthe Ġcharacter Ġembed d ings Ġare Ġtrained Ġby Ġthe Ġcharacter - level ĠL ST M Ġnetworks Ġon Ġthe Ġun annot ated Ġinput Ġsequences . ĠNon linear Ġtransformations Ġare Ġthen Ġapplied Ġto Ġthe Ġcharacter Ġembed d ings Ġby Ġhighway Ġnetworks ĠB IB REF 18 Ġ, Ġwhich Ġmap Ġthe Ġcharacter - level Ġfeatures Ġinto Ġdifferent Ġsemantic Ġspaces . </s> We Ġalso Ġobserve Ġthat Ġa Ġpun Ġtends Ġto Ġappear Ġat Ġthe Ġend Ġof Ġa Ġsentence . ĠSpecifically , Ġbased Ġon Ġthe Ġstatistics , Ġwe Ġfound Ġthat Ġsentences Ġwith Ġa Ġpun Ġthat Ġlocate Ġat Ġthe Ġsecond Ġhalf Ġof Ġthe Ġtext Ġaccount Ġfor Ġaround Ġ88 % Ġand Ġ92 % Ġin Ġhom ographic Ġand Ġheter ographic Ġdatasets , Ġrespectively . ĠWe Ġthus Ġintroduce Ġa Ġbinary Ġfeature Ġthat Ġindicates Ġif Ġa Ġword Ġis Ġlocated Ġat Ġthe Ġfirst Ġor Ġthe Ġsecond Ġhalf Ġof Ġan Ġinput Ġsentence Ġto Ġcapture Ġsuch Ġpositional Ġinformation . ĠA Ġbinary Ġindicator Ġcan Ġbe Ġmapped Ġto Ġa Ġvector Ġrepresentation Ġusing Ġa Ġrandomly Ġinitialized Ġembed ding Ġtable ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġdirectly Ġadopt Ġthe Ġvalue Ġof Ġthe Ġbinary Ġindicator Ġas Ġpart Ġof Ġthe Ġinput . </s> The Ġconc aten ation Ġof Ġthe Ġtransformed Ġcharacter Ġembed d ings , Ġthe Ġpre - trained Ġword Ġembed d ings ĠB IB REF 21 Ġ, Ġand Ġthe Ġposition Ġindicators Ġare Ġtaken Ġas Ġinput Ġof Ġour Ġmodel . </s> T agging . ĠThe Ġinput Ġis Ġthen Ġfed Ġinto Ġa ĠBi L ST M Ġnetwork , Ġwhich Ġwill Ġbe Ġable Ġto Ġcapture Ġcontextual Ġinformation . ĠFor Ġa Ġtraining Ġinstance ĠIN LINE FORM 0 Ġ, Ġwe Ġsuppose Ġthe Ġoutput Ġby Ġthe Ġword - level ĠBi L ST M Ġis ĠIN LINE FORM 1 Ġ. ĠThe ĠCR F Ġlayer Ġis Ġadopted Ġto Ġcapture Ġlabel Ġdependencies Ġand Ġmake Ġfinal Ġtagging Ġdecisions Ġat Ġeach Ġposition , Ġwhich Ġhas Ġbeen Ġincluded Ġin Ġmany Ġstate - of - the - art Ġsequence Ġlabeling Ġmodels ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠThe Ġconditional Ġprobability Ġis Ġdefined Ġas : </s> where ĠIN LINE FORM 0 Ġis Ġa Ġset Ġof Ġall Ġpossible Ġlabel Ġsequences Ġconsisting Ġof Ġtags Ġfrom ĠIN LINE FORM 1 Ġ( or ĠIN LINE FORM 2 Ġ), ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġare Ġweight Ġand Ġbias Ġparameters Ġcorresponding Ġto Ġthe Ġlabel Ġpair ĠIN LINE FORM 5 Ġ. ĠDuring Ġtraining , Ġwe Ġminimize Ġthe Ġnegative Ġlog - like lihood Ġsummed Ġover Ġall Ġtraining Ġinstances : </s> where ĠIN LINE FORM 0 Ġrefers Ġto Ġthe ĠIN LINE FORM 1 Ġ- th Ġinstance Ġin Ġthe Ġtraining Ġset . ĠDuring Ġtesting , Ġwe Ġaim Ġto Ġfind Ġthe Ġoptimal Ġlabel Ġsequence Ġfor Ġa Ġnew Ġinput ĠIN LINE FORM 2 Ġ: </s> This Ġsearch Ġprocess Ġcan Ġbe Ġdone Ġefficiently Ġusing Ġthe ĠV iter bi Ġalgorithm . </s> Dat as ets Ġand ĠSettings </s> We Ġevaluate Ġour Ġmodel Ġon Ġtwo Ġbenchmark Ġdatasets ĠB IB REF 9 Ġ. ĠThe Ġhom ographic Ġdataset Ġcontains Ġ2 , 250 Ġcontexts , Ġ1 , 607 Ġof Ġwhich Ġcontain Ġa Ġpun . ĠThe Ġheter ographic Ġdataset Ġconsists Ġof Ġ1 , 780 Ġcontexts Ġwith Ġ1 , 271 Ġcontaining Ġa Ġpun . ĠWe Ġnotice Ġthere Ġis Ġno Ġstandard Ġsplitting Ġinformation Ġprovided Ġfor Ġboth Ġdatasets . ĠThus Ġwe Ġapply Ġ10 - fold Ġcross Ġvalidation . ĠTo Ġmake Ġdirect Ġcomparisons Ġwith Ġprior Ġstudies , Ġfollowing ĠB IB REF 4 Ġ, Ġwe Ġaccumulated Ġthe Ġpredictions Ġfor Ġall Ġten Ġfolds Ġand Ġcalculate Ġthe Ġscores Ġin Ġthe Ġend . </s> For Ġeach Ġfold , Ġwe Ġrandomly Ġselect Ġ10 % Ġof Ġthe Ġinstances Ġfrom Ġthe Ġtraining Ġset Ġfor Ġdevelopment . ĠWord Ġembed d ings Ġare Ġinitialized Ġwith Ġthe Ġ100 - dimensional ĠGl ove ĠB IB REF 21 Ġ. ĠThe Ġdimension Ġof Ġcharacter Ġembed d ings Ġis Ġ30 Ġand Ġthey Ġare Ġrandomly Ġinitialized , Ġwhich Ġcan Ġbe Ġfine Ġtuned Ġduring Ġtraining . ĠThe Ġpre - trained Ġword Ġembed d ings Ġare Ġnot Ġupdated Ġduring Ġtraining . ĠThe Ġdimensions Ġof Ġhidden Ġvectors Ġfor Ġboth Ġchar - level Ġand Ġword - level ĠL ST M Ġunits Ġare Ġset Ġto Ġ300 . ĠWe Ġadopt Ġsto ch astic Ġgradient Ġdescent Ġ( SG D ) ĠB IB REF 26 Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 015 . </s> For Ġthe Ġpun Ġdetection Ġtask , Ġif Ġthe Ġpredicted Ġtag Ġsequence Ġcontains Ġat Ġleast Ġone ĠIN LINE FORM 0 Ġtag , Ġwe Ġregard Ġthe Ġoutput Ġ( i . e ., Ġthe Ġprediction Ġof Ġour Ġpun Ġdetection Ġmodel ) Ġfor Ġthis Ġtask Ġas Ġtrue , Ġotherwise Ġfalse . ĠFor Ġthe Ġpun Ġlocation Ġtask , Ġa Ġpredicted Ġpun Ġis Ġregarded Ġas Ġcorrect Ġif Ġand Ġonly Ġif Ġit Ġis Ġlabeled Ġas Ġthe Ġgold Ġpun Ġin Ġthe Ġdataset . ĠAs Ġto Ġpun Ġlocation , Ġto Ġmake Ġfair Ġcomparisons Ġwith Ġprior Ġstudies , Ġwe Ġonly Ġconsider Ġthe Ġinstances Ġthat Ġare Ġlabeled Ġas Ġthe Ġones Ġcontaining Ġa Ġpun . ĠWe Ġreport Ġprecision , Ġrecall Ġand ĠIN LINE FORM 1 Ġscore Ġin ĠTable ĠT AB REF 11 Ġ. ĠA Ġlist Ġof Ġprior Ġworks Ġthat Ġdid Ġnot Ġemploy Ġjoint Ġlearning Ġare Ġalso Ġshown Ġin Ġthe Ġfirst Ġblock Ġof ĠTable ĠT AB REF 11 Ġ. </s> Results </s> We Ġalso Ġimplemented Ġa Ġbaseline Ġmodel Ġbased Ġon Ġconditional Ġrandom Ġfields Ġ( CR F ), Ġwhere Ġfeatures Ġlike ĠPOS Ġtags Ġproduced Ġby Ġthe ĠStanford ĠPOS Ġtag ger ĠB IB REF 27 Ġ, Ġn - gram s , Ġlabel Ġtransitions , Ġword Ġsuffix es Ġand Ġrelative Ġposition Ġto Ġthe Ġend Ġof Ġthe Ġtext Ġare Ġconsidered . ĠWe Ġcan Ġsee Ġthat Ġour Ġmodel Ġwith Ġthe ĠIN LINE FORM 0 Ġtagging Ġscheme Ġyields Ġnew Ġstate - of - the - art ĠIN LINE FORM 1 Ġscores Ġon Ġpun Ġdetection Ġand Ġcompetitive Ġresults Ġon Ġpun Ġlocation , Ġcompared Ġto Ġbas elines Ġthat Ġdo Ġnot Ġadopt Ġjoint Ġlearning Ġin Ġthe Ġfirst Ġblock . ĠFor Ġlocation Ġon Ġheter ographic Ġpun s , Ġour Ġmodel 's Ġperformance Ġis Ġslightly Ġlower Ġthan Ġthe Ġsystem Ġof ĠB IB REF 25 Ġ, Ġwhich Ġis Ġa Ġrule - based Ġloc ator . ĠCompared Ġto ĠCR F , Ġwe Ġcan Ġsee Ġthat Ġour Ġmodel , Ġeither Ġwith Ġthe ĠIN LINE FORM 2 Ġor Ġthe ĠIN LINE FORM 3 Ġscheme , Ġyields Ġsignificantly Ġhigher Ġrecall Ġon Ġboth Ġdetection Ġand Ġlocation Ġtasks , Ġwhile Ġthe Ġprec isions Ġare Ġrelatively Ġclose . ĠThis Ġdemonstrates Ġthe Ġeffectiveness Ġof ĠBi L ST M , Ġwhich Ġlearns Ġthe Ġcontextual Ġfeatures Ġof Ġgiven Ġtexts ĠâĢĵ Ġsuch Ġinformation Ġappears Ġto Ġbe Ġhelpful Ġin Ġrecalling Ġmore Ġpun s . </s> Compared Ġto Ġthe ĠIN LINE FORM 0 Ġscheme , Ġthe ĠIN LINE FORM 1 Ġtagging Ġscheme Ġis Ġable Ġto Ġyield Ġbetter Ġperformance Ġon Ġthese Ġtwo Ġtasks . ĠAfter Ġstudying Ġoutputs Ġfrom Ġthese Ġtwo Ġapproaches , Ġwe Ġfound Ġthat Ġone Ġleading Ġsource Ġof Ġerror Ġfor Ġthe ĠIN LINE FORM 2 Ġapproach Ġis Ġthat Ġthere Ġexist Ġmore Ġthan Ġone Ġwords Ġin Ġa Ġsingle Ġinstance Ġthat Ġare Ġassigned Ġwith Ġthe ĠIN LINE FORM 3 Ġtag . ĠHowever , Ġaccording Ġto Ġthe Ġdescription Ġof Ġpun Ġin ĠB IB REF 9 Ġ, Ġeach Ġcontext Ġcontains Ġa Ġmaximum Ġof Ġone Ġpun . ĠThus , Ġsuch Ġa Ġuseful Ġstructural Ġconstraint Ġis Ġnot Ġwell Ġcaptured Ġby Ġthe Ġsimple Ġapproach Ġbased Ġon Ġthe ĠIN LINE FORM 4 Ġtagging Ġscheme . ĠOn Ġthe Ġother Ġhand , Ġby Ġapplying Ġthe ĠIN LINE FORM 5 Ġtagging Ġscheme , Ġsuch Ġa Ġconstraint Ġis Ġproperly Ġcaptured Ġin Ġthe Ġmodel . ĠAs Ġa Ġresult , Ġthe Ġresults Ġfor Ġsuch Ġa Ġapproach Ġare Ġsignificantly Ġbetter Ġthan Ġthe Ġapproach Ġbased Ġon Ġthe ĠIN LINE FORM 6 Ġtagging Ġscheme , Ġas Ġwe Ġcan Ġobserve Ġfrom Ġthe Ġtable . ĠUnder Ġthe Ġsame Ġexperimental Ġsetup , Ġwe Ġalso Ġattempted Ġto Ġexclude Ġword Ġposition Ġfeatures . ĠResults Ġare Ġgiven Ġby ĠIN LINE FORM 7 Ġ- ĠIN LINE FORM 8 Ġ. ĠIt Ġis Ġexpected Ġthat Ġthe Ġperformance Ġof Ġpun Ġlocation Ġdrops , Ġsince Ġsuch Ġposition Ġfeatures Ġare Ġable Ġto Ġcapture Ġthe Ġinteresting Ġproperty Ġthat Ġa Ġpun Ġtends Ġto Ġappear Ġin Ġthe Ġsecond Ġhalf Ġof Ġa Ġsentence . ĠWhile Ġsuch Ġknowledge Ġis Ġhelpful Ġfor Ġthe Ġlocation Ġtask , Ġinterestingly , Ġa Ġmodel Ġwithout Ġposition Ġknowledge Ġyields Ġimproved Ġperformance Ġon Ġthe Ġpun Ġdetection Ġtask . ĠOne Ġpossible Ġreason Ġis Ġthat Ġdetecting Ġwhether Ġa Ġsentence Ġcontains Ġa Ġpun Ġis Ġnot Ġconcerned Ġwith Ġsuch Ġword Ġposition Ġinformation . </s> Additionally , Ġwe Ġconduct Ġexperiments Ġover Ġsentences Ġcontaining Ġa Ġpun Ġonly , Ġnamely Ġ1 , 607 Ġand Ġ1 , 271 Ġinstances Ġfrom Ġhom ographic Ġand Ġheter ographic Ġpun Ġcorpor a Ġseparately . ĠIt Ġcan Ġbe Ġregarded Ġas Ġa ĠâĢ ľ p ip eline âĢ Ŀ Ġmethod Ġwhere Ġthe Ġclass ifier Ġfor Ġpun Ġdetection Ġis Ġregarded Ġas Ġperfect . ĠFollowing Ġthe Ġprior Ġwork Ġof ĠB IB REF 4 Ġ, Ġwe Ġapply Ġ10 - fold Ġcross Ġvalidation . ĠSince Ġwe Ġare Ġgiven Ġthat Ġall Ġinput Ġsentences Ġcontain Ġa Ġpun , Ġwe Ġonly Ġreport Ġaccumulated Ġresults Ġon Ġpun Ġlocation , Ġden oted Ġas ĠPipeline Ġin ĠTable ĠT AB REF 11 Ġ. ĠCompared Ġwith Ġour Ġapproaches , Ġthe Ġperformance Ġof Ġsuch Ġan Ġapproach Ġdrops Ġsignificantly . ĠOn Ġthe Ġother Ġhand , Ġsuch Ġa Ġfact Ġdemonstrates Ġthat Ġthe Ġtwo Ġtask , Ġdetection Ġand Ġlocation Ġof Ġpun s , Ġcan Ġreinforce Ġeach Ġother . ĠThese Ġfigures Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġour Ġsequence Ġlabeling Ġmethod Ġto Ġdetect Ġand Ġlocate ĠEnglish Ġpun s Ġin Ġa Ġjoint Ġmanner . </s> Error ĠAnalysis </s> We Ġstudied Ġthe Ġoutputs Ġfrom Ġour Ġsystem Ġand Ġmake Ġsome Ġerror Ġanalysis . ĠWe Ġfound Ġthe Ġerrors Ġcan Ġbe Ġbroadly Ġcategorized Ġinto Ġseveral Ġtypes , Ġand Ġwe Ġelaborate Ġthem Ġhere . Ġ1 ) ĠLow Ġword Ġcoverage : Ġsince Ġthe Ġcorpor a Ġare Ġrelatively Ġsmall , Ġthere Ġexist Ġmany Ġunseen Ġwords Ġin Ġthe Ġtest Ġset . ĠLearning Ġthe Ġrepresentations Ġof Ġsuch Ġunseen Ġwords Ġis Ġchallenging , Ġwhich Ġaffects Ġthe Ġmodel 's Ġperformance . ĠSuch Ġerrors Ġcontribute Ġaround Ġ40 % Ġof Ġthe Ġtotal Ġerrors Ġmade Ġby Ġour Ġsystem . Ġ2 ) ĠDetection Ġerrors : Ġwe Ġfound Ġmany Ġerrors Ġare Ġdue Ġto Ġthe Ġmodel 's Ġinability Ġto Ġmake Ġcorrect Ġpun Ġdetection . ĠSuch Ġinability Ġharms Ġboth Ġpun Ġdetection Ġand Ġpun Ġlocation . ĠAlthough Ġour Ġapproach Ġbased Ġon Ġthe ĠIN LINE FORM 0 Ġtagging Ġscheme Ġyields Ġrelatively Ġhigher Ġscores Ġon Ġthe Ġdetection Ġtask , Ġwe Ġstill Ġfound Ġthat Ġ40 % Ġof Ġthe Ġincorrectly Ġpredicted Ġinstances Ġfall Ġinto Ġthis Ġgroup . Ġ3 ) ĠShort Ġsentences : Ġwe Ġfound Ġit Ġwas Ġchallenging Ġfor Ġour Ġmodel Ġto Ġmake Ġcorrect Ġpredictions Ġwhen Ġthe Ġgiven Ġtext Ġis Ġshort . ĠConsider Ġthe Ġexample ĠâĢ ľ Super gl ue ! ĠTom Ġrejo ined ," Ġhere Ġthe Ġword Ġrejo ined Ġis Ġthe Ġcorresponding Ġpun . ĠHowever , Ġit Ġwould Ġbe Ġchallenging Ġto Ġfigure Ġout Ġthe Ġpun Ġwith Ġsuch Ġlimited Ġcontextual Ġinformation . </s> Related ĠWork </s> Most Ġexisting Ġsystems Ġaddress Ġpun Ġdetection Ġand Ġlocation Ġseparately . ĠB IB REF 22 Ġapplied Ġword Ġsense Ġknowledge Ġto Ġconduct Ġpun Ġdetection . ĠB IB REF 24 Ġtrained Ġa Ġbid irection al ĠR NN Ġclass ifier Ġfor Ġdetecting Ġhom ographic Ġpun s . ĠNext , Ġa Ġknowledge - based Ġapproach Ġis Ġadopted Ġto Ġfind Ġthe Ġexact Ġpun . ĠSuch Ġa Ġsystem Ġis Ġnot Ġapplicable Ġto Ġheter ographic Ġpun s . ĠB IB REF 28 Ġapplied ĠGoogle Ġn - gram Ġand Ġword 2 vec Ġto Ġmake Ġdecisions . ĠThe Ġphon etic Ġdistance Ġvia Ġthe ĠCM U ĠP ron ouncing ĠDictionary Ġis Ġcomputed Ġto Ġdetect Ġheter ographic Ġpun s . ĠB IB REF 10 Ġused Ġthe Ġhidden ĠMark ov Ġmodel Ġand Ġa Ġcycl ic Ġdependency Ġnetwork Ġwith Ġrich Ġfeatures Ġto Ġdetect Ġand Ġlocate Ġpun s . ĠB IB REF 23 Ġused Ġa Ġsupervised Ġapproach Ġto Ġpun Ġdetection Ġand Ġa Ġweak ly Ġsupervised Ġapproach Ġto Ġpun Ġlocation Ġbased Ġon Ġthe Ġposition Ġwithin Ġthe Ġcontext Ġand Ġpart Ġof Ġspeech Ġfeatures . ĠB IB REF 25 Ġproposed Ġa Ġrule - based Ġsystem Ġfor Ġpun Ġlocation Ġthat Ġscores Ġcandidate Ġwords Ġaccording Ġto Ġeleven Ġsimple Ġhe ur istics . ĠTwo Ġsystems Ġare Ġdeveloped Ġto Ġconduct Ġdetection Ġand Ġlocation Ġseparately Ġin Ġthe Ġsystem Ġknown Ġas ĠUW AV ĠB IB REF 3 Ġ. ĠThe Ġpun Ġdetector Ġcombines Ġpredictions Ġfrom Ġthree Ġclass ifiers . ĠThe Ġpun Ġloc ator Ġconsiders Ġword 2 vec Ġsimilarity Ġbetween Ġevery Ġpair Ġof Ġwords Ġin Ġthe Ġcontext Ġand Ġposition Ġto Ġpinpoint Ġthe Ġpun . ĠThe Ġstate - of - the - art Ġsystem Ġfor Ġhom ographic Ġpun Ġlocation Ġis Ġa Ġneural Ġmethod ĠB IB REF 4 Ġ, Ġwhere Ġthe Ġword Ġsenses Ġare Ġincorporated Ġinto Ġa Ġbid irection al ĠL ST M Ġmodel . ĠThis Ġmethod Ġonly Ġsupports Ġthe Ġpun Ġlocation Ġtask Ġon Ġhom ographic Ġpun s . ĠAnother Ġline Ġof Ġresearch Ġefforts Ġrelated Ġto Ġthis Ġwork Ġis Ġsequence Ġlabeling , Ġsuch Ġas ĠPOS Ġtagging , Ġchunk ing , Ġword Ġsegment ation Ġand ĠN ER . ĠThe Ġneural Ġmethods Ġhave Ġshown Ġtheir Ġeffectiveness Ġin Ġthis Ġtask , Ġsuch Ġas ĠBi L ST M - CNN ĠB IB REF 13 Ġ, ĠGR NN ĠB IB REF 29 Ġ, ĠL ST M - CR F ĠB IB REF 30 Ġ, ĠL ST M - CNN - CR F ĠB IB REF 14 Ġ, ĠLM - L ST M - CR F ĠB IB REF 15 Ġ. </s> In Ġthis Ġwork , Ġwe Ġcombine Ġpun Ġdetection Ġand Ġlocation Ġtasks Ġas Ġa Ġsingle Ġsequence Ġlabeling Ġproblem . ĠInspired Ġby Ġthe Ġwork Ġof ĠB IB REF 15 Ġ, Ġwe Ġalso Ġadopt Ġa ĠL ST M - CR F Ġwith Ġcharacter Ġembed d ings Ġto Ġmake Ġlabeling Ġdecisions . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġto Ġperform Ġpun Ġdetection Ġand Ġlocation Ġtasks Ġin Ġa Ġjoint Ġmanner Ġfrom Ġa Ġsequence Ġlabeling Ġperspective . ĠWe Ġobserve Ġthat Ġeach Ġtext Ġin Ġour Ġcorpor a Ġcontains Ġa Ġmaximum Ġof Ġone Ġpun . ĠHence , Ġwe Ġdesign Ġa Ġnovel Ġtagging Ġscheme Ġto Ġincorporate Ġsuch Ġa Ġconstraint . ĠSuch Ġa Ġscheme Ġguarantees Ġthat Ġthere Ġis Ġa Ġmaximum Ġof Ġone Ġword Ġthat Ġwill Ġbe Ġtagged Ġas Ġa Ġpun Ġduring Ġthe Ġtesting Ġphase . ĠWe Ġalso Ġfound Ġthe Ġinteresting Ġstructural Ġproperty Ġsuch Ġas Ġthe Ġfact Ġthat Ġmost Ġpun s Ġtend Ġto Ġappear Ġat Ġthe Ġsecond Ġhalf Ġof Ġthe Ġsentences Ġcan Ġbe Ġhelpful Ġfor Ġsuch Ġa Ġtask , Ġbut Ġwas Ġnot Ġexplored Ġin Ġprevious Ġworks . ĠFurthermore , Ġunlike Ġmany Ġprevious Ġapproaches , Ġour Ġapproach , Ġthough Ġsimple , Ġis Ġgenerally Ġapplicable Ġto Ġboth Ġheter ographic Ġand Ġhom ographic Ġpun s . ĠEm pir ical Ġresults Ġon Ġthe Ġbenchmark Ġdatasets Ġprove Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġapproach Ġthat Ġthe Ġtwo Ġtasks Ġof Ġpun Ġdetection Ġand Ġlocation Ġcan Ġbe Ġaddressed Ġby Ġa Ġsingle Ġmodel Ġfrom Ġa Ġsequence Ġlabeling Ġperspective . </s> Future Ġresearch Ġincludes Ġthe Ġinvestigations Ġon Ġhow Ġto Ġmake Ġuse Ġof Ġricher Ġsemantic Ġand Ġlinguistic Ġinformation Ġfor Ġdetection Ġand Ġlocation Ġof Ġpun s . ĠResearch Ġon Ġpun s Ġfor Ġother Ġlanguages Ġsuch Ġas ĠChinese Ġis Ġstill Ġunder - expl ored , Ġwhich Ġcould Ġalso Ġbe Ġan Ġinteresting Ġdirection Ġfor Ġour Ġfuture Ġstudies . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġthree Ġanonymous Ġreviewers Ġfor Ġtheir Ġthoughtful Ġand Ġconstructive Ġcomments . ĠThis Ġwork Ġis Ġsupported Ġby ĠSingapore ĠMinistry Ġof ĠEducation ĠAcademic ĠResearch ĠFund Ġ( Ac RF ) ĠTier Ġ2 ĠProject ĠMO E 2017 - T 2 - 1 - 156 , Ġand Ġis Ġpartially Ġsupported Ġby ĠS UT D Ġproject ĠP IE - S GP - AI - 2018 - 01 .
Introduction There exists a class of language construction known as pun in natural language texts and utterances, where a certain word or other lexical items are used to exploit two or more separate meanings. It has been shown that understanding of puns is an important research question with various real-world applications, such as human-computer interaction BIBREF0 , BIBREF1 and machine translation BIBREF2 . Recently, many researchers show their interests in studying puns, like detecting pun sentences BIBREF3 , locating puns in the text BIBREF4 , interpreting pun sentences BIBREF5 and generating sentences containing puns BIBREF6 , BIBREF7 , BIBREF8 . A pun is a wordplay in which a certain word suggests two or more meanings by exploiting polysemy, homonymy, or phonological similarity to another sign, for an intended humorous or rhetorical effect. Puns can be generally categorized into two groups, namely heterographic puns (where the pun and its latent target are phonologically similar) and homographic puns (where the two meanings of the pun reflect its two distinct senses) BIBREF9 . Consider the following two examples: The first punning joke exploits the sound similarity between the word “propane" and the latent target “profane", which can be categorized into the group of heterographic puns. Another categorization of English puns is homographic pun, exemplified by the second instance leveraging distinct senses of the word “gut". Pun detection is the task of detecting whether there is a pun residing in the given text. The goal of pun location is to find the exact word appearing in the text that implies more than one meanings. Most previous work addresses such two tasks separately and develop separate systems BIBREF10 , BIBREF5 . Typically, a system for pun detection is built to make a binary prediction on whether a sentence contains a pun or not, where all instances (with or without puns) are taken into account during training. For the task of pun location, a separate system is used to make a single prediction as to which word in the given sentence in the text that trigger more than one semantic interpretations of the text, where the training data involves only sentences that contain a pun. Therefore, if one is interested in solving both problems at the same time, a pipeline approach that performs pun detection followed by pun location can be used. Compared to the pipeline methods, joint learning has been shown effective BIBREF11 , BIBREF12 since it is able to reduce error propagation and allows information exchange between tasks which is potentially beneficial to all the tasks. In this work, we demonstrate that the detection and location of puns can be jointly addressed by a single model. The pun detection and location tasks can be combined as a sequence labeling problem, which allows us to jointly detect and locate a pun in a sentence by assigning each word a tag. Since each context contains a maximum of one pun BIBREF9 , we design a novel tagging scheme to capture this structural constraint. Statistics on the corpora also show that a pun tends to appear in the second half of a context. To capture such a structural property, we also incorporate word position knowledge into our structured prediction model. Experiments on the benchmark datasets show that detection and location tasks can reinforce each other, leading to new state-of-the-art performance on these two tasks. To the best of our knowledge, this is the first work that performs joint detection and location of English puns by using a sequence labeling approach. Problem Definition We first design a simple tagging scheme consisting of two tags { INLINEFORM0 }: INLINEFORM0 tag means the current word is not a pun. INLINEFORM0 tag means the current word is a pun. If the tag sequence of a sentence contains a INLINEFORM0 tag, then the text contains a pun and the word corresponding to INLINEFORM1 is the pun. The contexts have the characteristic that each context contains a maximum of one pun BIBREF9 . In other words, there exists only one pun if the given sentence is detected as the one containing a pun. Otherwise, there is no pun residing in the text. To capture this interesting property, we propose a new tagging scheme consisting of three tags, namely { INLINEFORM0 }. INLINEFORM0 tag indicates that the current word appears before the pun in the given context. INLINEFORM0 tag highlights the current word is a pun. INLINEFORM0 tag indicates that the current word appears after the pun. We empirically show that the INLINEFORM0 scheme can guarantee the context property that there exists a maximum of one pun residing in the text. Given a context from the training set, we will be able to generate its corresponding gold tag sequence using a deterministic procedure. Under the two schemes, if a sentence does not contain any puns, all words will be tagged with INLINEFORM0 or INLINEFORM1 , respectively. Exemplified by the second sentence “Some diets cause a gut reaction," the pun is given as “gut." Thus, under the INLINEFORM2 scheme, it should be tagged with INLINEFORM3 , while the words before it are assigned with the tag INLINEFORM4 and words after it are with INLINEFORM5 , as illustrated in Figure FIGREF8 . Likewise, the INLINEFORM6 scheme tags the word “gut" with INLINEFORM7 , while other words are tagged with INLINEFORM8 . Therefore, we can combine the pun detection and location tasks into one problem which can be solved by the sequence labeling approach. Model Neural models have shown their effectiveness on sequence labeling tasks BIBREF13 , BIBREF14 , BIBREF15 . In this work, we adopt the bidirectional Long Short Term Memory (BiLSTM) BIBREF16 networks on top of the Conditional Random Fields BIBREF17 (CRF) architecture to make labeling decisions, which is one of the classical models for sequence labeling. Our model architecture is illustrated in Figure FIGREF8 with a running example. Given a context/sentence INLINEFORM0 where INLINEFORM1 is the length of the context, we generate the corresponding tag sequence INLINEFORM2 based on our designed tagging schemes and the original annotations for pun detection and location provided by the corpora. Our model is then trained on pairs of INLINEFORM3 . Input. The contexts in the pun corpus hold the property that each pun contains exactly one content word, which can be either a noun, a verb, an adjective, or an adverb. To capture this characteristic, we consider lexical features at the character level. Similar to the work of BIBREF15 , the character embeddings are trained by the character-level LSTM networks on the unannotated input sequences. Nonlinear transformations are then applied to the character embeddings by highway networks BIBREF18 , which map the character-level features into different semantic spaces. We also observe that a pun tends to appear at the end of a sentence. Specifically, based on the statistics, we found that sentences with a pun that locate at the second half of the text account for around 88% and 92% in homographic and heterographic datasets, respectively. We thus introduce a binary feature that indicates if a word is located at the first or the second half of an input sentence to capture such positional information. A binary indicator can be mapped to a vector representation using a randomly initialized embedding table BIBREF19 , BIBREF20 . In this work, we directly adopt the value of the binary indicator as part of the input. The concatenation of the transformed character embeddings, the pre-trained word embeddings BIBREF21 , and the position indicators are taken as input of our model. Tagging. The input is then fed into a BiLSTM network, which will be able to capture contextual information. For a training instance INLINEFORM0 , we suppose the output by the word-level BiLSTM is INLINEFORM1 . The CRF layer is adopted to capture label dependencies and make final tagging decisions at each position, which has been included in many state-of-the-art sequence labeling models BIBREF14 , BIBREF15 . The conditional probability is defined as: where INLINEFORM0 is a set of all possible label sequences consisting of tags from INLINEFORM1 (or INLINEFORM2 ), INLINEFORM3 and INLINEFORM4 are weight and bias parameters corresponding to the label pair INLINEFORM5 . During training, we minimize the negative log-likelihood summed over all training instances: where INLINEFORM0 refers to the INLINEFORM1 -th instance in the training set. During testing, we aim to find the optimal label sequence for a new input INLINEFORM2 : This search process can be done efficiently using the Viterbi algorithm. Datasets and Settings We evaluate our model on two benchmark datasets BIBREF9 . The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun. The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun. We notice there is no standard splitting information provided for both datasets. Thus we apply 10-fold cross validation. To make direct comparisons with prior studies, following BIBREF4 , we accumulated the predictions for all ten folds and calculate the scores in the end. For each fold, we randomly select 10% of the instances from the training set for development. Word embeddings are initialized with the 100-dimensional Glove BIBREF21 . The dimension of character embeddings is 30 and they are randomly initialized, which can be fine tuned during training. The pre-trained word embeddings are not updated during training. The dimensions of hidden vectors for both char-level and word-level LSTM units are set to 300. We adopt stochastic gradient descent (SGD) BIBREF26 with a learning rate of 0.015. For the pun detection task, if the predicted tag sequence contains at least one INLINEFORM0 tag, we regard the output (i.e., the prediction of our pun detection model) for this task as true, otherwise false. For the pun location task, a predicted pun is regarded as correct if and only if it is labeled as the gold pun in the dataset. As to pun location, to make fair comparisons with prior studies, we only consider the instances that are labeled as the ones containing a pun. We report precision, recall and INLINEFORM1 score in Table TABREF11 . A list of prior works that did not employ joint learning are also shown in the first block of Table TABREF11 . Results We also implemented a baseline model based on conditional random fields (CRF), where features like POS tags produced by the Stanford POS tagger BIBREF27 , n-grams, label transitions, word suffixes and relative position to the end of the text are considered. We can see that our model with the INLINEFORM0 tagging scheme yields new state-of-the-art INLINEFORM1 scores on pun detection and competitive results on pun location, compared to baselines that do not adopt joint learning in the first block. For location on heterographic puns, our model's performance is slightly lower than the system of BIBREF25 , which is a rule-based locator. Compared to CRF, we can see that our model, either with the INLINEFORM2 or the INLINEFORM3 scheme, yields significantly higher recall on both detection and location tasks, while the precisions are relatively close. This demonstrates the effectiveness of BiLSTM, which learns the contextual features of given texts – such information appears to be helpful in recalling more puns. Compared to the INLINEFORM0 scheme, the INLINEFORM1 tagging scheme is able to yield better performance on these two tasks. After studying outputs from these two approaches, we found that one leading source of error for the INLINEFORM2 approach is that there exist more than one words in a single instance that are assigned with the INLINEFORM3 tag. However, according to the description of pun in BIBREF9 , each context contains a maximum of one pun. Thus, such a useful structural constraint is not well captured by the simple approach based on the INLINEFORM4 tagging scheme. On the other hand, by applying the INLINEFORM5 tagging scheme, such a constraint is properly captured in the model. As a result, the results for such a approach are significantly better than the approach based on the INLINEFORM6 tagging scheme, as we can observe from the table. Under the same experimental setup, we also attempted to exclude word position features. Results are given by INLINEFORM7 - INLINEFORM8 . It is expected that the performance of pun location drops, since such position features are able to capture the interesting property that a pun tends to appear in the second half of a sentence. While such knowledge is helpful for the location task, interestingly, a model without position knowledge yields improved performance on the pun detection task. One possible reason is that detecting whether a sentence contains a pun is not concerned with such word position information. Additionally, we conduct experiments over sentences containing a pun only, namely 1,607 and 1,271 instances from homographic and heterographic pun corpora separately. It can be regarded as a “pipeline” method where the classifier for pun detection is regarded as perfect. Following the prior work of BIBREF4 , we apply 10-fold cross validation. Since we are given that all input sentences contain a pun, we only report accumulated results on pun location, denoted as Pipeline in Table TABREF11 . Compared with our approaches, the performance of such an approach drops significantly. On the other hand, such a fact demonstrates that the two task, detection and location of puns, can reinforce each other. These figures demonstrate the effectiveness of our sequence labeling method to detect and locate English puns in a joint manner. Error Analysis We studied the outputs from our system and make some error analysis. We found the errors can be broadly categorized into several types, and we elaborate them here. 1) Low word coverage: since the corpora are relatively small, there exist many unseen words in the test set. Learning the representations of such unseen words is challenging, which affects the model's performance. Such errors contribute around 40% of the total errors made by our system. 2) Detection errors: we found many errors are due to the model's inability to make correct pun detection. Such inability harms both pun detection and pun location. Although our approach based on the INLINEFORM0 tagging scheme yields relatively higher scores on the detection task, we still found that 40% of the incorrectly predicted instances fall into this group. 3) Short sentences: we found it was challenging for our model to make correct predictions when the given text is short. Consider the example “Superglue! Tom rejoined," here the word rejoined is the corresponding pun. However, it would be challenging to figure out the pun with such limited contextual information. Related Work Most existing systems address pun detection and location separately. BIBREF22 applied word sense knowledge to conduct pun detection. BIBREF24 trained a bidirectional RNN classifier for detecting homographic puns. Next, a knowledge-based approach is adopted to find the exact pun. Such a system is not applicable to heterographic puns. BIBREF28 applied Google n-gram and word2vec to make decisions. The phonetic distance via the CMU Pronouncing Dictionary is computed to detect heterographic puns. BIBREF10 used the hidden Markov model and a cyclic dependency network with rich features to detect and locate puns. BIBREF23 used a supervised approach to pun detection and a weakly supervised approach to pun location based on the position within the context and part of speech features. BIBREF25 proposed a rule-based system for pun location that scores candidate words according to eleven simple heuristics. Two systems are developed to conduct detection and location separately in the system known as UWAV BIBREF3 . The pun detector combines predictions from three classifiers. The pun locator considers word2vec similarity between every pair of words in the context and position to pinpoint the pun. The state-of-the-art system for homographic pun location is a neural method BIBREF4 , where the word senses are incorporated into a bidirectional LSTM model. This method only supports the pun location task on homographic puns. Another line of research efforts related to this work is sequence labeling, such as POS tagging, chunking, word segmentation and NER. The neural methods have shown their effectiveness in this task, such as BiLSTM-CNN BIBREF13 , GRNN BIBREF29 , LSTM-CRF BIBREF30 , LSTM-CNN-CRF BIBREF14 , LM-LSTM-CRF BIBREF15 . In this work, we combine pun detection and location tasks as a single sequence labeling problem. Inspired by the work of BIBREF15 , we also adopt a LSTM-CRF with character embeddings to make labeling decisions. Conclusion In this paper, we propose to perform pun detection and location tasks in a joint manner from a sequence labeling perspective. We observe that each text in our corpora contains a maximum of one pun. Hence, we design a novel tagging scheme to incorporate such a constraint. Such a scheme guarantees that there is a maximum of one word that will be tagged as a pun during the testing phase. We also found the interesting structural property such as the fact that most puns tend to appear at the second half of the sentences can be helpful for such a task, but was not explored in previous works. Furthermore, unlike many previous approaches, our approach, though simple, is generally applicable to both heterographic and homographic puns. Empirical results on the benchmark datasets prove the effectiveness of the proposed approach that the two tasks of pun detection and location can be addressed by a single model from a sequence labeling perspective. Future research includes the investigations on how to make use of richer semantic and linguistic information for detection and location of puns. Research on puns for other languages such as Chinese is still under-explored, which could also be an interesting direction for our future studies. Acknowledgments We would like to thank the three anonymous reviewers for their thoughtful and constructive comments. This work is supported by Singapore Ministry of Education Academic Research Fund (AcRF) Tier 2 Project MOE2017-T2-1-156, and is partially supported by SUTD project PIE-SGP-AI-2018-01.
What datasets are used in evaluation?
f56d07f73b31a9c72ea737b40103d7004ef6a079
What Ġdatasets Ġare Ġused Ġin Ġevaluation ?
The Ġhom ographic Ġdataset Ġcontains Ġ2 , 250 Ġcontexts , Ġ1 , 607 Ġof Ġwhich Ġcontain Ġa Ġpun . ĠThe Ġheter ographic Ġdataset Ġconsists Ġof Ġ1 , 780 Ġcontexts Ġwith Ġ1 , 271 Ġcontaining Ġa Ġpun .
<s> What Ġdatasets Ġare Ġused Ġin Ġevaluation ? </s> Introduction </s> There Ġexists Ġa Ġclass Ġof Ġlanguage Ġconstruction Ġknown Ġas Ġpun Ġin Ġnatural Ġlanguage Ġtexts Ġand Ġutter ances , Ġwhere Ġa Ġcertain Ġword Ġor Ġother Ġlex ical Ġitems Ġare Ġused Ġto Ġexploit Ġtwo Ġor Ġmore Ġseparate Ġmeanings . ĠIt Ġhas Ġbeen Ġshown Ġthat Ġunderstanding Ġof Ġpun s Ġis Ġan Ġimportant Ġresearch Ġquestion Ġwith Ġvarious Ġreal - world Ġapplications , Ġsuch Ġas Ġhuman - computer Ġinteraction ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġand Ġmachine Ġtranslation ĠB IB REF 2 Ġ. ĠRecently , Ġmany Ġresearchers Ġshow Ġtheir Ġinterests Ġin Ġstudying Ġpun s , Ġlike Ġdetecting Ġpun Ġsentences ĠB IB REF 3 Ġ, Ġlocating Ġpun s Ġin Ġthe Ġtext ĠB IB REF 4 Ġ, Ġinterpreting Ġpun Ġsentences ĠB IB REF 5 Ġand Ġgenerating Ġsentences Ġcontaining Ġpun s ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠA Ġpun Ġis Ġa Ġword play Ġin Ġwhich Ġa Ġcertain Ġword Ġsuggests Ġtwo Ġor Ġmore Ġmeanings Ġby Ġexploiting Ġpoly se my , Ġhom ony my , Ġor Ġphon ological Ġsimilarity Ġto Ġanother Ġsign , Ġfor Ġan Ġintended Ġhumorous Ġor Ġrhetorical Ġeffect . ĠP uns Ġcan Ġbe Ġgenerally Ġcategorized Ġinto Ġtwo Ġgroups , Ġnamely Ġheter ographic Ġpun s Ġ( where Ġthe Ġpun Ġand Ġits Ġlatent Ġtarget Ġare Ġphon ologically Ġsimilar ) Ġand Ġhom ographic Ġpun s Ġ( where Ġthe Ġtwo Ġmeanings Ġof Ġthe Ġpun Ġreflect Ġits Ġtwo Ġdistinct Ġsenses ) ĠB IB REF 9 Ġ. ĠConsider Ġthe Ġfollowing Ġtwo Ġexamples : </s> The Ġfirst Ġpun ning Ġjoke Ġexploits Ġthe Ġsound Ġsimilarity Ġbetween Ġthe Ġword ĠâĢ ľ prop ane " Ġand Ġthe Ġlatent Ġtarget ĠâĢ ľ prof ane ", Ġwhich Ġcan Ġbe Ġcategorized Ġinto Ġthe Ġgroup Ġof Ġheter ographic Ġpun s . ĠAnother Ġcategor ization Ġof ĠEnglish Ġpun s Ġis Ġhom ographic Ġpun , Ġexempl ified Ġby Ġthe Ġsecond Ġinstance Ġleveraging Ġdistinct Ġsenses Ġof Ġthe Ġword ĠâĢ ľ g ut ". </s> P un Ġdetection Ġis Ġthe Ġtask Ġof Ġdetecting Ġwhether Ġthere Ġis Ġa Ġpun Ġresiding Ġin Ġthe Ġgiven Ġtext . ĠThe Ġgoal Ġof Ġpun Ġlocation Ġis Ġto Ġfind Ġthe Ġexact Ġword Ġappearing Ġin Ġthe Ġtext Ġthat Ġimplies Ġmore Ġthan Ġone Ġmeanings . ĠMost Ġprevious Ġwork Ġaddresses Ġsuch Ġtwo Ġtasks Ġseparately Ġand Ġdevelop Ġseparate Ġsystems ĠB IB REF 10 Ġ, ĠB IB REF 5 Ġ. ĠTypically , Ġa Ġsystem Ġfor Ġpun Ġdetection Ġis Ġbuilt Ġto Ġmake Ġa Ġbinary Ġprediction Ġon Ġwhether Ġa Ġsentence Ġcontains Ġa Ġpun Ġor Ġnot , Ġwhere Ġall Ġinstances Ġ( with Ġor Ġwithout Ġpun s ) Ġare Ġtaken Ġinto Ġaccount Ġduring Ġtraining . ĠFor Ġthe Ġtask Ġof Ġpun Ġlocation , Ġa Ġseparate Ġsystem Ġis Ġused Ġto Ġmake Ġa Ġsingle Ġprediction Ġas Ġto Ġwhich Ġword Ġin Ġthe Ġgiven Ġsentence Ġin Ġthe Ġtext Ġthat Ġtrigger Ġmore Ġthan Ġone Ġsemantic Ġinterpretations Ġof Ġthe Ġtext , Ġwhere Ġthe Ġtraining Ġdata Ġinvolves Ġonly Ġsentences Ġthat Ġcontain Ġa Ġpun . ĠTherefore , Ġif Ġone Ġis Ġinterested Ġin Ġsolving Ġboth Ġproblems Ġat Ġthe Ġsame Ġtime , Ġa Ġpipeline Ġapproach Ġthat Ġperforms Ġpun Ġdetection Ġfollowed Ġby Ġpun Ġlocation Ġcan Ġbe Ġused . </s> Compared Ġto Ġthe Ġpipeline Ġmethods , Ġjoint Ġlearning Ġhas Ġbeen Ġshown Ġeffective ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġsince Ġit Ġis Ġable Ġto Ġreduce Ġerror Ġpropagation Ġand Ġallows Ġinformation Ġexchange Ġbetween Ġtasks Ġwhich Ġis Ġpotentially Ġbeneficial Ġto Ġall Ġthe Ġtasks . ĠIn Ġthis Ġwork , Ġwe Ġdemonstrate Ġthat Ġthe Ġdetection Ġand Ġlocation Ġof Ġpun s Ġcan Ġbe Ġjointly Ġaddressed Ġby Ġa Ġsingle Ġmodel . ĠThe Ġpun Ġdetection Ġand Ġlocation Ġtasks Ġcan Ġbe Ġcombined Ġas Ġa Ġsequence Ġlabeling Ġproblem , Ġwhich Ġallows Ġus Ġto Ġjointly Ġdetect Ġand Ġlocate Ġa Ġpun Ġin Ġa Ġsentence Ġby Ġassigning Ġeach Ġword Ġa Ġtag . ĠSince Ġeach Ġcontext Ġcontains Ġa Ġmaximum Ġof Ġone Ġpun ĠB IB REF 9 Ġ, Ġwe Ġdesign Ġa Ġnovel Ġtagging Ġscheme Ġto Ġcapture Ġthis Ġstructural Ġconstraint . ĠStatistics Ġon Ġthe Ġcorpor a Ġalso Ġshow Ġthat Ġa Ġpun Ġtends Ġto Ġappear Ġin Ġthe Ġsecond Ġhalf Ġof Ġa Ġcontext . ĠTo Ġcapture Ġsuch Ġa Ġstructural Ġproperty , Ġwe Ġalso Ġincorporate Ġword Ġposition Ġknowledge Ġinto Ġour Ġstructured Ġprediction Ġmodel . ĠExper iments Ġon Ġthe Ġbenchmark Ġdatasets Ġshow Ġthat Ġdetection Ġand Ġlocation Ġtasks Ġcan Ġreinforce Ġeach Ġother , Ġleading Ġto Ġnew Ġstate - of - the - art Ġperformance Ġon Ġthese Ġtwo Ġtasks . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġwork Ġthat Ġperforms Ġjoint Ġdetection Ġand Ġlocation Ġof ĠEnglish Ġpun s Ġby Ġusing Ġa Ġsequence Ġlabeling Ġapproach . </s> Problem ĠDefinition </s> We Ġfirst Ġdesign Ġa Ġsimple Ġtagging Ġscheme Ġconsisting Ġof Ġtwo Ġtags Ġ{ ĠIN LINE FORM 0 Ġ} : </s> IN LINE FORM 0 Ġtag Ġmeans Ġthe Ġcurrent Ġword Ġis Ġnot Ġa Ġpun . </s> IN LINE FORM 0 Ġtag Ġmeans Ġthe Ġcurrent Ġword Ġis Ġa Ġpun . </s> If Ġthe Ġtag Ġsequence Ġof Ġa Ġsentence Ġcontains Ġa ĠIN LINE FORM 0 Ġtag , Ġthen Ġthe Ġtext Ġcontains Ġa Ġpun Ġand Ġthe Ġword Ġcorresponding Ġto ĠIN LINE FORM 1 Ġis Ġthe Ġpun . </s> The Ġcontexts Ġhave Ġthe Ġcharacteristic Ġthat Ġeach Ġcontext Ġcontains Ġa Ġmaximum Ġof Ġone Ġpun ĠB IB REF 9 Ġ. ĠIn Ġother Ġwords , Ġthere Ġexists Ġonly Ġone Ġpun Ġif Ġthe Ġgiven Ġsentence Ġis Ġdetected Ġas Ġthe Ġone Ġcontaining Ġa Ġpun . ĠOtherwise , Ġthere Ġis Ġno Ġpun Ġresiding Ġin Ġthe Ġtext . ĠTo Ġcapture Ġthis Ġinteresting Ġproperty , Ġwe Ġpropose Ġa Ġnew Ġtagging Ġscheme Ġconsisting Ġof Ġthree Ġtags , Ġnamely Ġ{ ĠIN LINE FORM 0 Ġ} . </s> IN LINE FORM 0 Ġtag Ġindicates Ġthat Ġthe Ġcurrent Ġword Ġappears Ġbefore Ġthe Ġpun Ġin Ġthe Ġgiven Ġcontext . </s> IN LINE FORM 0 Ġtag Ġhighlights Ġthe Ġcurrent Ġword Ġis Ġa Ġpun . </s> IN LINE FORM 0 Ġtag Ġindicates Ġthat Ġthe Ġcurrent Ġword Ġappears Ġafter Ġthe Ġpun . </s> We Ġempir ically Ġshow Ġthat Ġthe ĠIN LINE FORM 0 Ġscheme Ġcan Ġguarantee Ġthe Ġcontext Ġproperty Ġthat Ġthere Ġexists Ġa Ġmaximum Ġof Ġone Ġpun Ġresiding Ġin Ġthe Ġtext . </s> Given Ġa Ġcontext Ġfrom Ġthe Ġtraining Ġset , Ġwe Ġwill Ġbe Ġable Ġto Ġgenerate Ġits Ġcorresponding Ġgold Ġtag Ġsequence Ġusing Ġa Ġdeter ministic Ġprocedure . ĠUnder Ġthe Ġtwo Ġschemes , Ġif Ġa Ġsentence Ġdoes Ġnot Ġcontain Ġany Ġpun s , Ġall Ġwords Ġwill Ġbe Ġtagged Ġwith ĠIN LINE FORM 0 Ġor ĠIN LINE FORM 1 Ġ, Ġrespectively . ĠEx empl ified Ġby Ġthe Ġsecond Ġsentence ĠâĢ ľ Some Ġdiets Ġcause Ġa Ġgut Ġreaction ," Ġthe Ġpun Ġis Ġgiven Ġas ĠâĢ ľ g ut ." ĠThus , Ġunder Ġthe ĠIN LINE FORM 2 Ġscheme , Ġit Ġshould Ġbe Ġtagged Ġwith ĠIN LINE FORM 3 Ġ, Ġwhile Ġthe Ġwords Ġbefore Ġit Ġare Ġassigned Ġwith Ġthe Ġtag ĠIN LINE FORM 4 Ġand Ġwords Ġafter Ġit Ġare Ġwith ĠIN LINE FORM 5 Ġ, Ġas Ġillustrated Ġin ĠFigure ĠFIG REF 8 Ġ. ĠLikewise , Ġthe ĠIN LINE FORM 6 Ġscheme Ġtags Ġthe Ġword ĠâĢ ľ g ut " Ġwith ĠIN LINE FORM 7 Ġ, Ġwhile Ġother Ġwords Ġare Ġtagged Ġwith ĠIN LINE FORM 8 Ġ. ĠTherefore , Ġwe Ġcan Ġcombine Ġthe Ġpun Ġdetection Ġand Ġlocation Ġtasks Ġinto Ġone Ġproblem Ġwhich Ġcan Ġbe Ġsolved Ġby Ġthe Ġsequence Ġlabeling Ġapproach . </s> Model </s> Ne ural Ġmodels Ġhave Ġshown Ġtheir Ġeffectiveness Ġon Ġsequence Ġlabeling Ġtasks ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġadopt Ġthe Ġbid irection al ĠLong ĠShort ĠTerm ĠMemory Ġ( Bi L ST M ) ĠB IB REF 16 Ġnetworks Ġon Ġtop Ġof Ġthe ĠCond itional ĠRandom ĠFields ĠB IB REF 17 Ġ( CR F ) Ġarchitecture Ġto Ġmake Ġlabeling Ġdecisions , Ġwhich Ġis Ġone Ġof Ġthe Ġclassical Ġmodels Ġfor Ġsequence Ġlabeling . ĠOur Ġmodel Ġarchitecture Ġis Ġillustrated Ġin ĠFigure ĠFIG REF 8 Ġwith Ġa Ġrunning Ġexample . ĠGiven Ġa Ġcontext / sent ence ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġlength Ġof Ġthe Ġcontext , Ġwe Ġgenerate Ġthe Ġcorresponding Ġtag Ġsequence ĠIN LINE FORM 2 Ġbased Ġon Ġour Ġdesigned Ġtagging Ġschemes Ġand Ġthe Ġoriginal Ġannotations Ġfor Ġpun Ġdetection Ġand Ġlocation Ġprovided Ġby Ġthe Ġcorpor a . ĠOur Ġmodel Ġis Ġthen Ġtrained Ġon Ġpairs Ġof ĠIN LINE FORM 3 Ġ. </s> Input . ĠThe Ġcontexts Ġin Ġthe Ġpun Ġcorpus Ġhold Ġthe Ġproperty Ġthat Ġeach Ġpun Ġcontains Ġexactly Ġone Ġcontent Ġword , Ġwhich Ġcan Ġbe Ġeither Ġa Ġnoun , Ġa Ġverb , Ġan Ġadjective , Ġor Ġan Ġad verb . ĠTo Ġcapture Ġthis Ġcharacteristic , Ġwe Ġconsider Ġlex ical Ġfeatures Ġat Ġthe Ġcharacter Ġlevel . ĠSimilar Ġto Ġthe Ġwork Ġof ĠB IB REF 15 Ġ, Ġthe Ġcharacter Ġembed d ings Ġare Ġtrained Ġby Ġthe Ġcharacter - level ĠL ST M Ġnetworks Ġon Ġthe Ġun annot ated Ġinput Ġsequences . ĠNon linear Ġtransformations Ġare Ġthen Ġapplied Ġto Ġthe Ġcharacter Ġembed d ings Ġby Ġhighway Ġnetworks ĠB IB REF 18 Ġ, Ġwhich Ġmap Ġthe Ġcharacter - level Ġfeatures Ġinto Ġdifferent Ġsemantic Ġspaces . </s> We Ġalso Ġobserve Ġthat Ġa Ġpun Ġtends Ġto Ġappear Ġat Ġthe Ġend Ġof Ġa Ġsentence . ĠSpecifically , Ġbased Ġon Ġthe Ġstatistics , Ġwe Ġfound Ġthat Ġsentences Ġwith Ġa Ġpun Ġthat Ġlocate Ġat Ġthe Ġsecond Ġhalf Ġof Ġthe Ġtext Ġaccount Ġfor Ġaround Ġ88 % Ġand Ġ92 % Ġin Ġhom ographic Ġand Ġheter ographic Ġdatasets , Ġrespectively . ĠWe Ġthus Ġintroduce Ġa Ġbinary Ġfeature Ġthat Ġindicates Ġif Ġa Ġword Ġis Ġlocated Ġat Ġthe Ġfirst Ġor Ġthe Ġsecond Ġhalf Ġof Ġan Ġinput Ġsentence Ġto Ġcapture Ġsuch Ġpositional Ġinformation . ĠA Ġbinary Ġindicator Ġcan Ġbe Ġmapped Ġto Ġa Ġvector Ġrepresentation Ġusing Ġa Ġrandomly Ġinitialized Ġembed ding Ġtable ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġdirectly Ġadopt Ġthe Ġvalue Ġof Ġthe Ġbinary Ġindicator Ġas Ġpart Ġof Ġthe Ġinput . </s> The Ġconc aten ation Ġof Ġthe Ġtransformed Ġcharacter Ġembed d ings , Ġthe Ġpre - trained Ġword Ġembed d ings ĠB IB REF 21 Ġ, Ġand Ġthe Ġposition Ġindicators Ġare Ġtaken Ġas Ġinput Ġof Ġour Ġmodel . </s> T agging . ĠThe Ġinput Ġis Ġthen Ġfed Ġinto Ġa ĠBi L ST M Ġnetwork , Ġwhich Ġwill Ġbe Ġable Ġto Ġcapture Ġcontextual Ġinformation . ĠFor Ġa Ġtraining Ġinstance ĠIN LINE FORM 0 Ġ, Ġwe Ġsuppose Ġthe Ġoutput Ġby Ġthe Ġword - level ĠBi L ST M Ġis ĠIN LINE FORM 1 Ġ. ĠThe ĠCR F Ġlayer Ġis Ġadopted Ġto Ġcapture Ġlabel Ġdependencies Ġand Ġmake Ġfinal Ġtagging Ġdecisions Ġat Ġeach Ġposition , Ġwhich Ġhas Ġbeen Ġincluded Ġin Ġmany Ġstate - of - the - art Ġsequence Ġlabeling Ġmodels ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠThe Ġconditional Ġprobability Ġis Ġdefined Ġas : </s> where ĠIN LINE FORM 0 Ġis Ġa Ġset Ġof Ġall Ġpossible Ġlabel Ġsequences Ġconsisting Ġof Ġtags Ġfrom ĠIN LINE FORM 1 Ġ( or ĠIN LINE FORM 2 Ġ), ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġare Ġweight Ġand Ġbias Ġparameters Ġcorresponding Ġto Ġthe Ġlabel Ġpair ĠIN LINE FORM 5 Ġ. ĠDuring Ġtraining , Ġwe Ġminimize Ġthe Ġnegative Ġlog - like lihood Ġsummed Ġover Ġall Ġtraining Ġinstances : </s> where ĠIN LINE FORM 0 Ġrefers Ġto Ġthe ĠIN LINE FORM 1 Ġ- th Ġinstance Ġin Ġthe Ġtraining Ġset . ĠDuring Ġtesting , Ġwe Ġaim Ġto Ġfind Ġthe Ġoptimal Ġlabel Ġsequence Ġfor Ġa Ġnew Ġinput ĠIN LINE FORM 2 Ġ: </s> This Ġsearch Ġprocess Ġcan Ġbe Ġdone Ġefficiently Ġusing Ġthe ĠV iter bi Ġalgorithm . </s> Dat as ets Ġand ĠSettings </s> We Ġevaluate Ġour Ġmodel Ġon Ġtwo Ġbenchmark Ġdatasets ĠB IB REF 9 Ġ. ĠThe Ġhom ographic Ġdataset Ġcontains Ġ2 , 250 Ġcontexts , Ġ1 , 607 Ġof Ġwhich Ġcontain Ġa Ġpun . ĠThe Ġheter ographic Ġdataset Ġconsists Ġof Ġ1 , 780 Ġcontexts Ġwith Ġ1 , 271 Ġcontaining Ġa Ġpun . ĠWe Ġnotice Ġthere Ġis Ġno Ġstandard Ġsplitting Ġinformation Ġprovided Ġfor Ġboth Ġdatasets . ĠThus Ġwe Ġapply Ġ10 - fold Ġcross Ġvalidation . ĠTo Ġmake Ġdirect Ġcomparisons Ġwith Ġprior Ġstudies , Ġfollowing ĠB IB REF 4 Ġ, Ġwe Ġaccumulated Ġthe Ġpredictions Ġfor Ġall Ġten Ġfolds Ġand Ġcalculate Ġthe Ġscores Ġin Ġthe Ġend . </s> For Ġeach Ġfold , Ġwe Ġrandomly Ġselect Ġ10 % Ġof Ġthe Ġinstances Ġfrom Ġthe Ġtraining Ġset Ġfor Ġdevelopment . ĠWord Ġembed d ings Ġare Ġinitialized Ġwith Ġthe Ġ100 - dimensional ĠGl ove ĠB IB REF 21 Ġ. ĠThe Ġdimension Ġof Ġcharacter Ġembed d ings Ġis Ġ30 Ġand Ġthey Ġare Ġrandomly Ġinitialized , Ġwhich Ġcan Ġbe Ġfine Ġtuned Ġduring Ġtraining . ĠThe Ġpre - trained Ġword Ġembed d ings Ġare Ġnot Ġupdated Ġduring Ġtraining . ĠThe Ġdimensions Ġof Ġhidden Ġvectors Ġfor Ġboth Ġchar - level Ġand Ġword - level ĠL ST M Ġunits Ġare Ġset Ġto Ġ300 . ĠWe Ġadopt Ġsto ch astic Ġgradient Ġdescent Ġ( SG D ) ĠB IB REF 26 Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 015 . </s> For Ġthe Ġpun Ġdetection Ġtask , Ġif Ġthe Ġpredicted Ġtag Ġsequence Ġcontains Ġat Ġleast Ġone ĠIN LINE FORM 0 Ġtag , Ġwe Ġregard Ġthe Ġoutput Ġ( i . e ., Ġthe Ġprediction Ġof Ġour Ġpun Ġdetection Ġmodel ) Ġfor Ġthis Ġtask Ġas Ġtrue , Ġotherwise Ġfalse . ĠFor Ġthe Ġpun Ġlocation Ġtask , Ġa Ġpredicted Ġpun Ġis Ġregarded Ġas Ġcorrect Ġif Ġand Ġonly Ġif Ġit Ġis Ġlabeled Ġas Ġthe Ġgold Ġpun Ġin Ġthe Ġdataset . ĠAs Ġto Ġpun Ġlocation , Ġto Ġmake Ġfair Ġcomparisons Ġwith Ġprior Ġstudies , Ġwe Ġonly Ġconsider Ġthe Ġinstances Ġthat Ġare Ġlabeled Ġas Ġthe Ġones Ġcontaining Ġa Ġpun . ĠWe Ġreport Ġprecision , Ġrecall Ġand ĠIN LINE FORM 1 Ġscore Ġin ĠTable ĠT AB REF 11 Ġ. ĠA Ġlist Ġof Ġprior Ġworks Ġthat Ġdid Ġnot Ġemploy Ġjoint Ġlearning Ġare Ġalso Ġshown Ġin Ġthe Ġfirst Ġblock Ġof ĠTable ĠT AB REF 11 Ġ. </s> Results </s> We Ġalso Ġimplemented Ġa Ġbaseline Ġmodel Ġbased Ġon Ġconditional Ġrandom Ġfields Ġ( CR F ), Ġwhere Ġfeatures Ġlike ĠPOS Ġtags Ġproduced Ġby Ġthe ĠStanford ĠPOS Ġtag ger ĠB IB REF 27 Ġ, Ġn - gram s , Ġlabel Ġtransitions , Ġword Ġsuffix es Ġand Ġrelative Ġposition Ġto Ġthe Ġend Ġof Ġthe Ġtext Ġare Ġconsidered . ĠWe Ġcan Ġsee Ġthat Ġour Ġmodel Ġwith Ġthe ĠIN LINE FORM 0 Ġtagging Ġscheme Ġyields Ġnew Ġstate - of - the - art ĠIN LINE FORM 1 Ġscores Ġon Ġpun Ġdetection Ġand Ġcompetitive Ġresults Ġon Ġpun Ġlocation , Ġcompared Ġto Ġbas elines Ġthat Ġdo Ġnot Ġadopt Ġjoint Ġlearning Ġin Ġthe Ġfirst Ġblock . ĠFor Ġlocation Ġon Ġheter ographic Ġpun s , Ġour Ġmodel 's Ġperformance Ġis Ġslightly Ġlower Ġthan Ġthe Ġsystem Ġof ĠB IB REF 25 Ġ, Ġwhich Ġis Ġa Ġrule - based Ġloc ator . ĠCompared Ġto ĠCR F , Ġwe Ġcan Ġsee Ġthat Ġour Ġmodel , Ġeither Ġwith Ġthe ĠIN LINE FORM 2 Ġor Ġthe ĠIN LINE FORM 3 Ġscheme , Ġyields Ġsignificantly Ġhigher Ġrecall Ġon Ġboth Ġdetection Ġand Ġlocation Ġtasks , Ġwhile Ġthe Ġprec isions Ġare Ġrelatively Ġclose . ĠThis Ġdemonstrates Ġthe Ġeffectiveness Ġof ĠBi L ST M , Ġwhich Ġlearns Ġthe Ġcontextual Ġfeatures Ġof Ġgiven Ġtexts ĠâĢĵ Ġsuch Ġinformation Ġappears Ġto Ġbe Ġhelpful Ġin Ġrecalling Ġmore Ġpun s . </s> Compared Ġto Ġthe ĠIN LINE FORM 0 Ġscheme , Ġthe ĠIN LINE FORM 1 Ġtagging Ġscheme Ġis Ġable Ġto Ġyield Ġbetter Ġperformance Ġon Ġthese Ġtwo Ġtasks . ĠAfter Ġstudying Ġoutputs Ġfrom Ġthese Ġtwo Ġapproaches , Ġwe Ġfound Ġthat Ġone Ġleading Ġsource Ġof Ġerror Ġfor Ġthe ĠIN LINE FORM 2 Ġapproach Ġis Ġthat Ġthere Ġexist Ġmore Ġthan Ġone Ġwords Ġin Ġa Ġsingle Ġinstance Ġthat Ġare Ġassigned Ġwith Ġthe ĠIN LINE FORM 3 Ġtag . ĠHowever , Ġaccording Ġto Ġthe Ġdescription Ġof Ġpun Ġin ĠB IB REF 9 Ġ, Ġeach Ġcontext Ġcontains Ġa Ġmaximum Ġof Ġone Ġpun . ĠThus , Ġsuch Ġa Ġuseful Ġstructural Ġconstraint Ġis Ġnot Ġwell Ġcaptured Ġby Ġthe Ġsimple Ġapproach Ġbased Ġon Ġthe ĠIN LINE FORM 4 Ġtagging Ġscheme . ĠOn Ġthe Ġother Ġhand , Ġby Ġapplying Ġthe ĠIN LINE FORM 5 Ġtagging Ġscheme , Ġsuch Ġa Ġconstraint Ġis Ġproperly Ġcaptured Ġin Ġthe Ġmodel . ĠAs Ġa Ġresult , Ġthe Ġresults Ġfor Ġsuch Ġa Ġapproach Ġare Ġsignificantly Ġbetter Ġthan Ġthe Ġapproach Ġbased Ġon Ġthe ĠIN LINE FORM 6 Ġtagging Ġscheme , Ġas Ġwe Ġcan Ġobserve Ġfrom Ġthe Ġtable . ĠUnder Ġthe Ġsame Ġexperimental Ġsetup , Ġwe Ġalso Ġattempted Ġto Ġexclude Ġword Ġposition Ġfeatures . ĠResults Ġare Ġgiven Ġby ĠIN LINE FORM 7 Ġ- ĠIN LINE FORM 8 Ġ. ĠIt Ġis Ġexpected Ġthat Ġthe Ġperformance Ġof Ġpun Ġlocation Ġdrops , Ġsince Ġsuch Ġposition Ġfeatures Ġare Ġable Ġto Ġcapture Ġthe Ġinteresting Ġproperty Ġthat Ġa Ġpun Ġtends Ġto Ġappear Ġin Ġthe Ġsecond Ġhalf Ġof Ġa Ġsentence . ĠWhile Ġsuch Ġknowledge Ġis Ġhelpful Ġfor Ġthe Ġlocation Ġtask , Ġinterestingly , Ġa Ġmodel Ġwithout Ġposition Ġknowledge Ġyields Ġimproved Ġperformance Ġon Ġthe Ġpun Ġdetection Ġtask . ĠOne Ġpossible Ġreason Ġis Ġthat Ġdetecting Ġwhether Ġa Ġsentence Ġcontains Ġa Ġpun Ġis Ġnot Ġconcerned Ġwith Ġsuch Ġword Ġposition Ġinformation . </s> Additionally , Ġwe Ġconduct Ġexperiments Ġover Ġsentences Ġcontaining Ġa Ġpun Ġonly , Ġnamely Ġ1 , 607 Ġand Ġ1 , 271 Ġinstances Ġfrom Ġhom ographic Ġand Ġheter ographic Ġpun Ġcorpor a Ġseparately . ĠIt Ġcan Ġbe Ġregarded Ġas Ġa ĠâĢ ľ p ip eline âĢ Ŀ Ġmethod Ġwhere Ġthe Ġclass ifier Ġfor Ġpun Ġdetection Ġis Ġregarded Ġas Ġperfect . ĠFollowing Ġthe Ġprior Ġwork Ġof ĠB IB REF 4 Ġ, Ġwe Ġapply Ġ10 - fold Ġcross Ġvalidation . ĠSince Ġwe Ġare Ġgiven Ġthat Ġall Ġinput Ġsentences Ġcontain Ġa Ġpun , Ġwe Ġonly Ġreport Ġaccumulated Ġresults Ġon Ġpun Ġlocation , Ġden oted Ġas ĠPipeline Ġin ĠTable ĠT AB REF 11 Ġ. ĠCompared Ġwith Ġour Ġapproaches , Ġthe Ġperformance Ġof Ġsuch Ġan Ġapproach Ġdrops Ġsignificantly . ĠOn Ġthe Ġother Ġhand , Ġsuch Ġa Ġfact Ġdemonstrates Ġthat Ġthe Ġtwo Ġtask , Ġdetection Ġand Ġlocation Ġof Ġpun s , Ġcan Ġreinforce Ġeach Ġother . ĠThese Ġfigures Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġour Ġsequence Ġlabeling Ġmethod Ġto Ġdetect Ġand Ġlocate ĠEnglish Ġpun s Ġin Ġa Ġjoint Ġmanner . </s> Error ĠAnalysis </s> We Ġstudied Ġthe Ġoutputs Ġfrom Ġour Ġsystem Ġand Ġmake Ġsome Ġerror Ġanalysis . ĠWe Ġfound Ġthe Ġerrors Ġcan Ġbe Ġbroadly Ġcategorized Ġinto Ġseveral Ġtypes , Ġand Ġwe Ġelaborate Ġthem Ġhere . Ġ1 ) ĠLow Ġword Ġcoverage : Ġsince Ġthe Ġcorpor a Ġare Ġrelatively Ġsmall , Ġthere Ġexist Ġmany Ġunseen Ġwords Ġin Ġthe Ġtest Ġset . ĠLearning Ġthe Ġrepresentations Ġof Ġsuch Ġunseen Ġwords Ġis Ġchallenging , Ġwhich Ġaffects Ġthe Ġmodel 's Ġperformance . ĠSuch Ġerrors Ġcontribute Ġaround Ġ40 % Ġof Ġthe Ġtotal Ġerrors Ġmade Ġby Ġour Ġsystem . Ġ2 ) ĠDetection Ġerrors : Ġwe Ġfound Ġmany Ġerrors Ġare Ġdue Ġto Ġthe Ġmodel 's Ġinability Ġto Ġmake Ġcorrect Ġpun Ġdetection . ĠSuch Ġinability Ġharms Ġboth Ġpun Ġdetection Ġand Ġpun Ġlocation . ĠAlthough Ġour Ġapproach Ġbased Ġon Ġthe ĠIN LINE FORM 0 Ġtagging Ġscheme Ġyields Ġrelatively Ġhigher Ġscores Ġon Ġthe Ġdetection Ġtask , Ġwe Ġstill Ġfound Ġthat Ġ40 % Ġof Ġthe Ġincorrectly Ġpredicted Ġinstances Ġfall Ġinto Ġthis Ġgroup . Ġ3 ) ĠShort Ġsentences : Ġwe Ġfound Ġit Ġwas Ġchallenging Ġfor Ġour Ġmodel Ġto Ġmake Ġcorrect Ġpredictions Ġwhen Ġthe Ġgiven Ġtext Ġis Ġshort . ĠConsider Ġthe Ġexample ĠâĢ ľ Super gl ue ! ĠTom Ġrejo ined ," Ġhere Ġthe Ġword Ġrejo ined Ġis Ġthe Ġcorresponding Ġpun . ĠHowever , Ġit Ġwould Ġbe Ġchallenging Ġto Ġfigure Ġout Ġthe Ġpun Ġwith Ġsuch Ġlimited Ġcontextual Ġinformation . </s> Related ĠWork </s> Most Ġexisting Ġsystems Ġaddress Ġpun Ġdetection Ġand Ġlocation Ġseparately . ĠB IB REF 22 Ġapplied Ġword Ġsense Ġknowledge Ġto Ġconduct Ġpun Ġdetection . ĠB IB REF 24 Ġtrained Ġa Ġbid irection al ĠR NN Ġclass ifier Ġfor Ġdetecting Ġhom ographic Ġpun s . ĠNext , Ġa Ġknowledge - based Ġapproach Ġis Ġadopted Ġto Ġfind Ġthe Ġexact Ġpun . ĠSuch Ġa Ġsystem Ġis Ġnot Ġapplicable Ġto Ġheter ographic Ġpun s . ĠB IB REF 28 Ġapplied ĠGoogle Ġn - gram Ġand Ġword 2 vec Ġto Ġmake Ġdecisions . ĠThe Ġphon etic Ġdistance Ġvia Ġthe ĠCM U ĠP ron ouncing ĠDictionary Ġis Ġcomputed Ġto Ġdetect Ġheter ographic Ġpun s . ĠB IB REF 10 Ġused Ġthe Ġhidden ĠMark ov Ġmodel Ġand Ġa Ġcycl ic Ġdependency Ġnetwork Ġwith Ġrich Ġfeatures Ġto Ġdetect Ġand Ġlocate Ġpun s . ĠB IB REF 23 Ġused Ġa Ġsupervised Ġapproach Ġto Ġpun Ġdetection Ġand Ġa Ġweak ly Ġsupervised Ġapproach Ġto Ġpun Ġlocation Ġbased Ġon Ġthe Ġposition Ġwithin Ġthe Ġcontext Ġand Ġpart Ġof Ġspeech Ġfeatures . ĠB IB REF 25 Ġproposed Ġa Ġrule - based Ġsystem Ġfor Ġpun Ġlocation Ġthat Ġscores Ġcandidate Ġwords Ġaccording Ġto Ġeleven Ġsimple Ġhe ur istics . ĠTwo Ġsystems Ġare Ġdeveloped Ġto Ġconduct Ġdetection Ġand Ġlocation Ġseparately Ġin Ġthe Ġsystem Ġknown Ġas ĠUW AV ĠB IB REF 3 Ġ. ĠThe Ġpun Ġdetector Ġcombines Ġpredictions Ġfrom Ġthree Ġclass ifiers . ĠThe Ġpun Ġloc ator Ġconsiders Ġword 2 vec Ġsimilarity Ġbetween Ġevery Ġpair Ġof Ġwords Ġin Ġthe Ġcontext Ġand Ġposition Ġto Ġpinpoint Ġthe Ġpun . ĠThe Ġstate - of - the - art Ġsystem Ġfor Ġhom ographic Ġpun Ġlocation Ġis Ġa Ġneural Ġmethod ĠB IB REF 4 Ġ, Ġwhere Ġthe Ġword Ġsenses Ġare Ġincorporated Ġinto Ġa Ġbid irection al ĠL ST M Ġmodel . ĠThis Ġmethod Ġonly Ġsupports Ġthe Ġpun Ġlocation Ġtask Ġon Ġhom ographic Ġpun s . ĠAnother Ġline Ġof Ġresearch Ġefforts Ġrelated Ġto Ġthis Ġwork Ġis Ġsequence Ġlabeling , Ġsuch Ġas ĠPOS Ġtagging , Ġchunk ing , Ġword Ġsegment ation Ġand ĠN ER . ĠThe Ġneural Ġmethods Ġhave Ġshown Ġtheir Ġeffectiveness Ġin Ġthis Ġtask , Ġsuch Ġas ĠBi L ST M - CNN ĠB IB REF 13 Ġ, ĠGR NN ĠB IB REF 29 Ġ, ĠL ST M - CR F ĠB IB REF 30 Ġ, ĠL ST M - CNN - CR F ĠB IB REF 14 Ġ, ĠLM - L ST M - CR F ĠB IB REF 15 Ġ. </s> In Ġthis Ġwork , Ġwe Ġcombine Ġpun Ġdetection Ġand Ġlocation Ġtasks Ġas Ġa Ġsingle Ġsequence Ġlabeling Ġproblem . ĠInspired Ġby Ġthe Ġwork Ġof ĠB IB REF 15 Ġ, Ġwe Ġalso Ġadopt Ġa ĠL ST M - CR F Ġwith Ġcharacter Ġembed d ings Ġto Ġmake Ġlabeling Ġdecisions . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġto Ġperform Ġpun Ġdetection Ġand Ġlocation Ġtasks Ġin Ġa Ġjoint Ġmanner Ġfrom Ġa Ġsequence Ġlabeling Ġperspective . ĠWe Ġobserve Ġthat Ġeach Ġtext Ġin Ġour Ġcorpor a Ġcontains Ġa Ġmaximum Ġof Ġone Ġpun . ĠHence , Ġwe Ġdesign Ġa Ġnovel Ġtagging Ġscheme Ġto Ġincorporate Ġsuch Ġa Ġconstraint . ĠSuch Ġa Ġscheme Ġguarantees Ġthat Ġthere Ġis Ġa Ġmaximum Ġof Ġone Ġword Ġthat Ġwill Ġbe Ġtagged Ġas Ġa Ġpun Ġduring Ġthe Ġtesting Ġphase . ĠWe Ġalso Ġfound Ġthe Ġinteresting Ġstructural Ġproperty Ġsuch Ġas Ġthe Ġfact Ġthat Ġmost Ġpun s Ġtend Ġto Ġappear Ġat Ġthe Ġsecond Ġhalf Ġof Ġthe Ġsentences Ġcan Ġbe Ġhelpful Ġfor Ġsuch Ġa Ġtask , Ġbut Ġwas Ġnot Ġexplored Ġin Ġprevious Ġworks . ĠFurthermore , Ġunlike Ġmany Ġprevious Ġapproaches , Ġour Ġapproach , Ġthough Ġsimple , Ġis Ġgenerally Ġapplicable Ġto Ġboth Ġheter ographic Ġand Ġhom ographic Ġpun s . ĠEm pir ical Ġresults Ġon Ġthe Ġbenchmark Ġdatasets Ġprove Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġapproach Ġthat Ġthe Ġtwo Ġtasks Ġof Ġpun Ġdetection Ġand Ġlocation Ġcan Ġbe Ġaddressed Ġby Ġa Ġsingle Ġmodel Ġfrom Ġa Ġsequence Ġlabeling Ġperspective . </s> Future Ġresearch Ġincludes Ġthe Ġinvestigations Ġon Ġhow Ġto Ġmake Ġuse Ġof Ġricher Ġsemantic Ġand Ġlinguistic Ġinformation Ġfor Ġdetection Ġand Ġlocation Ġof Ġpun s . ĠResearch Ġon Ġpun s Ġfor Ġother Ġlanguages Ġsuch Ġas ĠChinese Ġis Ġstill Ġunder - expl ored , Ġwhich Ġcould Ġalso Ġbe Ġan Ġinteresting Ġdirection Ġfor Ġour Ġfuture Ġstudies . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġthree Ġanonymous Ġreviewers Ġfor Ġtheir Ġthoughtful Ġand Ġconstructive Ġcomments . ĠThis Ġwork Ġis Ġsupported Ġby ĠSingapore ĠMinistry Ġof ĠEducation ĠAcademic ĠResearch ĠFund Ġ( Ac RF ) ĠTier Ġ2 ĠProject ĠMO E 2017 - T 2 - 1 - 156 , Ġand Ġis Ġpartially Ġsupported Ġby ĠS UT D Ġproject ĠP IE - S GP - AI - 2018 - 01 .