PER, LOC, ORG, MISC
EXTRACTIVE
PER, LOC, ORG, MISC
EXTRACTIVE
The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC). These captions are collected exclusively from snaps submitted to public and crowd-sourced stories (aka Snapchat Live Stories or Our Stories). Examples of such public crowd-sourced stories are “New York Story” or “Thanksgiving Story”, which comprise snaps that are aggregated for various public events, venues, etc. All snaps were posted between year 2016 and 2017, and do not contain raw images or other associated information (only textual captions and obfuscated visual descriptor features extracted from the pre-trained InceptionNet are available). We split the dataset into train (70%), validation (15%), and test sets (15%). The captions data have average length of 30.7 characters (5.81 words) with vocabulary size 15,733, where 6,612 are considered unknown tokens from Stanford GloVE embeddings BIBREF22 . Named entities annotated in the SnapCaptions dataset include many of new and emerging entities, and they are found in various surface forms (various nicknames, typos, etc.) To the best of our knowledge, SnapCaptions is the only dataset that contains natural image-caption pairs with expert-annotated named entities.
The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC). These captions are collected exclusively from snaps submitted to public and crowd-sourced stories (aka Snapchat Live Stories or Our Stories). Examples of such public crowd-sourced stories are “New York Story” or “Thanksgiving Story”, which comprise snaps that are aggregated for various public events, venues, etc. All snaps were posted between year 2016 and 2017, and do not contain raw images or other associated information (only textual captions and obfuscated visual descriptor features extracted from the pre-trained InceptionNet are available). We split the dataset into train (70%), validation (15%), and test sets (15%). The captions data have average length of 30.7 characters (5.81 words) with vocabulary size 15,733, where 6,612 are considered unknown tokens from Stanford GloVE embeddings BIBREF22 . Named entities annotated in the SnapCaptions dataset include many of new and emerging entities, and they are found in various surface forms (various nicknames, typos, etc.) To the best of our knowledge, SnapCaptions is the only dataset that contains natural image-caption pairs with expert-annotated named entities.
1802.07862
Introduction </s> Social Ġmedia Ġwith Ġabundant Ġuser - generated Ġposts Ġprovide Ġa Ġrich Ġplatform Ġfor Ġunderstanding Ġevents , Ġopinions Ġand Ġpreferences Ġof Ġgroups Ġand Ġindividuals . ĠThese Ġinsights Ġare Ġprimarily Ġhidden Ġin Ġun struct ured Ġforms Ġof Ġsocial Ġmedia Ġposts , Ġsuch Ġas Ġin Ġfree - form Ġtext Ġor Ġimages Ġwithout Ġtags . ĠNamed Ġentity Ġrecognition Ġ( NER ), Ġthe Ġtask Ġof Ġrecognizing Ġnamed Ġentities Ġfrom Ġfree - form Ġtext , Ġis Ġthus Ġa Ġcritical Ġstep Ġfor Ġbuilding Ġstructural Ġinformation , Ġallowing Ġfor Ġits Ġuse Ġin Ġpersonalized Ġassistance , Ġrecommendations , Ġadvertisement , Ġetc . </s> While Ġmany Ġprevious Ġapproaches ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġon ĠN ER Ġhave Ġshown Ġsuccess Ġfor Ġwell - formed Ġtext Ġin Ġrecognizing Ġnamed Ġentities Ġvia Ġword Ġcontext Ġresolution Ġ( e . g . ĠL ST M Ġwith Ġword Ġembed d ings ) Ġcombined Ġwith Ġcharacter - level Ġfeatures Ġ( e . g . ĠChar L ST M / CNN ), Ġseveral Ġadditional Ġchallenges Ġremain Ġfor Ġrecognizing Ġnamed Ġentities Ġfrom Ġextremely Ġshort Ġand Ġcoarse Ġtext Ġfound Ġin Ġsocial Ġmedia Ġposts . ĠFor Ġinstance , Ġshort Ġsocial Ġmedia Ġposts Ġoften Ġdo Ġnot Ġprovide Ġenough Ġtextual Ġcontexts Ġto Ġresolve Ġpoly sem ous Ġentities Ġ( e . g . ĠâĢ ľ mon opoly Ġis Ġda Ġbest Ġ", Ġwhere Ġ` mon opoly ' Ġmay Ġrefer Ġto Ġa Ġboard Ġgame Ġ( named Ġentity ) Ġor Ġa Ġterm Ġin Ġeconomics ). ĠIn Ġaddition , Ġnoisy Ġtext Ġincludes Ġa Ġhuge Ġnumber Ġof Ġunknown Ġtokens Ġdue Ġto Ġinconsistent Ġlex ical Ġnot ations Ġand Ġfrequent Ġmentions Ġof Ġvarious Ġnewly Ġtrending Ġentities Ġ( e . g . ĠâĢ ľ x ox o ĠMarsh m ell oooo Ġ", Ġwhere Ġ` Marsh m ell oooo ' Ġis Ġa Ġmis - spe lling Ġof Ġa Ġknown Ġentity Ġ` Marsh m ello ', Ġa Ġmusic Ġproducer ), Ġmaking Ġword Ġembed d ings Ġbased Ġneural Ġnetworks ĠN ER Ġmodels Ġvulnerable . </s> To Ġaddress Ġthe Ġchallenges Ġabove Ġfor Ġsocial Ġmedia Ġposts , Ġwe Ġbuild Ġupon Ġthe Ġstate - of - the - art Ġneural Ġarchitecture Ġfor ĠN ER Ġwith Ġthe Ġfollowing Ġtwo Ġnovel Ġapproaches Ġ( Figure ĠFIG REF 1 Ġ). ĠFirst , Ġwe Ġpropose Ġto Ġleverage Ġauxiliary Ġmod alities Ġfor Ġadditional Ġcontext Ġresolution Ġof Ġentities . ĠFor Ġexample , Ġmany Ġpopular Ġsocial Ġmedia Ġplatforms Ġnow Ġprovide Ġways Ġto Ġcompose Ġa Ġpost Ġin Ġmultiple Ġmod alities Ġ- Ġspecifically Ġimage Ġand Ġtext Ġ( e . g . ĠSnapchat Ġcapt ions , ĠTwitter Ġposts Ġwith Ġimage ĠURLs ), Ġfrom Ġwhich Ġwe Ġcan Ġobtain Ġadditional Ġcontext Ġfor Ġunderstanding Ġposts . ĠWhile ĠâĢ ľ mon opoly " Ġin Ġthe Ġprevious Ġexample Ġis Ġambiguous Ġin Ġits Ġtextual Ġform , Ġan Ġaccompanying Ġsnap Ġimage Ġof Ġa Ġboard Ġgame Ġcan Ġhelp Ġdis amb ig uate Ġamong Ġpoly sem ous Ġentities , Ġthereby Ġcorrectly Ġrecognizing Ġit Ġas Ġa Ġnamed Ġentity . </s> Second , Ġwe Ġalso Ġpropose Ġa Ġgeneral Ġmod ality Ġattention Ġmodule Ġwhich Ġchooses Ġper Ġdecoding Ġstep Ġthe Ġmost Ġinformative Ġmod ality Ġamong Ġavailable Ġones Ġ( in Ġour Ġcase , Ġword Ġembed d ings , Ġcharacter Ġembed d ings , Ġor Ġvisual Ġfeatures ) Ġto Ġextract Ġcontext Ġfrom . ĠFor Ġexample , Ġthe Ġmod ality Ġattention Ġmodule Ġlets Ġthe Ġdec oder Ġatten uate Ġthe Ġword - level Ġsignals Ġfor Ġunknown Ġword Ġtokens Ġ( âĢ ľ Marsh m ell oooo o " Ġwith Ġtrailing Ġ` o 's ) Ġand Ġampl ifies Ġcharacter - level Ġfeatures Ġint se ad Ġ( capital ized Ġfirst Ġletter , Ġlex ical Ġsimilarity Ġto Ġother Ġknown Ġnamed Ġentity Ġtoken Ġ` Marsh m ello ', Ġetc .), Ġthereby Ġsuppressing Ġnoise Ġinformation Ġ( âĢ ľ UN K " Ġtoken Ġembed ding ) Ġin Ġdecoding Ġsteps . ĠNote Ġthat Ġmost Ġof Ġthe Ġprevious Ġliterature Ġin ĠN ER Ġor Ġother ĠN LP Ġtasks Ġcombine Ġword Ġand Ġcharacter - level Ġinformation Ġwith Ġnaive Ġconc aten ation , Ġwhich Ġis Ġvulnerable Ġto Ġnoisy Ġsocial Ġmedia Ġposts . ĠWhen Ġan Ġauxiliary Ġimage Ġis Ġavailable , Ġthe Ġmod ality Ġattention Ġmodule Ġdetermines Ġto Ġamplify Ġthis Ġvisual Ġcontext Ġin Ġdis amb ig uating Ġpoly sem ous Ġentities , Ġor Ġto Ġatten uate Ġvisual Ġcontexts Ġwhen Ġthey Ġare Ġirrelevant Ġto Ġtarget Ġnamed Ġentities , Ġselfies , Ġetc . ĠNote Ġthat Ġthe Ġproposed Ġmod ality Ġattention Ġmodule Ġis Ġdistinct Ġfrom Ġhow Ġattention Ġis Ġused Ġin Ġother Ġsequence - to - sequence Ġliterature Ġ( e . g . Ġattending Ġto Ġa Ġspecific Ġtoken Ġwithin Ġan Ġinput Ġsequence ). ĠSection ĠSEC REF 2 Ġprovides Ġthe Ġdetailed Ġliterature Ġreview . </s> Our Ġcontributions Ġare Ġthree - fold : Ġwe Ġpropose Ġ( 1 ) Ġan ĠL ST M - CNN Ġhybrid Ġmultim od al ĠN ER Ġnetwork Ġthat Ġtakes Ġas Ġinput Ġboth Ġimage Ġand Ġtext Ġfor Ġrecognition Ġof Ġa Ġnamed Ġentity Ġin Ġtext Ġinput . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġour Ġapproach Ġis Ġthe Ġfirst Ġwork Ġto Ġincorporate Ġvisual Ġcontexts Ġfor Ġnamed Ġentity Ġrecognition Ġtasks . Ġ( 2 ) ĠWe Ġpropose Ġa Ġgeneral Ġmod ality Ġattention Ġmodule Ġthat Ġselectively Ġchooses Ġmod alities Ġto Ġextract Ġprimary Ġcontext Ġfrom , Ġmaximizing Ġinformation Ġgain Ġand Ġsuppressing Ġirrelevant Ġcontexts Ġfrom Ġeach Ġmod ality Ġ( we Ġtreat Ġwords , Ġcharacters , Ġand Ġimages Ġas Ġseparate Ġmod alities ). Ġ( 3 ) ĠWe Ġshow Ġthat Ġthe Ġproposed Ġapproaches Ġoutper form Ġthe Ġstate - of - the - art ĠN ER Ġmodels Ġ( both Ġwith Ġand Ġwithout Ġusing Ġadditional Ġvisual Ġcontexts ) Ġon Ġour Ġnew ĠM NER Ġdataset ĠSnap Capt ions , Ġa Ġlarge Ġcollection Ġof Ġinformal Ġand Ġextremely Ġshort Ġsocial Ġmedia Ġposts Ġpaired Ġwith Ġunique Ġimages . </s> Related ĠWork </s> Ne ural Ġmodels Ġfor ĠN ER Ġhave Ġbeen Ġrecently Ġproposed , Ġproducing Ġstate - of - the - art Ġperformance Ġon Ġstandard ĠN ER Ġtasks . ĠFor Ġexample , Ġsome Ġof Ġthe Ġend - to - end ĠN ER Ġsystems ĠB IB REF 4 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġuse Ġa Ġrecurrent Ġneural Ġnetwork Ġusually Ġwith Ġa ĠCR F ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġfor Ġsequence Ġlabeling , Ġaccompanied Ġwith Ġfeature Ġextract ors Ġfor Ġwords Ġand Ġcharacters Ġ( CNN , ĠL ST Ms , Ġetc .), Ġand Ġachieve Ġthe Ġstate - of - the - art Ġperformance Ġmostly Ġwithout Ġany Ġuse Ġof Ġg az ette ers Ġinformation . ĠNote Ġthat Ġmost Ġof Ġthese Ġwork Ġaggregate Ġtextual Ġcontexts Ġvia Ġconc aten ation Ġof Ġword Ġembed d ings Ġand Ġcharacter Ġembed d ings . ĠRecently , Ġseveral Ġwork Ġhave Ġaddressed Ġthe ĠN ER Ġtask Ġspecifically Ġon Ġnoisy Ġshort Ġtext Ġsegments Ġsuch Ġas ĠTwe ets , Ġetc . ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠThey Ġreport Ġperformance Ġgains Ġfrom Ġleveraging Ġexternal Ġsources Ġof Ġinformation Ġsuch Ġas Ġlex ical Ġinformation Ġ( POS Ġtags , Ġetc .) Ġand / or Ġfrom Ġseveral Ġpre processing Ġsteps Ġ( token Ġsubstitution , Ġetc .). ĠOur Ġmodel Ġbuilds Ġupon Ġthese Ġstate - of - the - art Ġneural Ġmodels Ġfor ĠN ER Ġtasks , Ġand Ġimproves Ġthe Ġmodel Ġin Ġtwo Ġcritical Ġways : Ġ( 1 ) Ġincorporation Ġof Ġvisual Ġcontexts Ġto Ġprovide Ġauxiliary Ġinformation Ġfor Ġshort Ġmedia Ġposts , Ġand Ġ( 2 ) Ġaddition Ġof Ġthe Ġmod ality Ġattention Ġmodule , Ġwhich Ġbetter Ġincorporates Ġword Ġembed d ings Ġand Ġcharacter Ġembed d ings , Ġespecially Ġwhen Ġthere Ġare Ġmany Ġmissing Ġtokens Ġin Ġthe Ġgiven Ġword Ġembed ding Ġmatrix . ĠNote Ġthat Ġwe Ġdo Ġnot Ġexplore Ġthe Ġuse Ġof Ġg az ette ers Ġinformation Ġor Ġother Ġauxiliary Ġinformation Ġ( POS Ġtags , Ġetc .) ĠB IB REF 9 Ġas Ġit Ġis Ġnot Ġthe Ġfocus Ġof Ġour Ġstudy . </s> Att ention Ġmodules Ġare Ġwidely Ġapplied Ġin Ġseveral Ġdeep Ġlearning Ġtasks ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠFor Ġexample , Ġthey Ġuse Ġan Ġattention Ġmodule Ġto Ġattend Ġto Ġa Ġsubset Ġwithin Ġa Ġsingle Ġinput Ġ( a Ġpart / region Ġof Ġan Ġimage , Ġa Ġspecific Ġtoken Ġin Ġan Ġinput Ġsequence Ġof Ġtokens , Ġetc .) Ġat Ġeach Ġdecoding Ġstep Ġin Ġan Ġenc oder - dec oder Ġframework Ġfor Ġimage Ġcaption ing Ġtasks , Ġetc . ĠB IB REF 14 Ġexplore Ġvarious Ġattention Ġmechanisms Ġin ĠN LP Ġtasks , Ġbut Ġdo Ġnot Ġincorporate Ġvisual Ġcomponents Ġor Ġinvestigate Ġthe Ġimpact Ġof Ġsuch Ġmodels Ġon Ġnoisy Ġsocial Ġmedia Ġdata . ĠB IB REF 15 Ġpropose Ġto Ġuse Ġattention Ġfor Ġa Ġsubset Ġof Ġdiscrete Ġsource Ġsamples Ġin Ġtransfer Ġlearning Ġsettings . ĠOur Ġmod ality Ġattention Ġdiffers Ġfrom Ġthe Ġprevious Ġapproaches Ġin Ġthat Ġwe Ġatten uate Ġor Ġampl ifies Ġeach Ġmod ality Ġinput Ġas Ġa Ġwhole Ġamong Ġmultiple Ġavailable Ġmod alities , Ġand Ġthat Ġwe Ġuse Ġthe Ġattention Ġmechanism Ġessentially Ġto Ġmap Ġheter ogeneous Ġmod alities Ġin Ġa Ġsingle Ġjoint Ġembed ding Ġspace . ĠOur Ġapproach Ġalso Ġallows Ġfor Ġre - use Ġof Ġthe Ġsame Ġmodel Ġfor Ġpredicting Ġlabels Ġeven Ġwhen Ġsome Ġof Ġthe Ġmod alities Ġare Ġmissing Ġin Ġinput , Ġas Ġother Ġmod alities Ġwould Ġstill Ġpreserve Ġthe Ġsame Ġsemantics Ġin Ġthe Ġembed d ings Ġspace . </s> Mult im od al Ġlearning Ġis Ġstudied Ġin Ġvarious Ġdomains Ġand Ġapplications , Ġaimed Ġat Ġbuilding Ġa Ġjoint Ġmodel Ġthat Ġextracts Ġcontextual Ġinformation Ġfrom Ġmultiple Ġmod alities Ġ( views ) Ġof Ġparallel Ġdatasets . </s> The Ġmost Ġrelevant Ġtask Ġto Ġour Ġmultim od al ĠN ER Ġsystem Ġis Ġthe Ġtask Ġof Ġmultim od al Ġmachine Ġtranslation ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, Ġwhich Ġaims Ġat Ġbuilding Ġa Ġbetter Ġmachine Ġtranslation Ġsystem Ġby Ġtaking Ġas Ġinput Ġa Ġsentence Ġin Ġa Ġsource Ġlanguage Ġas Ġwell Ġas Ġa Ġcorresponding Ġimage . ĠSeveral Ġstandard Ġsequence - to - sequence Ġarchitectures Ġare Ġexplored Ġ( a Ġtarget - language ĠL ST M Ġdec oder Ġthat Ġtakes Ġas Ġinput Ġan Ġimage Ġfirst ). ĠOther Ġprevious Ġliterature Ġinclude Ġstudy Ġof ĠCanon ical ĠCor relation ĠAnalysis Ġ( CC A ) ĠB IB REF 18 Ġto Ġlearn Ġfeature Ġcorrelations Ġamong Ġmultiple Ġmod alities , Ġwhich Ġis Ġwidely Ġused Ġin Ġmany Ġapplications . ĠOther Ġapplications Ġinclude Ġimage Ġcaption ing ĠB IB REF 10 Ġ, Ġaudio - visual Ġrecognition ĠB IB REF 19 Ġ, Ġvisual Ġquestion Ġanswering Ġsystems ĠB IB REF 20 Ġ, Ġetc . </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġour Ġapproach Ġis Ġthe Ġfirst Ġwork Ġto Ġincorporate Ġvisual Ġcontexts Ġfor Ġnamed Ġentity Ġrecognition Ġtasks . </s> Prop osed ĠMethods </s> Figure ĠFIG REF 2 Ġillustrates Ġthe Ġproposed Ġmultim od al ĠN ER Ġ( M NER ) Ġmodel . ĠFirst , Ġwe Ġobtain Ġword Ġembed d ings , Ġcharacter Ġembed d ings , Ġand Ġvisual Ġfeatures Ġ( Section ĠSEC REF 3 Ġ). ĠA ĠBi - L ST M - CR F Ġmodel Ġthen Ġtakes Ġas Ġinput Ġa Ġsequence Ġof Ġtokens , Ġeach Ġof Ġwhich Ġcomprises Ġa Ġword Ġtoken , Ġa Ġcharacter Ġsequence , Ġand Ġan Ġimage , Ġin Ġtheir Ġrespective Ġrepresentation Ġ( Section ĠSEC REF 4 Ġ). ĠAt Ġeach Ġdecoding Ġstep , Ġrepresentations Ġfrom Ġeach Ġmod ality Ġare Ġcombined Ġvia Ġthe Ġmod ality Ġattention Ġmodule Ġto Ġproduce Ġan Ġentity Ġlabel Ġfor Ġeach Ġtoken Ġ( ĠSEC REF 5 Ġ). ĠWe Ġformulate Ġeach Ġcomponent Ġof Ġthe Ġmodel Ġin Ġthe Ġfollowing Ġsubsections . </s> Not ations : ĠLet ĠIN LINE FORM 0 Ġa Ġsequence Ġof Ġinput Ġtokens Ġwith Ġlength ĠIN LINE FORM 1 Ġ, Ġwith Ġa Ġcorresponding Ġlabel Ġsequence ĠIN LINE FORM 2 Ġindicating Ġnamed Ġentities Ġ( e . g . Ġin Ġstandard ĠB IO Ġformats ). ĠEach Ġinput Ġtoken Ġis Ġcomposed Ġof Ġthree Ġmod alities : ĠIN LINE FORM 3 Ġfor Ġword Ġembed d ings , Ġcharacter Ġembed d ings , Ġand Ġvisual Ġembed d ings Ġrepresentations , Ġrespectively . </s> Features </s> Similar Ġto Ġthe Ġstate - of - the - art ĠN ER Ġapproaches ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġwe Ġuse Ġboth Ġword Ġembed d ings Ġand Ġcharacter Ġembed d ings . </s> Word Ġembed d ings Ġare Ġobtained Ġfrom Ġan Ġun super vised Ġlearning Ġmodel Ġthat Ġlearns Ġco - occ urrence Ġstatistics Ġof Ġwords Ġfrom Ġa Ġlarge Ġexternal Ġcorpus , Ġyielding Ġword Ġembed d ings Ġas Ġdistribution al Ġsemantics ĠB IB REF 21 Ġ. ĠSpecifically , Ġwe Ġuse Ġpre - trained Ġembed d ings Ġfrom ĠGl o VE ĠB IB REF 22 Ġ. </s> Character Ġembed d ings Ġare Ġobtained Ġfrom Ġa ĠBi - L ST M Ġwhich Ġtakes Ġas Ġinput Ġa Ġsequence Ġof Ġcharacters Ġof Ġeach Ġtoken , Ġsimilarly Ġto ĠB IB REF 0 Ġ. ĠAn Ġalternative Ġapproach Ġfor Ġobtaining Ġcharacter Ġembed d ings Ġis Ġusing Ġa Ġconv olution al Ġneural Ġnetwork Ġas Ġin ĠB IB REF 1 Ġ, Ġbut Ġwe Ġfind Ġthat ĠBi - L ST M Ġrepresentation Ġof Ġcharacters Ġyields Ġempir ically Ġbetter Ġresults Ġin Ġour Ġexperiments . </s> Visual Ġembed d ings : ĠTo Ġextract Ġfeatures Ġfrom Ġan Ġimage , Ġwe Ġtake Ġthe Ġfinal Ġhidden Ġlayer Ġrepresentation Ġof Ġa Ġmodified Ġversion Ġof Ġthe Ġconv olution al Ġnetwork Ġmodel Ġcalled ĠIn ception Ġ( Go og Le Net ) ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġtrained Ġon Ġthe ĠImage Net Ġdataset ĠB IB REF 25 Ġto Ġclassify Ġmultiple Ġobjects Ġin Ġthe Ġscene . ĠOur Ġimplementation Ġof Ġthe ĠIn ception Ġmodel Ġhas Ġdeep Ġ22 Ġlayers , Ġtraining Ġof Ġwhich Ġis Ġmade Ġpossible Ġvia ĠâĢ ľ network Ġin Ġnetwork " Ġprinciples Ġand Ġseveral Ġdimension Ġreduction Ġtechniques Ġto Ġimprove Ġcomputing Ġresource Ġutilization . ĠThe Ġfinal Ġlayer Ġrepresentation Ġenc odes Ġdiscrim in ative Ġinformation Ġdescribing Ġwhat Ġobjects Ġare Ġshown Ġin Ġan Ġimage , Ġwhich Ġprovide Ġauxiliary Ġcontexts Ġfor Ġunderstanding Ġtextual Ġtokens Ġand Ġentities Ġin Ġaccompanying Ġcapt ions . </s> In cor por ating Ġthis Ġvisual Ġinformation Ġonto Ġthe Ġtraditional ĠN ER Ġsystem Ġis Ġan Ġopen Ġchallenge , Ġand Ġmultiple Ġapproaches Ġcan Ġbe Ġconsidered . ĠFor Ġinstance , Ġone Ġmay Ġprovide Ġvisual Ġcontexts Ġonly Ġas Ġan Ġinitial Ġinput Ġto Ġdec oder Ġas Ġin Ġsome Ġenc oder - dec oder Ġimage Ġcaption ing Ġsystems ĠB IB REF 26 Ġ. ĠHowever , Ġwe Ġempir ically Ġobserve Ġthat Ġan ĠN ER Ġdec oder Ġwhich Ġtakes Ġas Ġinput Ġthe Ġvisual Ġembed d ings Ġat Ġevery Ġdecoding Ġstep Ġ( Section ĠSEC REF 4 Ġ), Ġcombined Ġwith Ġthe Ġmod ality Ġattention Ġmodule Ġ( Section ĠSEC REF 5 Ġ), Ġyields Ġbetter Ġresults . </s> Lastly , Ġwe Ġadd Ġa Ġtransform Ġlayer Ġfor Ġeach Ġfeature ĠIN LINE FORM 0 Ġbefore Ġit Ġis Ġfed Ġto Ġthe ĠN ER Ġentity ĠL ST M . </s> Bi - L ST M Ġ+ ĠCR F Ġfor ĠMult im od al ĠN ER </s> Our ĠM NER Ġmodel Ġis Ġbuilt Ġon Ġa ĠBi - L ST M Ġand ĠCR F Ġhybrid Ġmodel . ĠWe Ġuse Ġthe Ġfollowing Ġimplementation Ġfor Ġthe Ġentity ĠBi - L ST M . </s> it Ġ= Ġ( W xi ht - 1 Ġ+ ĠW c ict - 1 ) </s> ct Ġ= Ġ( 1 - it ) Ġc t - 1 </s> + Ġit Ġtan h ( W xc xt Ġ+ ĠWh cht - 1 ) </s> ot Ġ= Ġ( W x o xt Ġ+ ĠWh oh t - 1 Ġ+ ĠW co ct ) </s> ht Ġ= ĠL ST M ( xt ) </s> = Ġot Ġtan h ( ct ) </s> where ĠIN LINE FORM 0 Ġis Ġa Ġweighted Ġaverage Ġof Ġthree Ġmod alities ĠIN LINE FORM 1 Ġvia Ġthe Ġmod ality Ġattention Ġmodule , Ġwhich Ġwill Ġbe Ġdefined Ġin ĠSection ĠSEC REF 5 Ġ. ĠB ias Ġterms Ġfor Ġgates Ġare Ġomitted Ġhere Ġfor Ġsimplicity Ġof Ġnotation . </s> We Ġthen Ġobtain Ġbi - direction al Ġentity Ġtoken Ġrepresentations ĠIN LINE FORM 0 Ġby Ġconc aten ating Ġits Ġleft Ġand Ġright Ġcontext Ġrepresentations . ĠTo Ġenforce Ġstructural Ġcorrelations Ġbetween Ġlabels Ġin Ġsequence Ġdecoding , ĠIN LINE FORM 1 Ġis Ġthen Ġpassed Ġto Ġa Ġconditional Ġrandom Ġfield Ġ( CR F ) Ġto Ġproduce Ġa Ġlabel Ġfor Ġeach Ġtoken Ġmaximizing Ġthe Ġfollowing Ġobjective . Ġy * Ġ= Ġy Ġp ( y | h ; ĠW CR F ) </s> p ( y | h ; ĠW CR F ) Ġ= Ġt Ġt Ġ( yt - 1 , yt ; h ) Ġy ' Ġt Ġt Ġ( y 't - 1 , y 't ; h ) </s> where ĠIN LINE FORM 0 Ġis Ġa Ġpotential Ġfunction , ĠIN LINE FORM 1 Ġis Ġa Ġset Ġof Ġparameters Ġthat Ġdefines Ġthe Ġpotential Ġfunctions Ġand Ġweight Ġvectors Ġfor Ġlabel Ġpairs Ġ( ĠIN LINE FORM 2 Ġ). ĠB ias Ġterms Ġare Ġomitted Ġfor Ġbre vity Ġof Ġformulation . </s> The Ġmodel Ġcan Ġbe Ġtrained Ġvia Ġlog - like lihood Ġmaxim ization Ġfor Ġthe Ġtraining Ġset ĠIN LINE FORM 0 Ġ: </s> L ( W CR F ) Ġ= Ġi Ġp ( y | h ; ĠW ) </s> Mod ality ĠAttention </s> The Ġmod ality Ġattention Ġmodule Ġlearns Ġa Ġunified Ġrepresentation Ġspace Ġfor Ġmultiple Ġavailable Ġmod alities Ġ( words , Ġcharacters , Ġimages , Ġetc .), Ġand Ġproduces Ġa Ġsingle Ġvector Ġrepresentation Ġwith Ġaggreg ated Ġknowledge Ġamong Ġmultiple Ġmod alities , Ġbased Ġon Ġtheir Ġweighted Ġimportance . ĠWe Ġmotivate Ġthis Ġmodule Ġfrom Ġthe Ġfollowing Ġobservations . </s> A Ġmajority Ġof Ġthe Ġprevious Ġliterature Ġcombine Ġthe Ġword Ġand Ġcharacter - level Ġcontexts Ġby Ġsimply Ġconc aten ating Ġthe Ġword Ġand Ġcharacter Ġembed d ings Ġat Ġeach Ġdecoding Ġstep , Ġe . g . ĠIN LINE FORM 0 Ġin ĠE q . ĠSEC REF 4 Ġ. ĠHowever , Ġthis Ġnaive Ġconc aten ation Ġof Ġtwo Ġmod alities Ġ( word Ġand Ġcharacters ) Ġresults Ġin Ġinaccurate Ġdecoding , Ġspecifically Ġfor Ġunknown Ġword Ġtoken Ġembed d ings Ġ( an Ġall - zero Ġvector ĠIN LINE FORM 1 Ġor Ġa Ġrandom Ġvector ĠIN LINE FORM 2 Ġis Ġassigned Ġfor Ġany Ġunknown Ġtoken ĠIN LINE FORM 3 Ġ, Ġthus ĠIN LINE FORM 4 Ġor ĠIN LINE FORM 5 Ġ). ĠWhile Ġthis Ġconc aten ation Ġapproach Ġdoes Ġnot Ġcause Ġsignificant Ġerrors Ġfor Ġwell - form atted Ġtext , Ġwe Ġobserve Ġthat Ġit Ġinduces Ġperformance Ġdegradation Ġfor Ġour Ġsocial Ġmedia Ġpost Ġdatasets Ġwhich Ġcontain Ġa Ġsignificant Ġnumber Ġof Ġmissing Ġtokens . </s> Similarly , Ġnaive Ġmerging Ġof Ġtextual Ġand Ġvisual Ġinformation Ġ( ĠIN LINE FORM 0 Ġ) Ġyields Ġsub opt imal Ġresults Ġas Ġeach Ġmod ality Ġis Ġtreated Ġequally Ġinformative , Ġwhereas Ġin Ġour Ġdatasets Ġsome Ġof Ġthe Ġimages Ġmay Ġcontain Ġirrelevant Ġcontexts Ġto Ġtextual Ġmod alities . ĠHence , Ġideally Ġthere Ġneeds Ġa Ġmechanism Ġin Ġwhich Ġthe Ġmodel Ġcan Ġeffectively Ġturn Ġthe Ġswitch Ġon Ġand Ġoff Ġthe Ġmod alities Ġadaptive Ġto Ġeach Ġsample . </s> To Ġthis Ġend , Ġwe Ġpropose Ġa Ġgeneral Ġmod ality Ġattention Ġmodule , Ġwhich Ġadapt ively Ġatten uates Ġor Ġemphasizes Ġeach Ġmod ality Ġas Ġa Ġwhole Ġat Ġeach Ġdecoding Ġstep ĠIN LINE FORM 0 Ġ, Ġand Ġproduces Ġa Ġsoft - att ended Ġcontext Ġvector ĠIN LINE FORM 1 Ġas Ġan Ġinput Ġtoken Ġfor Ġthe Ġentity ĠL ST M . Ġ[ at ( w ), at ( c ), at ( v )] Ġ= Ġ( W m [ xt ( w ); Ġ xt ( c ); Ġ xt ( v )] Ġ+ Ġb m Ġ) </s> t ( m ) Ġ= Ġ( at ( m )) m ' { w , c , v } ( at ( m ' )) Ġm Ġ{ w , c , v } </s> xt Ġ= Ġm { w , c , v } Ġt ( m ) xt ( m ) </s> where ĠIN LINE FORM 0 Ġis Ġan Ġattention Ġvector Ġat Ġeach Ġdecoding Ġstep ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġis Ġa Ġfinal Ġcontext Ġvector Ġat ĠIN LINE FORM 3 Ġthat Ġmaxim izes Ġinformation Ġgain Ġfor ĠIN LINE FORM 4 Ġ. ĠNote Ġthat Ġthe Ġoptimization Ġof Ġthe Ġobjective Ġfunction Ġ( E q . ĠSEC REF 4 Ġ) Ġwith Ġmod ality Ġattention Ġ( E q . ĠSEC REF 5 Ġ) Ġrequires Ġeach Ġmod ality Ġto Ġhave Ġthe Ġsame Ġdimension Ġ( ĠIN LINE FORM 5 Ġ), Ġand Ġthat Ġthe Ġtransformation Ġvia ĠIN LINE FORM 6 Ġessentially Ġen forces Ġeach Ġmod ality Ġto Ġbe Ġmapped Ġinto Ġthe Ġsame Ġunified Ġsub space , Ġwhere Ġthe Ġweighted Ġaverage Ġof Ġwhich Ġenc odes Ġdiscrim itive Ġfeatures Ġfor Ġrecognition Ġof Ġnamed Ġentities . </s> When Ġvisual Ġcontext Ġis Ġnot Ġprovided Ġwith Ġeach Ġtoken Ġ( as Ġin Ġthe Ġtraditional ĠN ER Ġtask ), Ġwe Ġcan Ġdefine Ġthe Ġmod ality Ġattention Ġfor Ġword Ġand Ġcharacter Ġembed d ings Ġonly Ġin Ġa Ġsimilar Ġway : Ġ[ at ( w ), at ( c )] Ġ= Ġ( W m [ xt ( w ); Ġ xt ( c )] Ġ+ Ġb m Ġ) </s> t ( m ) Ġ= Ġ( at ( m )) m ' { w , c } ( at ( m ' )) Ġm Ġ{ w , c } </s> xt Ġ= Ġm { w , c } Ġt ( m ) xt ( m ) </s> Note Ġthat Ġwhile Ġwe Ġapply Ġthis Ġmod ality Ġattention Ġmodule Ġto Ġthe ĠBi - L ST M + CR F Ġarchitecture Ġ( Section ĠSEC REF 4 Ġ) Ġfor Ġits Ġempirical Ġsuperiority , Ġthe Ġmodule Ġitself Ġis Ġflexible Ġand Ġthus Ġcan Ġwork Ġwith Ġother ĠN ER Ġarchitectures Ġor Ġfor Ġother Ġmultim od al Ġapplications . </s> Snap Capt ions ĠDat as et </s> The ĠSnap Capt ions Ġdataset Ġis Ġcomposed Ġof Ġ10 K Ġuser - generated Ġimage Ġ( snap ) Ġand Ġtextual Ġcaption Ġpairs Ġwhere Ġnamed Ġentities Ġin Ġcapt ions Ġare Ġmanually Ġlabeled Ġby Ġexpert Ġhuman Ġannot ators Ġ( entity Ġtypes : ĠPER , ĠLOC , ĠOR G , ĠM ISC ). ĠThese Ġcapt ions Ġare Ġcollected Ġexclusively Ġfrom Ġsnaps Ġsubmitted Ġto Ġpublic Ġand Ġcrowd - s ourced Ġstories Ġ( aka ĠSnapchat ĠLive ĠStories Ġor ĠOur ĠStories ). ĠExamples Ġof Ġsuch Ġpublic Ġcrowd - s ourced Ġstories Ġare ĠâĢ ľ New ĠYork ĠStory âĢ Ŀ Ġor ĠâĢ ľ Thanks giving ĠStory âĢ Ŀ , Ġwhich Ġcomprise Ġsnaps Ġthat Ġare Ġaggreg ated Ġfor Ġvarious Ġpublic Ġevents , Ġvenues , Ġetc . ĠAll Ġsnaps Ġwere Ġposted Ġbetween Ġyear Ġ2016 Ġand Ġ2017 , Ġand Ġdo Ġnot Ġcontain Ġraw Ġimages Ġor Ġother Ġassociated Ġinformation Ġ( only Ġtextual Ġcapt ions Ġand Ġobfusc ated Ġvisual Ġdescriptor Ġfeatures Ġextracted Ġfrom Ġthe Ġpre - trained ĠIn ception Net Ġare Ġavailable ). ĠWe Ġsplit Ġthe Ġdataset Ġinto Ġtrain Ġ( 70 %), Ġvalidation Ġ( 15 %), Ġand Ġtest Ġsets Ġ( 15 %). ĠThe Ġcapt ions Ġdata Ġhave Ġaverage Ġlength Ġof Ġ30 . 7 Ġcharacters Ġ( 5 . 81 Ġwords ) Ġwith Ġvocabulary Ġsize Ġ15 , 733 , Ġwhere Ġ6 , 612 Ġare Ġconsidered Ġunknown Ġtokens Ġfrom ĠStanford ĠGl o VE Ġembed d ings ĠB IB REF 22 Ġ. ĠNamed Ġentities Ġannot ated Ġin Ġthe ĠSnap Capt ions Ġdataset Ġinclude Ġmany Ġof Ġnew Ġand Ġemerging Ġentities , Ġand Ġthey Ġare Ġfound Ġin Ġvarious Ġsurface Ġforms Ġ( var ious Ġnick names , Ġty pos , Ġetc .) ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , ĠSnap Capt ions Ġis Ġthe Ġonly Ġdataset Ġthat Ġcontains Ġnatural Ġimage - ca ption Ġpairs Ġwith Ġexpert - annot ated Ġnamed Ġentities . </s> Bas elines </s> Task : Ġgiven Ġa Ġcaption Ġand Ġa Ġpaired Ġimage Ġ( if Ġused ), Ġthe Ġgoal Ġis Ġto Ġlabel Ġevery Ġtoken Ġin Ġa Ġcaption Ġin ĠB IO Ġscheme Ġ( B : Ġbeginning , ĠI : Ġinside , ĠO : Ġoutside ) ĠB IB REF 27 Ġ. ĠWe Ġreport Ġthe Ġperformance Ġof Ġthe Ġfollowing Ġstate - of - the - art ĠN ER Ġmodels Ġas Ġbas elines , Ġas Ġwell Ġas Ġseveral Ġconfigurations Ġof Ġour Ġproposed Ġapproach Ġto Ġexamine Ġcontributions Ġof Ġeach Ġcomponent Ġ( W : Ġword , ĠC : Ġchar , ĠV : Ġvisual ). </s> Bi - L ST M / CR F Ġ( W Ġonly ): Ġonly Ġtakes Ġword Ġtoken Ġembed d ings Ġ( Stan ford ĠGl o VE ) Ġas Ġinput . ĠThe Ġrest Ġof Ġthe Ġarchitecture Ġis Ġkept Ġthe Ġsame . </s> Bi - L ST M / CR F Ġ+ ĠBi - Char L ST M Ġ( C Ġonly ): Ġonly Ġtakes Ġa Ġcharacter Ġsequence Ġof Ġeach Ġword Ġtoken Ġas Ġinput . Ġ( No Ġword Ġembed d ings ) </s> Bi - L ST M / CR F Ġ+ ĠBi - Char L ST M Ġ( W + C ) ĠB IB REF 0 Ġ: Ġtakes Ġas Ġinput Ġboth Ġword Ġembed d ings Ġand Ġcharacter Ġembed d ings Ġextracted Ġfrom Ġa ĠBi - Char L ST M . ĠEntity ĠL ST M Ġtakes Ġconc aten ated Ġvectors Ġof Ġword Ġand Ġcharacter Ġembed d ings Ġas Ġinput Ġtokens . </s> Bi - L ST M / CR F Ġ+ ĠChar CNN Ġ( W + C ) ĠB IB REF 1 Ġ: Ġuses Ġcharacter Ġembed d ings Ġextracted Ġfrom Ġa ĠCNN Ġinstead . </s> Bi - L ST M / CR F Ġ+ ĠChar CNN Ġ( W + C ) Ġ+ ĠMulti - task ĠB IB REF 8 Ġ: Ġtrains Ġthe Ġmodel Ġto Ġperform Ġboth Ġrecognition Ġ( into Ġmultiple Ġentity Ġtypes ) Ġas Ġwell Ġas Ġsegment ation Ġ( binary ) Ġtasks . </s> ( prop osed ) ĠBi - L ST M / CR F Ġ+ ĠBi - Char L ST M Ġwith Ġmod ality Ġattention Ġ( W + C ): Ġuses Ġthe Ġmod ality Ġattention Ġto Ġmerge Ġword Ġand Ġcharacter Ġembed d ings . </s> ( prop osed ) ĠBi - L ST M / CR F Ġ+ ĠBi - Char L ST M Ġ+ ĠIn ception Ġ( W + C + V ): Ġtakes Ġas Ġinput Ġvisual Ġcontexts Ġextracted Ġfrom ĠIn ception Net Ġas Ġwell , Ġconc aten ated Ġwith Ġword Ġand Ġchar Ġvectors . </s> ( prop osed ) ĠBi - L ST M / CR F Ġ+ ĠBi - Char L ST M Ġ+ ĠIn ception Ġwith Ġmod ality Ġattention Ġ( W + C + V ): Ġuses Ġthe Ġmod ality Ġattention Ġto Ġmerge Ġword , Ġcharacter , Ġand Ġvisual Ġembed d ings Ġas Ġinput Ġto Ġentity ĠL ST M . </s> Results : ĠSnap Capt ions ĠDat as et </s> Table ĠT AB REF 6 Ġshows Ġthe ĠN ER Ġperformance Ġon Ġthe ĠSnap ĠCapt ions Ġdataset . ĠWe Ġreport Ġboth Ġentity Ġtypes Ġrecognition Ġ( PER , ĠLOC , ĠOR G , ĠM ISC ) Ġand Ġnamed Ġentity Ġsegment ation Ġ( named Ġentity Ġor Ġnot ) Ġresults . </s> Parameters : ĠWe Ġtune Ġthe Ġparameters Ġof Ġeach Ġmodel Ġwith Ġthe Ġfollowing Ġsearch Ġspace Ġ( bold Ġindicate Ġthe Ġchoice Ġfor Ġour Ġfinal Ġmodel ): Ġcharacter Ġembed d ings Ġdimension : Ġ{ 25 , Ġ50 , Ġ100 , Ġ150 , Ġ200 , Ġ300 }, Ġword Ġembed d ings Ġsize : Ġ{ 25 , Ġ50 , Ġ100 , Ġ150 , Ġ200 , Ġ300 }, ĠL ST M Ġhidden Ġstates : Ġ{ 25 , Ġ50 , Ġ100 , Ġ150 , Ġ200 , Ġ300 }, Ġand ĠIN LINE FORM 0 Ġdimension : Ġ{ 25 , Ġ50 , Ġ100 , Ġ150 , Ġ200 , Ġ300 }. ĠWe Ġoptimize Ġthe Ġparameters Ġwith ĠAd ag rad ĠB IB REF 28 Ġwith Ġbatch Ġsize Ġ10 , Ġlearning Ġrate Ġ0 . 02 , Ġe ps ilon ĠIN LINE FORM 1 Ġ, Ġand Ġdecay Ġ0 . 0 . </s> Main ĠResults : ĠWhen Ġvisual Ġcontext Ġis Ġavailable Ġ( W + C + V ), Ġwe Ġsee Ġthat Ġthe Ġmodel Ġperformance Ġgreatly Ġimproves Ġover Ġthe Ġtextual Ġmodels Ġ( W + C ), Ġshowing Ġthat Ġvisual Ġcontexts Ġare Ġcomplimentary Ġto Ġtextual Ġinformation Ġin Ġnamed Ġentity Ġrecognition Ġtasks . ĠIn Ġaddition , Ġit Ġcan Ġbe Ġseen Ġthat Ġthe Ġmod ality Ġattention Ġmodule Ġfurther Ġimproves Ġthe Ġentity Ġtype Ġrecognition Ġperformance Ġfor Ġ( W + C + V ). ĠThis Ġresult Ġindicates Ġthat Ġthe Ġmod ality Ġattention Ġis Ġable Ġto Ġfocus Ġon Ġthe Ġmost Ġeffective Ġmod ality Ġ( visual , Ġwords , Ġor Ġcharacters ) Ġadaptive Ġto Ġeach Ġsample Ġto Ġmaximize Ġinformation Ġgain . ĠNote Ġthat Ġour Ġtext - only Ġmodel Ġ( W + C ) Ġwith Ġthe Ġmod ality Ġattention Ġmodule Ġalso Ġsignificantly Ġoutper form Ġthe Ġstate - of - the - art Ġbas elines ĠB IB REF 8 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 0 Ġthat Ġuse Ġthe Ġsame Ġtextual Ġmod alities Ġ( W + C ), Ġshowing Ġthe Ġeffectiveness Ġof Ġthe Ġmod ality Ġattention Ġmodule Ġfor Ġtextual Ġmodels Ġas Ġwell . </s> Error ĠAnalysis : ĠTable ĠT AB REF 17 Ġshows Ġexample Ġcases Ġwhere Ġincorporation Ġof Ġvisual Ġcontexts Ġaffects Ġprediction Ġof Ġnamed Ġentities . ĠFor Ġexample , Ġthe Ġtoken Ġ` c urry ' Ġin Ġthe Ġcaption ĠâĢ ľ The Ġcurry 's Ġ" Ġis Ġpoly sem ous Ġand Ġmay Ġrefer Ġto Ġeither Ġa Ġtype Ġof Ġfood Ġor Ġa Ġfamous Ġbasketball Ġplayer Ġ` Stephen ĠCurry ', Ġand Ġthe Ġsurrounding Ġtextual Ġcontexts Ġdo Ġnot Ġprovide Ġenough Ġinformation Ġto Ġdis amb ig uate Ġit . ĠOn Ġthe Ġother Ġhand , Ġvisual Ġcontexts Ġ( visual Ġtags : Ġ` par ade ', Ġ` urban Ġarea ', Ġ... ) Ġprovide Ġsimilarities Ġto Ġthe Ġtoken 's Ġdistribution al Ġsemantics Ġfrom Ġother Ġtraining Ġexamples Ġ( sn aps Ġfrom ĠâĢ ľ NBA ĠChampionship ĠParade ĠStory "), Ġand Ġthus Ġthe Ġmodel Ġsuccessfully Ġpredicts Ġthe Ġtoken Ġas Ġa Ġnamed Ġentity . ĠSimilarly , Ġwhile Ġthe Ġtext - only Ġmodel Ġerrone ously Ġpredicts Ġ` Apple ' Ġin Ġthe Ġcaption ĠâĢ ľ Grand ma Ġw Ġdat Ġlit ĠApple ĠCris p " Ġas Ġan Ġorganization Ġ( Apple ĠInc .), Ġthe Ġvisual Ġcontexts Ġ( desc ribing Ġobjects Ġrelated Ġto Ġfood ) Ġhelp Ġdis amb ig uate Ġthe Ġtoken , Ġmaking Ġthe Ġmodel Ġpredict Ġit Ġcorrectly Ġas Ġa Ġnon - named Ġentity Ġ( a Ġfruit ). ĠTrend ing Ġentities Ġ( mus icians Ġor ĠDJs Ġsuch Ġas Ġ` C ID ', Ġ` D uke ĠDum ont ', Ġ` Marsh m ello ', Ġetc .) Ġare Ġalso Ġrecognized Ġcorrectly Ġwith Ġstrengthened Ġcontexts Ġfrom Ġvisual Ġinformation Ġ( desc ribing Ġconcert Ġscenes ) Ġdespite Ġlack Ġof Ġsurrounding Ġtextual Ġcontexts . ĠA Ġfew Ġcases Ġwhere Ġvisual Ġcontexts Ġharmed Ġthe Ġperformance Ġmostly Ġinclude Ġvisual Ġtags Ġthat Ġare Ġunrelated Ġto Ġa Ġtoken Ġor Ġits Ġsurrounding Ġtextual Ġcontexts . </s> Visual ization Ġof ĠMod ality ĠAttention : ĠFigure ĠFIG REF 19 Ġvisual izes Ġthe Ġmod ality Ġattention Ġmodule Ġat Ġeach Ġdecoding Ġstep Ġ( each Ġcolumn ), Ġwhere Ġamplified Ġmod ality Ġis Ġrepresented Ġwith Ġdarker Ġcolor , Ġand Ġatten uated Ġmod ality Ġis Ġrepresented Ġwith Ġlighter Ġcolor . </s> For Ġthe Ġimage - a ided Ġmodel Ġ( W + C + V ; Ġupper Ġrow Ġin ĠFigure ĠFIG REF 19 Ġ), Ġwe Ġconfirm Ġthat Ġthe Ġmod ality Ġattention Ġsuccessfully Ġatten uates Ġirrelevant Ġsignals Ġ( self ies , Ġetc .) Ġand Ġampl ifies Ġrelevant Ġmod ality - based Ġcontexts Ġin Ġprediction Ġof Ġa Ġgiven Ġtoken . ĠIn Ġthe Ġexample Ġof ĠâĢ ľ dis ney Ġword Ġessential Ġ= Ġcoffee " Ġwith Ġvisual Ġtags Ġselfie , Ġphone , Ġperson , Ġthe Ġmod ality Ġattention Ġsuccessfully Ġatten uates Ġdistracting Ġvisual Ġsignals Ġand Ġfocuses Ġon Ġtextual Ġmod alities , Ġconsequently Ġmaking Ġcorrect Ġpredictions . ĠThe Ġnamed Ġentities Ġin Ġthe Ġexamples Ġof ĠâĢ ľ Beaut iful Ġnight Ġatop ĠThe ĠSpace ĠNeed le " Ġand ĠâĢ ľ Spl ash ĠMountain " Ġare Ġchallenging Ġto Ġpredict Ġbecause Ġthey Ġare Ġcomposed Ġof Ġcommon Ġnoun s Ġ( space , Ġneedle , Ġsplash , Ġmountain ), Ġand Ġthus Ġthey Ġoften Ġneed Ġadditional Ġcontexts Ġto Ġcorrectly Ġpredict . ĠIn Ġthe Ġtraining Ġdata , Ġvisual Ġcontexts Ġmake Ġstronger Ġindicators Ġfor Ġthese Ġnamed Ġentities Ġ( space Ġneedle , Ġsplash Ġmountain ), Ġand Ġthe Ġmod ality Ġattention Ġmodule Ġsuccessfully Ġattends Ġmore Ġto Ġstronger Ġsignals . </s> For Ġtext - only Ġmodel Ġ( W + C ), Ġwe Ġobserve Ġthat Ġperformance Ġgains Ġmostly Ġcome Ġfrom Ġthe Ġmod ality Ġattention Ġmodule Ġbetter Ġhandling Ġtokens Ġunseen Ġduring Ġtraining Ġor Ġunknown Ġtokens Ġfrom Ġthe Ġpre - trained Ġword Ġembed d ings Ġmatrix . ĠFor Ġexample , Ġwhile ĠWa R ri O o O rs Ġand ĠK ool er ĠM atic Ġare Ġmissing Ġtokens Ġin Ġthe Ġword Ġembed d ings Ġmatrix , Ġit Ġsuccessfully Ġampl ifies Ġcharacter - based Ġcontexts Ġ( capital ized Ġfirst Ġletters , Ġsimilarity Ġto Ġknown Ġentities Ġ` Golden ĠState ĠWarriors ') Ġand Ġsupp resses Ġword - based Ġcontexts Ġ( word Ġembed d ings Ġfor Ġunknown Ġtokens Ġ` Wa R ri O o O rs '), Ġleading Ġto Ġcorrect Ġpredictions . ĠThis Ġresult Ġis Ġsignificant Ġbecause Ġit Ġshows Ġperformance Ġof Ġthe Ġmodel , Ġwith Ġan Ġalmost Ġidentical Ġarchitecture , Ġcan Ġstill Ġimprove Ġwithout Ġhaving Ġto Ġscale Ġthe Ġword Ġembed d ings Ġmatrix Ġindefinitely . </s> Figure ĠFIG REF 19 Ġ( b ) Ġshows Ġthe Ġcases Ġwhere Ġthe Ġmod ality Ġattention Ġled Ġto Ġincorrect Ġpredictions . ĠFor Ġexample , Ġthe Ġmodel Ġpredicts Ġmissing Ġtokens ĠH UU UGE Ġand ĠSh ampoo er Ġincorrectly Ġas Ġnamed Ġentities Ġby Ġampl ifying Ġmisleading Ġcharacter - based Ġcontexts Ġ( capital ized Ġfirst Ġletters ) Ġor Ġvisual Ġcontexts Ġ( concert Ġscenes , Ġassociated Ġcontexts Ġof Ġwhich Ġoften Ġinclude Ġnamed Ġentities Ġin Ġthe Ġtraining Ġdataset ). </s> S ensitivity Ġto ĠWord ĠEmb edd ings ĠVoc abulary ĠSize : ĠIn Ġorder Ġto Ġisolate Ġthe Ġeffectiveness Ġof Ġthe Ġmod ality Ġattention Ġmodule Ġon Ġtextual Ġmodels Ġin Ġhandling Ġmissing Ġtokens , Ġwe Ġreport Ġthe Ġperformance Ġwith Ġvarying Ġword Ġembed d ings Ġvocabulary Ġsizes Ġin ĠTable ĠT AB REF 20 Ġ. ĠBy Ġincreasing Ġthe Ġnumber Ġof Ġmissing Ġtokens Ġartificially Ġby Ġrandomly Ġremoving Ġwords Ġfrom Ġthe Ġword Ġembed d ings Ġmatrix Ġ( original Ġvoc ab Ġsize : Ġ400 K ), Ġwe Ġobserve Ġthat Ġwhile Ġthe Ġoverall Ġperformance Ġde grades , Ġthe Ġmod ality Ġattention Ġmodule Ġis Ġable Ġto Ġsuppress Ġthe Ġpe formance Ġdegradation . ĠNote Ġalso Ġthat Ġthe Ġperformance Ġgap Ġgenerally Ġgets Ġbigger Ġas Ġwe Ġdecrease Ġthe Ġvocabulary Ġsize Ġof Ġthe Ġword Ġembed d ings Ġmatrix . ĠThis Ġresult Ġis Ġsignificant Ġin Ġthat Ġthe Ġmod ality Ġattention Ġis Ġable Ġto Ġimprove Ġthe Ġmodel Ġmore Ġrobust Ġto Ġmissing Ġtokens Ġwithout Ġhaving Ġto Ġtrain Ġan Ġindefinitely Ġlarge Ġword Ġembed d ings Ġmatrix Ġfor Ġarbitrarily Ġnoisy Ġsocial Ġmedia Ġtext Ġdatasets . </s> Con clusions </s> We Ġproposed Ġa Ġnew Ġmultim od al ĠN ER Ġ( M NER : Ġimage Ġ+ Ġtext ) Ġtask Ġon Ġshort Ġsocial Ġmedia Ġposts . ĠWe Ġdemonstrated Ġfor Ġthe Ġfirst Ġtime Ġan Ġeffective ĠM NER Ġsystem , Ġwhere Ġvisual Ġinformation Ġis Ġcombined Ġwith Ġtextual Ġinformation Ġto Ġoutper form Ġtraditional Ġtext - based ĠN ER Ġbas elines . ĠOur Ġwork Ġcan Ġbe Ġapplied Ġto Ġmy ri ads Ġof Ġsocial Ġmedia Ġposts Ġor Ġother Ġarticles Ġacross Ġmultiple Ġplatforms Ġwhich Ġoften Ġinclude Ġboth Ġtext Ġand Ġaccompanying Ġimages . ĠIn Ġaddition , Ġwe Ġproposed Ġthe Ġmod ality Ġattention Ġmodule , Ġa Ġnew Ġneural Ġmechanism Ġwhich Ġlearns Ġoptimal Ġintegration Ġof Ġdifferent Ġmodes Ġof Ġcorrelated Ġinformation . ĠIn Ġessence , Ġthe Ġmod ality Ġattention Ġlearns Ġto Ġatten uate Ġirrelevant Ġor Ġunin form ative Ġmod al Ġinformation Ġwhile Ġampl ifying Ġthe Ġprimary Ġmod ality Ġto Ġextract Ġbetter Ġoverall Ġrepresentations . ĠWe Ġshowed Ġthat Ġthe Ġmod ality Ġattention Ġbased Ġmodel Ġoutper forms Ġother Ġstate - of - the - art Ġbas elines Ġwhen Ġtext Ġwas Ġthe Ġonly Ġmod ality Ġavailable , Ġby Ġbetter Ġcombining Ġword Ġand Ġcharacter Ġlevel Ġinformation .
Introduction Social media with abundant user-generated posts provide a rich platform for understanding events, opinions and preferences of groups and individuals. These insights are primarily hidden in unstructured forms of social media posts, such as in free-form text or images without tags. Named entity recognition (NER), the task of recognizing named entities from free-form text, is thus a critical step for building structural information, allowing for its use in personalized assistance, recommendations, advertisement, etc. While many previous approaches BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 on NER have shown success for well-formed text in recognizing named entities via word context resolution (e.g. LSTM with word embeddings) combined with character-level features (e.g. CharLSTM/CNN), several additional challenges remain for recognizing named entities from extremely short and coarse text found in social media posts. For instance, short social media posts often do not provide enough textual contexts to resolve polysemous entities (e.g. “monopoly is da best ", where `monopoly' may refer to a board game (named entity) or a term in economics). In addition, noisy text includes a huge number of unknown tokens due to inconsistent lexical notations and frequent mentions of various newly trending entities (e.g. “xoxo Marshmelloooo ", where `Marshmelloooo' is a mis-spelling of a known entity `Marshmello', a music producer), making word embeddings based neural networks NER models vulnerable. To address the challenges above for social media posts, we build upon the state-of-the-art neural architecture for NER with the following two novel approaches (Figure FIGREF1 ). First, we propose to leverage auxiliary modalities for additional context resolution of entities. For example, many popular social media platforms now provide ways to compose a post in multiple modalities - specifically image and text (e.g. Snapchat captions, Twitter posts with image URLs), from which we can obtain additional context for understanding posts. While “monopoly" in the previous example is ambiguous in its textual form, an accompanying snap image of a board game can help disambiguate among polysemous entities, thereby correctly recognizing it as a named entity. Second, we also propose a general modality attention module which chooses per decoding step the most informative modality among available ones (in our case, word embeddings, character embeddings, or visual features) to extract context from. For example, the modality attention module lets the decoder attenuate the word-level signals for unknown word tokens (“Marshmellooooo" with trailing `o's) and amplifies character-level features intsead (capitalized first letter, lexical similarity to other known named entity token `Marshmello', etc.), thereby suppressing noise information (“UNK" token embedding) in decoding steps. Note that most of the previous literature in NER or other NLP tasks combine word and character-level information with naive concatenation, which is vulnerable to noisy social media posts. When an auxiliary image is available, the modality attention module determines to amplify this visual context in disambiguating polysemous entities, or to attenuate visual contexts when they are irrelevant to target named entities, selfies, etc. Note that the proposed modality attention module is distinct from how attention is used in other sequence-to-sequence literature (e.g. attending to a specific token within an input sequence). Section SECREF2 provides the detailed literature review. Our contributions are three-fold: we propose (1) an LSTM-CNN hybrid multimodal NER network that takes as input both image and text for recognition of a named entity in text input. To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks. (2) We propose a general modality attention module that selectively chooses modalities to extract primary context from, maximizing information gain and suppressing irrelevant contexts from each modality (we treat words, characters, and images as separate modalities). (3) We show that the proposed approaches outperform the state-of-the-art NER models (both with and without using additional visual contexts) on our new MNER dataset SnapCaptions, a large collection of informal and extremely short social media posts paired with unique images. Related Work Neural models for NER have been recently proposed, producing state-of-the-art performance on standard NER tasks. For example, some of the end-to-end NER systems BIBREF4 , BIBREF2 , BIBREF3 , BIBREF0 , BIBREF1 use a recurrent neural network usually with a CRF BIBREF5 , BIBREF6 for sequence labeling, accompanied with feature extractors for words and characters (CNN, LSTMs, etc.), and achieve the state-of-the-art performance mostly without any use of gazetteers information. Note that most of these work aggregate textual contexts via concatenation of word embeddings and character embeddings. Recently, several work have addressed the NER task specifically on noisy short text segments such as Tweets, etc. BIBREF7 , BIBREF8 . They report performance gains from leveraging external sources of information such as lexical information (POS tags, etc.) and/or from several preprocessing steps (token substitution, etc.). Our model builds upon these state-of-the-art neural models for NER tasks, and improves the model in two critical ways: (1) incorporation of visual contexts to provide auxiliary information for short media posts, and (2) addition of the modality attention module, which better incorporates word embeddings and character embeddings, especially when there are many missing tokens in the given word embedding matrix. Note that we do not explore the use of gazetteers information or other auxiliary information (POS tags, etc.) BIBREF9 as it is not the focus of our study. Attention modules are widely applied in several deep learning tasks BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 . For example, they use an attention module to attend to a subset within a single input (a part/region of an image, a specific token in an input sequence of tokens, etc.) at each decoding step in an encoder-decoder framework for image captioning tasks, etc. BIBREF14 explore various attention mechanisms in NLP tasks, but do not incorporate visual components or investigate the impact of such models on noisy social media data. BIBREF15 propose to use attention for a subset of discrete source samples in transfer learning settings. Our modality attention differs from the previous approaches in that we attenuate or amplifies each modality input as a whole among multiple available modalities, and that we use the attention mechanism essentially to map heterogeneous modalities in a single joint embedding space. Our approach also allows for re-use of the same model for predicting labels even when some of the modalities are missing in input, as other modalities would still preserve the same semantics in the embeddings space. Multimodal learning is studied in various domains and applications, aimed at building a joint model that extracts contextual information from multiple modalities (views) of parallel datasets. The most relevant task to our multimodal NER system is the task of multimodal machine translation BIBREF16 , BIBREF17 , which aims at building a better machine translation system by taking as input a sentence in a source language as well as a corresponding image. Several standard sequence-to-sequence architectures are explored (a target-language LSTM decoder that takes as input an image first). Other previous literature include study of Canonical Correlation Analysis (CCA) BIBREF18 to learn feature correlations among multiple modalities, which is widely used in many applications. Other applications include image captioning BIBREF10 , audio-visual recognition BIBREF19 , visual question answering systems BIBREF20 , etc. To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks. Proposed Methods Figure FIGREF2 illustrates the proposed multimodal NER (MNER) model. First, we obtain word embeddings, character embeddings, and visual features (Section SECREF3 ). A Bi-LSTM-CRF model then takes as input a sequence of tokens, each of which comprises a word token, a character sequence, and an image, in their respective representation (Section SECREF4 ). At each decoding step, representations from each modality are combined via the modality attention module to produce an entity label for each token ( SECREF5 ). We formulate each component of the model in the following subsections. Notations: Let INLINEFORM0 a sequence of input tokens with length INLINEFORM1 , with a corresponding label sequence INLINEFORM2 indicating named entities (e.g. in standard BIO formats). Each input token is composed of three modalities: INLINEFORM3 for word embeddings, character embeddings, and visual embeddings representations, respectively. Features Similar to the state-of-the-art NER approaches BIBREF0 , BIBREF1 , BIBREF8 , BIBREF4 , BIBREF2 , BIBREF3 , we use both word embeddings and character embeddings. Word embeddings are obtained from an unsupervised learning model that learns co-occurrence statistics of words from a large external corpus, yielding word embeddings as distributional semantics BIBREF21 . Specifically, we use pre-trained embeddings from GloVE BIBREF22 . Character embeddings are obtained from a Bi-LSTM which takes as input a sequence of characters of each token, similarly to BIBREF0 . An alternative approach for obtaining character embeddings is using a convolutional neural network as in BIBREF1 , but we find that Bi-LSTM representation of characters yields empirically better results in our experiments. Visual embeddings: To extract features from an image, we take the final hidden layer representation of a modified version of the convolutional network model called Inception (GoogLeNet) BIBREF23 , BIBREF24 trained on the ImageNet dataset BIBREF25 to classify multiple objects in the scene. Our implementation of the Inception model has deep 22 layers, training of which is made possible via “network in network" principles and several dimension reduction techniques to improve computing resource utilization. The final layer representation encodes discriminative information describing what objects are shown in an image, which provide auxiliary contexts for understanding textual tokens and entities in accompanying captions. Incorporating this visual information onto the traditional NER system is an open challenge, and multiple approaches can be considered. For instance, one may provide visual contexts only as an initial input to decoder as in some encoder-decoder image captioning systems BIBREF26 . However, we empirically observe that an NER decoder which takes as input the visual embeddings at every decoding step (Section SECREF4 ), combined with the modality attention module (Section SECREF5 ), yields better results. Lastly, we add a transform layer for each feature INLINEFORM0 before it is fed to the NER entity LSTM. Bi-LSTM + CRF for Multimodal NER Our MNER model is built on a Bi-LSTM and CRF hybrid model. We use the following implementation for the entity Bi-LSTM. it = (Wxiht-1 + Wcict-1) ct = (1-it) ct-1 + it tanh(Wxcxt + Whcht-1) ot = (Wxoxt + Whoht-1 + Wcoct) ht = LSTM(xt) = ot tanh(ct) where INLINEFORM0 is a weighted average of three modalities INLINEFORM1 via the modality attention module, which will be defined in Section SECREF5 . Bias terms for gates are omitted here for simplicity of notation. We then obtain bi-directional entity token representations INLINEFORM0 by concatenating its left and right context representations. To enforce structural correlations between labels in sequence decoding, INLINEFORM1 is then passed to a conditional random field (CRF) to produce a label for each token maximizing the following objective. y* = y p(y|h; WCRF) p(y|h; WCRF) = t t (yt-1,yt;h) y' t t (y't-1,y't;h) where INLINEFORM0 is a potential function, INLINEFORM1 is a set of parameters that defines the potential functions and weight vectors for label pairs ( INLINEFORM2 ). Bias terms are omitted for brevity of formulation. The model can be trained via log-likelihood maximization for the training set INLINEFORM0 : L(WCRF) = i p(y|h; W) Modality Attention The modality attention module learns a unified representation space for multiple available modalities (words, characters, images, etc.), and produces a single vector representation with aggregated knowledge among multiple modalities, based on their weighted importance. We motivate this module from the following observations. A majority of the previous literature combine the word and character-level contexts by simply concatenating the word and character embeddings at each decoding step, e.g. INLINEFORM0 in Eq. SECREF4 . However, this naive concatenation of two modalities (word and characters) results in inaccurate decoding, specifically for unknown word token embeddings (an all-zero vector INLINEFORM1 or a random vector INLINEFORM2 is assigned for any unknown token INLINEFORM3 , thus INLINEFORM4 or INLINEFORM5 ). While this concatenation approach does not cause significant errors for well-formatted text, we observe that it induces performance degradation for our social media post datasets which contain a significant number of missing tokens. Similarly, naive merging of textual and visual information ( INLINEFORM0 ) yields suboptimal results as each modality is treated equally informative, whereas in our datasets some of the images may contain irrelevant contexts to textual modalities. Hence, ideally there needs a mechanism in which the model can effectively turn the switch on and off the modalities adaptive to each sample. To this end, we propose a general modality attention module, which adaptively attenuates or emphasizes each modality as a whole at each decoding step INLINEFORM0 , and produces a soft-attended context vector INLINEFORM1 as an input token for the entity LSTM. [at(w),at(c),at(v)] = (Wm[xt(w); xt(c); xt(v)] + bm ) t(m) = (at(m))m'{w,c,v}(at(m')) m {w,c,v} xt = m{w,c,v} t(m)xt(m) where INLINEFORM0 is an attention vector at each decoding step INLINEFORM1 , and INLINEFORM2 is a final context vector at INLINEFORM3 that maximizes information gain for INLINEFORM4 . Note that the optimization of the objective function (Eq. SECREF4 ) with modality attention (Eq. SECREF5 ) requires each modality to have the same dimension ( INLINEFORM5 ), and that the transformation via INLINEFORM6 essentially enforces each modality to be mapped into the same unified subspace, where the weighted average of which encodes discrimitive features for recognition of named entities. When visual context is not provided with each token (as in the traditional NER task), we can define the modality attention for word and character embeddings only in a similar way: [at(w),at(c)] = (Wm[xt(w); xt(c)] + bm ) t(m) = (at(m))m'{w,c}(at(m')) m {w,c} xt = m{w,c} t(m)xt(m) Note that while we apply this modality attention module to the Bi-LSTM+CRF architecture (Section SECREF4 ) for its empirical superiority, the module itself is flexible and thus can work with other NER architectures or for other multimodal applications. SnapCaptions Dataset The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC). These captions are collected exclusively from snaps submitted to public and crowd-sourced stories (aka Snapchat Live Stories or Our Stories). Examples of such public crowd-sourced stories are “New York Story” or “Thanksgiving Story”, which comprise snaps that are aggregated for various public events, venues, etc. All snaps were posted between year 2016 and 2017, and do not contain raw images or other associated information (only textual captions and obfuscated visual descriptor features extracted from the pre-trained InceptionNet are available). We split the dataset into train (70%), validation (15%), and test sets (15%). The captions data have average length of 30.7 characters (5.81 words) with vocabulary size 15,733, where 6,612 are considered unknown tokens from Stanford GloVE embeddings BIBREF22 . Named entities annotated in the SnapCaptions dataset include many of new and emerging entities, and they are found in various surface forms (various nicknames, typos, etc.) To the best of our knowledge, SnapCaptions is the only dataset that contains natural image-caption pairs with expert-annotated named entities. Baselines Task: given a caption and a paired image (if used), the goal is to label every token in a caption in BIO scheme (B: beginning, I: inside, O: outside) BIBREF27 . We report the performance of the following state-of-the-art NER models as baselines, as well as several configurations of our proposed approach to examine contributions of each component (W: word, C: char, V: visual). Bi-LSTM/CRF (W only): only takes word token embeddings (Stanford GloVE) as input. The rest of the architecture is kept the same. Bi-LSTM/CRF + Bi-CharLSTM (C only): only takes a character sequence of each word token as input. (No word embeddings) Bi-LSTM/CRF + Bi-CharLSTM (W+C) BIBREF0 : takes as input both word embeddings and character embeddings extracted from a Bi-CharLSTM. Entity LSTM takes concatenated vectors of word and character embeddings as input tokens. Bi-LSTM/CRF + CharCNN (W+C) BIBREF1 : uses character embeddings extracted from a CNN instead. Bi-LSTM/CRF + CharCNN (W+C) + Multi-task BIBREF8 : trains the model to perform both recognition (into multiple entity types) as well as segmentation (binary) tasks. (proposed) Bi-LSTM/CRF + Bi-CharLSTM with modality attention (W+C): uses the modality attention to merge word and character embeddings. (proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception (W+C+V): takes as input visual contexts extracted from InceptionNet as well, concatenated with word and char vectors. (proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception with modality attention (W+C+V): uses the modality attention to merge word, character, and visual embeddings as input to entity LSTM. Results: SnapCaptions Dataset Table TABREF6 shows the NER performance on the Snap Captions dataset. We report both entity types recognition (PER, LOC, ORG, MISC) and named entity segmentation (named entity or not) results. Parameters: We tune the parameters of each model with the following search space (bold indicate the choice for our final model): character embeddings dimension: {25, 50, 100, 150, 200, 300}, word embeddings size: {25, 50, 100, 150, 200, 300}, LSTM hidden states: {25, 50, 100, 150, 200, 300}, and INLINEFORM0 dimension: {25, 50, 100, 150, 200, 300}. We optimize the parameters with Adagrad BIBREF28 with batch size 10, learning rate 0.02, epsilon INLINEFORM1 , and decay 0.0. Main Results: When visual context is available (W+C+V), we see that the model performance greatly improves over the textual models (W+C), showing that visual contexts are complimentary to textual information in named entity recognition tasks. In addition, it can be seen that the modality attention module further improves the entity type recognition performance for (W+C+V). This result indicates that the modality attention is able to focus on the most effective modality (visual, words, or characters) adaptive to each sample to maximize information gain. Note that our text-only model (W+C) with the modality attention module also significantly outperform the state-of-the-art baselines BIBREF8 , BIBREF1 , BIBREF0 that use the same textual modalities (W+C), showing the effectiveness of the modality attention module for textual models as well. Error Analysis: Table TABREF17 shows example cases where incorporation of visual contexts affects prediction of named entities. For example, the token `curry' in the caption “The curry's " is polysemous and may refer to either a type of food or a famous basketball player `Stephen Curry', and the surrounding textual contexts do not provide enough information to disambiguate it. On the other hand, visual contexts (visual tags: `parade', `urban area', ...) provide similarities to the token's distributional semantics from other training examples (snaps from “NBA Championship Parade Story"), and thus the model successfully predicts the token as a named entity. Similarly, while the text-only model erroneously predicts `Apple' in the caption “Grandma w dat lit Apple Crisp" as an organization (Apple Inc.), the visual contexts (describing objects related to food) help disambiguate the token, making the model predict it correctly as a non-named entity (a fruit). Trending entities (musicians or DJs such as `CID', `Duke Dumont', `Marshmello', etc.) are also recognized correctly with strengthened contexts from visual information (describing concert scenes) despite lack of surrounding textual contexts. A few cases where visual contexts harmed the performance mostly include visual tags that are unrelated to a token or its surrounding textual contexts. Visualization of Modality Attention: Figure FIGREF19 visualizes the modality attention module at each decoding step (each column), where amplified modality is represented with darker color, and attenuated modality is represented with lighter color. For the image-aided model (W+C+V; upper row in Figure FIGREF19 ), we confirm that the modality attention successfully attenuates irrelevant signals (selfies, etc.) and amplifies relevant modality-based contexts in prediction of a given token. In the example of “disney word essential = coffee" with visual tags selfie, phone, person, the modality attention successfully attenuates distracting visual signals and focuses on textual modalities, consequently making correct predictions. The named entities in the examples of “Beautiful night atop The Space Needle" and “Splash Mountain" are challenging to predict because they are composed of common nouns (space, needle, splash, mountain), and thus they often need additional contexts to correctly predict. In the training data, visual contexts make stronger indicators for these named entities (space needle, splash mountain), and the modality attention module successfully attends more to stronger signals. For text-only model (W+C), we observe that performance gains mostly come from the modality attention module better handling tokens unseen during training or unknown tokens from the pre-trained word embeddings matrix. For example, while WaRriOoOrs and Kooler Matic are missing tokens in the word embeddings matrix, it successfully amplifies character-based contexts (capitalized first letters, similarity to known entities `Golden State Warriors') and suppresses word-based contexts (word embeddings for unknown tokens `WaRriOoOrs'), leading to correct predictions. This result is significant because it shows performance of the model, with an almost identical architecture, can still improve without having to scale the word embeddings matrix indefinitely. Figure FIGREF19 (b) shows the cases where the modality attention led to incorrect predictions. For example, the model predicts missing tokens HUUUGE and Shampooer incorrectly as named entities by amplifying misleading character-based contexts (capitalized first letters) or visual contexts (concert scenes, associated contexts of which often include named entities in the training dataset). Sensitivity to Word Embeddings Vocabulary Size: In order to isolate the effectiveness of the modality attention module on textual models in handling missing tokens, we report the performance with varying word embeddings vocabulary sizes in Table TABREF20 . By increasing the number of missing tokens artificially by randomly removing words from the word embeddings matrix (original vocab size: 400K), we observe that while the overall performance degrades, the modality attention module is able to suppress the peformance degradation. Note also that the performance gap generally gets bigger as we decrease the vocabulary size of the word embeddings matrix. This result is significant in that the modality attention is able to improve the model more robust to missing tokens without having to train an indefinitely large word embeddings matrix for arbitrarily noisy social media text datasets. Conclusions We proposed a new multimodal NER (MNER: image + text) task on short social media posts. We demonstrated for the first time an effective MNER system, where visual information is combined with textual information to outperform traditional text-based NER baselines. Our work can be applied to myriads of social media posts or other articles across multiple platforms which often include both text and accompanying images. In addition, we proposed the modality attention module, a new neural mechanism which learns optimal integration of different modes of correlated information. In essence, the modality attention learns to attenuate irrelevant or uninformative modal information while amplifying the primary modality to extract better overall representations. We showed that the modality attention based model outperforms other state-of-the-art baselines when text was the only modality available, by better combining word and character level information.
Which types of named entities do they recognize?
f7a89b9cd2792f23f2cb43d50a01b8218a6fbb24
Which Ġtypes Ġof Ġnamed Ġentities Ġdo Ġthey Ġrecognize ?
PER , ĠLOC , ĠOR G , ĠM ISC
<s> Which Ġtypes Ġof Ġnamed Ġentities Ġdo Ġthey Ġrecognize ? </s> Introduction </s> Social Ġmedia Ġwith Ġabundant Ġuser - generated Ġposts Ġprovide Ġa Ġrich Ġplatform Ġfor Ġunderstanding Ġevents , Ġopinions Ġand Ġpreferences Ġof Ġgroups Ġand Ġindividuals . ĠThese Ġinsights Ġare Ġprimarily Ġhidden Ġin Ġun struct ured Ġforms Ġof Ġsocial Ġmedia Ġposts , Ġsuch Ġas Ġin Ġfree - form Ġtext Ġor Ġimages Ġwithout Ġtags . ĠNamed Ġentity Ġrecognition Ġ( NER ), Ġthe Ġtask Ġof Ġrecognizing Ġnamed Ġentities Ġfrom Ġfree - form Ġtext , Ġis Ġthus Ġa Ġcritical Ġstep Ġfor Ġbuilding Ġstructural Ġinformation , Ġallowing Ġfor Ġits Ġuse Ġin Ġpersonalized Ġassistance , Ġrecommendations , Ġadvertisement , Ġetc . </s> While Ġmany Ġprevious Ġapproaches ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġon ĠN ER Ġhave Ġshown Ġsuccess Ġfor Ġwell - formed Ġtext Ġin Ġrecognizing Ġnamed Ġentities Ġvia Ġword Ġcontext Ġresolution Ġ( e . g . ĠL ST M Ġwith Ġword Ġembed d ings ) Ġcombined Ġwith Ġcharacter - level Ġfeatures Ġ( e . g . ĠChar L ST M / CNN ), Ġseveral Ġadditional Ġchallenges Ġremain Ġfor Ġrecognizing Ġnamed Ġentities Ġfrom Ġextremely Ġshort Ġand Ġcoarse Ġtext Ġfound Ġin Ġsocial Ġmedia Ġposts . ĠFor Ġinstance , Ġshort Ġsocial Ġmedia Ġposts Ġoften Ġdo Ġnot Ġprovide Ġenough Ġtextual Ġcontexts Ġto Ġresolve Ġpoly sem ous Ġentities Ġ( e . g . ĠâĢ ľ mon opoly Ġis Ġda Ġbest Ġ", Ġwhere Ġ` mon opoly ' Ġmay Ġrefer Ġto Ġa Ġboard Ġgame Ġ( named Ġentity ) Ġor Ġa Ġterm Ġin Ġeconomics ). ĠIn Ġaddition , Ġnoisy Ġtext Ġincludes Ġa Ġhuge Ġnumber Ġof Ġunknown Ġtokens Ġdue Ġto Ġinconsistent Ġlex ical Ġnot ations Ġand Ġfrequent Ġmentions Ġof Ġvarious Ġnewly Ġtrending Ġentities Ġ( e . g . ĠâĢ ľ x ox o ĠMarsh m ell oooo Ġ", Ġwhere Ġ` Marsh m ell oooo ' Ġis Ġa Ġmis - spe lling Ġof Ġa Ġknown Ġentity Ġ` Marsh m ello ', Ġa Ġmusic Ġproducer ), Ġmaking Ġword Ġembed d ings Ġbased Ġneural Ġnetworks ĠN ER Ġmodels Ġvulnerable . </s> To Ġaddress Ġthe Ġchallenges Ġabove Ġfor Ġsocial Ġmedia Ġposts , Ġwe Ġbuild Ġupon Ġthe Ġstate - of - the - art Ġneural Ġarchitecture Ġfor ĠN ER Ġwith Ġthe Ġfollowing Ġtwo Ġnovel Ġapproaches Ġ( Figure ĠFIG REF 1 Ġ). ĠFirst , Ġwe Ġpropose Ġto Ġleverage Ġauxiliary Ġmod alities Ġfor Ġadditional Ġcontext Ġresolution Ġof Ġentities . ĠFor Ġexample , Ġmany Ġpopular Ġsocial Ġmedia Ġplatforms Ġnow Ġprovide Ġways Ġto Ġcompose Ġa Ġpost Ġin Ġmultiple Ġmod alities Ġ- Ġspecifically Ġimage Ġand Ġtext Ġ( e . g . ĠSnapchat Ġcapt ions , ĠTwitter Ġposts Ġwith Ġimage ĠURLs ), Ġfrom Ġwhich Ġwe Ġcan Ġobtain Ġadditional Ġcontext Ġfor Ġunderstanding Ġposts . ĠWhile ĠâĢ ľ mon opoly " Ġin Ġthe Ġprevious Ġexample Ġis Ġambiguous Ġin Ġits Ġtextual Ġform , Ġan Ġaccompanying Ġsnap Ġimage Ġof Ġa Ġboard Ġgame Ġcan Ġhelp Ġdis amb ig uate Ġamong Ġpoly sem ous Ġentities , Ġthereby Ġcorrectly Ġrecognizing Ġit Ġas Ġa Ġnamed Ġentity . </s> Second , Ġwe Ġalso Ġpropose Ġa Ġgeneral Ġmod ality Ġattention Ġmodule Ġwhich Ġchooses Ġper Ġdecoding Ġstep Ġthe Ġmost Ġinformative Ġmod ality Ġamong Ġavailable Ġones Ġ( in Ġour Ġcase , Ġword Ġembed d ings , Ġcharacter Ġembed d ings , Ġor Ġvisual Ġfeatures ) Ġto Ġextract Ġcontext Ġfrom . ĠFor Ġexample , Ġthe Ġmod ality Ġattention Ġmodule Ġlets Ġthe Ġdec oder Ġatten uate Ġthe Ġword - level Ġsignals Ġfor Ġunknown Ġword Ġtokens Ġ( âĢ ľ Marsh m ell oooo o " Ġwith Ġtrailing Ġ` o 's ) Ġand Ġampl ifies Ġcharacter - level Ġfeatures Ġint se ad Ġ( capital ized Ġfirst Ġletter , Ġlex ical Ġsimilarity Ġto Ġother Ġknown Ġnamed Ġentity Ġtoken Ġ` Marsh m ello ', Ġetc .), Ġthereby Ġsuppressing Ġnoise Ġinformation Ġ( âĢ ľ UN K " Ġtoken Ġembed ding ) Ġin Ġdecoding Ġsteps . ĠNote Ġthat Ġmost Ġof Ġthe Ġprevious Ġliterature Ġin ĠN ER Ġor Ġother ĠN LP Ġtasks Ġcombine Ġword Ġand Ġcharacter - level Ġinformation Ġwith Ġnaive Ġconc aten ation , Ġwhich Ġis Ġvulnerable Ġto Ġnoisy Ġsocial Ġmedia Ġposts . ĠWhen Ġan Ġauxiliary Ġimage Ġis Ġavailable , Ġthe Ġmod ality Ġattention Ġmodule Ġdetermines Ġto Ġamplify Ġthis Ġvisual Ġcontext Ġin Ġdis amb ig uating Ġpoly sem ous Ġentities , Ġor Ġto Ġatten uate Ġvisual Ġcontexts Ġwhen Ġthey Ġare Ġirrelevant Ġto Ġtarget Ġnamed Ġentities , Ġselfies , Ġetc . ĠNote Ġthat Ġthe Ġproposed Ġmod ality Ġattention Ġmodule Ġis Ġdistinct Ġfrom Ġhow Ġattention Ġis Ġused Ġin Ġother Ġsequence - to - sequence Ġliterature Ġ( e . g . Ġattending Ġto Ġa Ġspecific Ġtoken Ġwithin Ġan Ġinput Ġsequence ). ĠSection ĠSEC REF 2 Ġprovides Ġthe Ġdetailed Ġliterature Ġreview . </s> Our Ġcontributions Ġare Ġthree - fold : Ġwe Ġpropose Ġ( 1 ) Ġan ĠL ST M - CNN Ġhybrid Ġmultim od al ĠN ER Ġnetwork Ġthat Ġtakes Ġas Ġinput Ġboth Ġimage Ġand Ġtext Ġfor Ġrecognition Ġof Ġa Ġnamed Ġentity Ġin Ġtext Ġinput . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġour Ġapproach Ġis Ġthe Ġfirst Ġwork Ġto Ġincorporate Ġvisual Ġcontexts Ġfor Ġnamed Ġentity Ġrecognition Ġtasks . Ġ( 2 ) ĠWe Ġpropose Ġa Ġgeneral Ġmod ality Ġattention Ġmodule Ġthat Ġselectively Ġchooses Ġmod alities Ġto Ġextract Ġprimary Ġcontext Ġfrom , Ġmaximizing Ġinformation Ġgain Ġand Ġsuppressing Ġirrelevant Ġcontexts Ġfrom Ġeach Ġmod ality Ġ( we Ġtreat Ġwords , Ġcharacters , Ġand Ġimages Ġas Ġseparate Ġmod alities ). Ġ( 3 ) ĠWe Ġshow Ġthat Ġthe Ġproposed Ġapproaches Ġoutper form Ġthe Ġstate - of - the - art ĠN ER Ġmodels Ġ( both Ġwith Ġand Ġwithout Ġusing Ġadditional Ġvisual Ġcontexts ) Ġon Ġour Ġnew ĠM NER Ġdataset ĠSnap Capt ions , Ġa Ġlarge Ġcollection Ġof Ġinformal Ġand Ġextremely Ġshort Ġsocial Ġmedia Ġposts Ġpaired Ġwith Ġunique Ġimages . </s> Related ĠWork </s> Ne ural Ġmodels Ġfor ĠN ER Ġhave Ġbeen Ġrecently Ġproposed , Ġproducing Ġstate - of - the - art Ġperformance Ġon Ġstandard ĠN ER Ġtasks . ĠFor Ġexample , Ġsome Ġof Ġthe Ġend - to - end ĠN ER Ġsystems ĠB IB REF 4 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġuse Ġa Ġrecurrent Ġneural Ġnetwork Ġusually Ġwith Ġa ĠCR F ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġfor Ġsequence Ġlabeling , Ġaccompanied Ġwith Ġfeature Ġextract ors Ġfor Ġwords Ġand Ġcharacters Ġ( CNN , ĠL ST Ms , Ġetc .), Ġand Ġachieve Ġthe Ġstate - of - the - art Ġperformance Ġmostly Ġwithout Ġany Ġuse Ġof Ġg az ette ers Ġinformation . ĠNote Ġthat Ġmost Ġof Ġthese Ġwork Ġaggregate Ġtextual Ġcontexts Ġvia Ġconc aten ation Ġof Ġword Ġembed d ings Ġand Ġcharacter Ġembed d ings . ĠRecently , Ġseveral Ġwork Ġhave Ġaddressed Ġthe ĠN ER Ġtask Ġspecifically Ġon Ġnoisy Ġshort Ġtext Ġsegments Ġsuch Ġas ĠTwe ets , Ġetc . ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠThey Ġreport Ġperformance Ġgains Ġfrom Ġleveraging Ġexternal Ġsources Ġof Ġinformation Ġsuch Ġas Ġlex ical Ġinformation Ġ( POS Ġtags , Ġetc .) Ġand / or Ġfrom Ġseveral Ġpre processing Ġsteps Ġ( token Ġsubstitution , Ġetc .). ĠOur Ġmodel Ġbuilds Ġupon Ġthese Ġstate - of - the - art Ġneural Ġmodels Ġfor ĠN ER Ġtasks , Ġand Ġimproves Ġthe Ġmodel Ġin Ġtwo Ġcritical Ġways : Ġ( 1 ) Ġincorporation Ġof Ġvisual Ġcontexts Ġto Ġprovide Ġauxiliary Ġinformation Ġfor Ġshort Ġmedia Ġposts , Ġand Ġ( 2 ) Ġaddition Ġof Ġthe Ġmod ality Ġattention Ġmodule , Ġwhich Ġbetter Ġincorporates Ġword Ġembed d ings Ġand Ġcharacter Ġembed d ings , Ġespecially Ġwhen Ġthere Ġare Ġmany Ġmissing Ġtokens Ġin Ġthe Ġgiven Ġword Ġembed ding Ġmatrix . ĠNote Ġthat Ġwe Ġdo Ġnot Ġexplore Ġthe Ġuse Ġof Ġg az ette ers Ġinformation Ġor Ġother Ġauxiliary Ġinformation Ġ( POS Ġtags , Ġetc .) ĠB IB REF 9 Ġas Ġit Ġis Ġnot Ġthe Ġfocus Ġof Ġour Ġstudy . </s> Att ention Ġmodules Ġare Ġwidely Ġapplied Ġin Ġseveral Ġdeep Ġlearning Ġtasks ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠFor Ġexample , Ġthey Ġuse Ġan Ġattention Ġmodule Ġto Ġattend Ġto Ġa Ġsubset Ġwithin Ġa Ġsingle Ġinput Ġ( a Ġpart / region Ġof Ġan Ġimage , Ġa Ġspecific Ġtoken Ġin Ġan Ġinput Ġsequence Ġof Ġtokens , Ġetc .) Ġat Ġeach Ġdecoding Ġstep Ġin Ġan Ġenc oder - dec oder Ġframework Ġfor Ġimage Ġcaption ing Ġtasks , Ġetc . ĠB IB REF 14 Ġexplore Ġvarious Ġattention Ġmechanisms Ġin ĠN LP Ġtasks , Ġbut Ġdo Ġnot Ġincorporate Ġvisual Ġcomponents Ġor Ġinvestigate Ġthe Ġimpact Ġof Ġsuch Ġmodels Ġon Ġnoisy Ġsocial Ġmedia Ġdata . ĠB IB REF 15 Ġpropose Ġto Ġuse Ġattention Ġfor Ġa Ġsubset Ġof Ġdiscrete Ġsource Ġsamples Ġin Ġtransfer Ġlearning Ġsettings . ĠOur Ġmod ality Ġattention Ġdiffers Ġfrom Ġthe Ġprevious Ġapproaches Ġin Ġthat Ġwe Ġatten uate Ġor Ġampl ifies Ġeach Ġmod ality Ġinput Ġas Ġa Ġwhole Ġamong Ġmultiple Ġavailable Ġmod alities , Ġand Ġthat Ġwe Ġuse Ġthe Ġattention Ġmechanism Ġessentially Ġto Ġmap Ġheter ogeneous Ġmod alities Ġin Ġa Ġsingle Ġjoint Ġembed ding Ġspace . ĠOur Ġapproach Ġalso Ġallows Ġfor Ġre - use Ġof Ġthe Ġsame Ġmodel Ġfor Ġpredicting Ġlabels Ġeven Ġwhen Ġsome Ġof Ġthe Ġmod alities Ġare Ġmissing Ġin Ġinput , Ġas Ġother Ġmod alities Ġwould Ġstill Ġpreserve Ġthe Ġsame Ġsemantics Ġin Ġthe Ġembed d ings Ġspace . </s> Mult im od al Ġlearning Ġis Ġstudied Ġin Ġvarious Ġdomains Ġand Ġapplications , Ġaimed Ġat Ġbuilding Ġa Ġjoint Ġmodel Ġthat Ġextracts Ġcontextual Ġinformation Ġfrom Ġmultiple Ġmod alities Ġ( views ) Ġof Ġparallel Ġdatasets . </s> The Ġmost Ġrelevant Ġtask Ġto Ġour Ġmultim od al ĠN ER Ġsystem Ġis Ġthe Ġtask Ġof Ġmultim od al Ġmachine Ġtranslation ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, Ġwhich Ġaims Ġat Ġbuilding Ġa Ġbetter Ġmachine Ġtranslation Ġsystem Ġby Ġtaking Ġas Ġinput Ġa Ġsentence Ġin Ġa Ġsource Ġlanguage Ġas Ġwell Ġas Ġa Ġcorresponding Ġimage . ĠSeveral Ġstandard Ġsequence - to - sequence Ġarchitectures Ġare Ġexplored Ġ( a Ġtarget - language ĠL ST M Ġdec oder Ġthat Ġtakes Ġas Ġinput Ġan Ġimage Ġfirst ). ĠOther Ġprevious Ġliterature Ġinclude Ġstudy Ġof ĠCanon ical ĠCor relation ĠAnalysis Ġ( CC A ) ĠB IB REF 18 Ġto Ġlearn Ġfeature Ġcorrelations Ġamong Ġmultiple Ġmod alities , Ġwhich Ġis Ġwidely Ġused Ġin Ġmany Ġapplications . ĠOther Ġapplications Ġinclude Ġimage Ġcaption ing ĠB IB REF 10 Ġ, Ġaudio - visual Ġrecognition ĠB IB REF 19 Ġ, Ġvisual Ġquestion Ġanswering Ġsystems ĠB IB REF 20 Ġ, Ġetc . </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġour Ġapproach Ġis Ġthe Ġfirst Ġwork Ġto Ġincorporate Ġvisual Ġcontexts Ġfor Ġnamed Ġentity Ġrecognition Ġtasks . </s> Prop osed ĠMethods </s> Figure ĠFIG REF 2 Ġillustrates Ġthe Ġproposed Ġmultim od al ĠN ER Ġ( M NER ) Ġmodel . ĠFirst , Ġwe Ġobtain Ġword Ġembed d ings , Ġcharacter Ġembed d ings , Ġand Ġvisual Ġfeatures Ġ( Section ĠSEC REF 3 Ġ). ĠA ĠBi - L ST M - CR F Ġmodel Ġthen Ġtakes Ġas Ġinput Ġa Ġsequence Ġof Ġtokens , Ġeach Ġof Ġwhich Ġcomprises Ġa Ġword Ġtoken , Ġa Ġcharacter Ġsequence , Ġand Ġan Ġimage , Ġin Ġtheir Ġrespective Ġrepresentation Ġ( Section ĠSEC REF 4 Ġ). ĠAt Ġeach Ġdecoding Ġstep , Ġrepresentations Ġfrom Ġeach Ġmod ality Ġare Ġcombined Ġvia Ġthe Ġmod ality Ġattention Ġmodule Ġto Ġproduce Ġan Ġentity Ġlabel Ġfor Ġeach Ġtoken Ġ( ĠSEC REF 5 Ġ). ĠWe Ġformulate Ġeach Ġcomponent Ġof Ġthe Ġmodel Ġin Ġthe Ġfollowing Ġsubsections . </s> Not ations : ĠLet ĠIN LINE FORM 0 Ġa Ġsequence Ġof Ġinput Ġtokens Ġwith Ġlength ĠIN LINE FORM 1 Ġ, Ġwith Ġa Ġcorresponding Ġlabel Ġsequence ĠIN LINE FORM 2 Ġindicating Ġnamed Ġentities Ġ( e . g . Ġin Ġstandard ĠB IO Ġformats ). ĠEach Ġinput Ġtoken Ġis Ġcomposed Ġof Ġthree Ġmod alities : ĠIN LINE FORM 3 Ġfor Ġword Ġembed d ings , Ġcharacter Ġembed d ings , Ġand Ġvisual Ġembed d ings Ġrepresentations , Ġrespectively . </s> Features </s> Similar Ġto Ġthe Ġstate - of - the - art ĠN ER Ġapproaches ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġwe Ġuse Ġboth Ġword Ġembed d ings Ġand Ġcharacter Ġembed d ings . </s> Word Ġembed d ings Ġare Ġobtained Ġfrom Ġan Ġun super vised Ġlearning Ġmodel Ġthat Ġlearns Ġco - occ urrence Ġstatistics Ġof Ġwords Ġfrom Ġa Ġlarge Ġexternal Ġcorpus , Ġyielding Ġword Ġembed d ings Ġas Ġdistribution al Ġsemantics ĠB IB REF 21 Ġ. ĠSpecifically , Ġwe Ġuse Ġpre - trained Ġembed d ings Ġfrom ĠGl o VE ĠB IB REF 22 Ġ. </s> Character Ġembed d ings Ġare Ġobtained Ġfrom Ġa ĠBi - L ST M Ġwhich Ġtakes Ġas Ġinput Ġa Ġsequence Ġof Ġcharacters Ġof Ġeach Ġtoken , Ġsimilarly Ġto ĠB IB REF 0 Ġ. ĠAn Ġalternative Ġapproach Ġfor Ġobtaining Ġcharacter Ġembed d ings Ġis Ġusing Ġa Ġconv olution al Ġneural Ġnetwork Ġas Ġin ĠB IB REF 1 Ġ, Ġbut Ġwe Ġfind Ġthat ĠBi - L ST M Ġrepresentation Ġof Ġcharacters Ġyields Ġempir ically Ġbetter Ġresults Ġin Ġour Ġexperiments . </s> Visual Ġembed d ings : ĠTo Ġextract Ġfeatures Ġfrom Ġan Ġimage , Ġwe Ġtake Ġthe Ġfinal Ġhidden Ġlayer Ġrepresentation Ġof Ġa Ġmodified Ġversion Ġof Ġthe Ġconv olution al Ġnetwork Ġmodel Ġcalled ĠIn ception Ġ( Go og Le Net ) ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġtrained Ġon Ġthe ĠImage Net Ġdataset ĠB IB REF 25 Ġto Ġclassify Ġmultiple Ġobjects Ġin Ġthe Ġscene . ĠOur Ġimplementation Ġof Ġthe ĠIn ception Ġmodel Ġhas Ġdeep Ġ22 Ġlayers , Ġtraining Ġof Ġwhich Ġis Ġmade Ġpossible Ġvia ĠâĢ ľ network Ġin Ġnetwork " Ġprinciples Ġand Ġseveral Ġdimension Ġreduction Ġtechniques Ġto Ġimprove Ġcomputing Ġresource Ġutilization . ĠThe Ġfinal Ġlayer Ġrepresentation Ġenc odes Ġdiscrim in ative Ġinformation Ġdescribing Ġwhat Ġobjects Ġare Ġshown Ġin Ġan Ġimage , Ġwhich Ġprovide Ġauxiliary Ġcontexts Ġfor Ġunderstanding Ġtextual Ġtokens Ġand Ġentities Ġin Ġaccompanying Ġcapt ions . </s> In cor por ating Ġthis Ġvisual Ġinformation Ġonto Ġthe Ġtraditional ĠN ER Ġsystem Ġis Ġan Ġopen Ġchallenge , Ġand Ġmultiple Ġapproaches Ġcan Ġbe Ġconsidered . ĠFor Ġinstance , Ġone Ġmay Ġprovide Ġvisual Ġcontexts Ġonly Ġas Ġan Ġinitial Ġinput Ġto Ġdec oder Ġas Ġin Ġsome Ġenc oder - dec oder Ġimage Ġcaption ing Ġsystems ĠB IB REF 26 Ġ. ĠHowever , Ġwe Ġempir ically Ġobserve Ġthat Ġan ĠN ER Ġdec oder Ġwhich Ġtakes Ġas Ġinput Ġthe Ġvisual Ġembed d ings Ġat Ġevery Ġdecoding Ġstep Ġ( Section ĠSEC REF 4 Ġ), Ġcombined Ġwith Ġthe Ġmod ality Ġattention Ġmodule Ġ( Section ĠSEC REF 5 Ġ), Ġyields Ġbetter Ġresults . </s> Lastly , Ġwe Ġadd Ġa Ġtransform Ġlayer Ġfor Ġeach Ġfeature ĠIN LINE FORM 0 Ġbefore Ġit Ġis Ġfed Ġto Ġthe ĠN ER Ġentity ĠL ST M . </s> Bi - L ST M Ġ+ ĠCR F Ġfor ĠMult im od al ĠN ER </s> Our ĠM NER Ġmodel Ġis Ġbuilt Ġon Ġa ĠBi - L ST M Ġand ĠCR F Ġhybrid Ġmodel . ĠWe Ġuse Ġthe Ġfollowing Ġimplementation Ġfor Ġthe Ġentity ĠBi - L ST M . </s> it Ġ= Ġ( W xi ht - 1 Ġ+ ĠW c ict - 1 ) </s> ct Ġ= Ġ( 1 - it ) Ġc t - 1 </s> + Ġit Ġtan h ( W xc xt Ġ+ ĠWh cht - 1 ) </s> ot Ġ= Ġ( W x o xt Ġ+ ĠWh oh t - 1 Ġ+ ĠW co ct ) </s> ht Ġ= ĠL ST M ( xt ) </s> = Ġot Ġtan h ( ct ) </s> where ĠIN LINE FORM 0 Ġis Ġa Ġweighted Ġaverage Ġof Ġthree Ġmod alities ĠIN LINE FORM 1 Ġvia Ġthe Ġmod ality Ġattention Ġmodule , Ġwhich Ġwill Ġbe Ġdefined Ġin ĠSection ĠSEC REF 5 Ġ. ĠB ias Ġterms Ġfor Ġgates Ġare Ġomitted Ġhere Ġfor Ġsimplicity Ġof Ġnotation . </s> We Ġthen Ġobtain Ġbi - direction al Ġentity Ġtoken Ġrepresentations ĠIN LINE FORM 0 Ġby Ġconc aten ating Ġits Ġleft Ġand Ġright Ġcontext Ġrepresentations . ĠTo Ġenforce Ġstructural Ġcorrelations Ġbetween Ġlabels Ġin Ġsequence Ġdecoding , ĠIN LINE FORM 1 Ġis Ġthen Ġpassed Ġto Ġa Ġconditional Ġrandom Ġfield Ġ( CR F ) Ġto Ġproduce Ġa Ġlabel Ġfor Ġeach Ġtoken Ġmaximizing Ġthe Ġfollowing Ġobjective . Ġy * Ġ= Ġy Ġp ( y | h ; ĠW CR F ) </s> p ( y | h ; ĠW CR F ) Ġ= Ġt Ġt Ġ( yt - 1 , yt ; h ) Ġy ' Ġt Ġt Ġ( y 't - 1 , y 't ; h ) </s> where ĠIN LINE FORM 0 Ġis Ġa Ġpotential Ġfunction , ĠIN LINE FORM 1 Ġis Ġa Ġset Ġof Ġparameters Ġthat Ġdefines Ġthe Ġpotential Ġfunctions Ġand Ġweight Ġvectors Ġfor Ġlabel Ġpairs Ġ( ĠIN LINE FORM 2 Ġ). ĠB ias Ġterms Ġare Ġomitted Ġfor Ġbre vity Ġof Ġformulation . </s> The Ġmodel Ġcan Ġbe Ġtrained Ġvia Ġlog - like lihood Ġmaxim ization Ġfor Ġthe Ġtraining Ġset ĠIN LINE FORM 0 Ġ: </s> L ( W CR F ) Ġ= Ġi Ġp ( y | h ; ĠW ) </s> Mod ality ĠAttention </s> The Ġmod ality Ġattention Ġmodule Ġlearns Ġa Ġunified Ġrepresentation Ġspace Ġfor Ġmultiple Ġavailable Ġmod alities Ġ( words , Ġcharacters , Ġimages , Ġetc .), Ġand Ġproduces Ġa Ġsingle Ġvector Ġrepresentation Ġwith Ġaggreg ated Ġknowledge Ġamong Ġmultiple Ġmod alities , Ġbased Ġon Ġtheir Ġweighted Ġimportance . ĠWe Ġmotivate Ġthis Ġmodule Ġfrom Ġthe Ġfollowing Ġobservations . </s> A Ġmajority Ġof Ġthe Ġprevious Ġliterature Ġcombine Ġthe Ġword Ġand Ġcharacter - level Ġcontexts Ġby Ġsimply Ġconc aten ating Ġthe Ġword Ġand Ġcharacter Ġembed d ings Ġat Ġeach Ġdecoding Ġstep , Ġe . g . ĠIN LINE FORM 0 Ġin ĠE q . ĠSEC REF 4 Ġ. ĠHowever , Ġthis Ġnaive Ġconc aten ation Ġof Ġtwo Ġmod alities Ġ( word Ġand Ġcharacters ) Ġresults Ġin Ġinaccurate Ġdecoding , Ġspecifically Ġfor Ġunknown Ġword Ġtoken Ġembed d ings Ġ( an Ġall - zero Ġvector ĠIN LINE FORM 1 Ġor Ġa Ġrandom Ġvector ĠIN LINE FORM 2 Ġis Ġassigned Ġfor Ġany Ġunknown Ġtoken ĠIN LINE FORM 3 Ġ, Ġthus ĠIN LINE FORM 4 Ġor ĠIN LINE FORM 5 Ġ). ĠWhile Ġthis Ġconc aten ation Ġapproach Ġdoes Ġnot Ġcause Ġsignificant Ġerrors Ġfor Ġwell - form atted Ġtext , Ġwe Ġobserve Ġthat Ġit Ġinduces Ġperformance Ġdegradation Ġfor Ġour Ġsocial Ġmedia Ġpost Ġdatasets Ġwhich Ġcontain Ġa Ġsignificant Ġnumber Ġof Ġmissing Ġtokens . </s> Similarly , Ġnaive Ġmerging Ġof Ġtextual Ġand Ġvisual Ġinformation Ġ( ĠIN LINE FORM 0 Ġ) Ġyields Ġsub opt imal Ġresults Ġas Ġeach Ġmod ality Ġis Ġtreated Ġequally Ġinformative , Ġwhereas Ġin Ġour Ġdatasets Ġsome Ġof Ġthe Ġimages Ġmay Ġcontain Ġirrelevant Ġcontexts Ġto Ġtextual Ġmod alities . ĠHence , Ġideally Ġthere Ġneeds Ġa Ġmechanism Ġin Ġwhich Ġthe Ġmodel Ġcan Ġeffectively Ġturn Ġthe Ġswitch Ġon Ġand Ġoff Ġthe Ġmod alities Ġadaptive Ġto Ġeach Ġsample . </s> To Ġthis Ġend , Ġwe Ġpropose Ġa Ġgeneral Ġmod ality Ġattention Ġmodule , Ġwhich Ġadapt ively Ġatten uates Ġor Ġemphasizes Ġeach Ġmod ality Ġas Ġa Ġwhole Ġat Ġeach Ġdecoding Ġstep ĠIN LINE FORM 0 Ġ, Ġand Ġproduces Ġa Ġsoft - att ended Ġcontext Ġvector ĠIN LINE FORM 1 Ġas Ġan Ġinput Ġtoken Ġfor Ġthe Ġentity ĠL ST M . Ġ[ at ( w ), at ( c ), at ( v )] Ġ= Ġ( W m [ xt ( w ); Ġ xt ( c ); Ġ xt ( v )] Ġ+ Ġb m Ġ) </s> t ( m ) Ġ= Ġ( at ( m )) m ' { w , c , v } ( at ( m ' )) Ġm Ġ{ w , c , v } </s> xt Ġ= Ġm { w , c , v } Ġt ( m ) xt ( m ) </s> where ĠIN LINE FORM 0 Ġis Ġan Ġattention Ġvector Ġat Ġeach Ġdecoding Ġstep ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġis Ġa Ġfinal Ġcontext Ġvector Ġat ĠIN LINE FORM 3 Ġthat Ġmaxim izes Ġinformation Ġgain Ġfor ĠIN LINE FORM 4 Ġ. ĠNote Ġthat Ġthe Ġoptimization Ġof Ġthe Ġobjective Ġfunction Ġ( E q . ĠSEC REF 4 Ġ) Ġwith Ġmod ality Ġattention Ġ( E q . ĠSEC REF 5 Ġ) Ġrequires Ġeach Ġmod ality Ġto Ġhave Ġthe Ġsame Ġdimension Ġ( ĠIN LINE FORM 5 Ġ), Ġand Ġthat Ġthe Ġtransformation Ġvia ĠIN LINE FORM 6 Ġessentially Ġen forces Ġeach Ġmod ality Ġto Ġbe Ġmapped Ġinto Ġthe Ġsame Ġunified Ġsub space , Ġwhere Ġthe Ġweighted Ġaverage Ġof Ġwhich Ġenc odes Ġdiscrim itive Ġfeatures Ġfor Ġrecognition Ġof Ġnamed Ġentities . </s> When Ġvisual Ġcontext Ġis Ġnot Ġprovided Ġwith Ġeach Ġtoken Ġ( as Ġin Ġthe Ġtraditional ĠN ER Ġtask ), Ġwe Ġcan Ġdefine Ġthe Ġmod ality Ġattention Ġfor Ġword Ġand Ġcharacter Ġembed d ings Ġonly Ġin Ġa Ġsimilar Ġway : Ġ[ at ( w ), at ( c )] Ġ= Ġ( W m [ xt ( w ); Ġ xt ( c )] Ġ+ Ġb m Ġ) </s> t ( m ) Ġ= Ġ( at ( m )) m ' { w , c } ( at ( m ' )) Ġm Ġ{ w , c } </s> xt Ġ= Ġm { w , c } Ġt ( m ) xt ( m ) </s> Note Ġthat Ġwhile Ġwe Ġapply Ġthis Ġmod ality Ġattention Ġmodule Ġto Ġthe ĠBi - L ST M + CR F Ġarchitecture Ġ( Section ĠSEC REF 4 Ġ) Ġfor Ġits Ġempirical Ġsuperiority , Ġthe Ġmodule Ġitself Ġis Ġflexible Ġand Ġthus Ġcan Ġwork Ġwith Ġother ĠN ER Ġarchitectures Ġor Ġfor Ġother Ġmultim od al Ġapplications . </s> Snap Capt ions ĠDat as et </s> The ĠSnap Capt ions Ġdataset Ġis Ġcomposed Ġof Ġ10 K Ġuser - generated Ġimage Ġ( snap ) Ġand Ġtextual Ġcaption Ġpairs Ġwhere Ġnamed Ġentities Ġin Ġcapt ions Ġare Ġmanually Ġlabeled Ġby Ġexpert Ġhuman Ġannot ators Ġ( entity Ġtypes : ĠPER , ĠLOC , ĠOR G , ĠM ISC ). ĠThese Ġcapt ions Ġare Ġcollected Ġexclusively Ġfrom Ġsnaps Ġsubmitted Ġto Ġpublic Ġand Ġcrowd - s ourced Ġstories Ġ( aka ĠSnapchat ĠLive ĠStories Ġor ĠOur ĠStories ). ĠExamples Ġof Ġsuch Ġpublic Ġcrowd - s ourced Ġstories Ġare ĠâĢ ľ New ĠYork ĠStory âĢ Ŀ Ġor ĠâĢ ľ Thanks giving ĠStory âĢ Ŀ , Ġwhich Ġcomprise Ġsnaps Ġthat Ġare Ġaggreg ated Ġfor Ġvarious Ġpublic Ġevents , Ġvenues , Ġetc . ĠAll Ġsnaps Ġwere Ġposted Ġbetween Ġyear Ġ2016 Ġand Ġ2017 , Ġand Ġdo Ġnot Ġcontain Ġraw Ġimages Ġor Ġother Ġassociated Ġinformation Ġ( only Ġtextual Ġcapt ions Ġand Ġobfusc ated Ġvisual Ġdescriptor Ġfeatures Ġextracted Ġfrom Ġthe Ġpre - trained ĠIn ception Net Ġare Ġavailable ). ĠWe Ġsplit Ġthe Ġdataset Ġinto Ġtrain Ġ( 70 %), Ġvalidation Ġ( 15 %), Ġand Ġtest Ġsets Ġ( 15 %). ĠThe Ġcapt ions Ġdata Ġhave Ġaverage Ġlength Ġof Ġ30 . 7 Ġcharacters Ġ( 5 . 81 Ġwords ) Ġwith Ġvocabulary Ġsize Ġ15 , 733 , Ġwhere Ġ6 , 612 Ġare Ġconsidered Ġunknown Ġtokens Ġfrom ĠStanford ĠGl o VE Ġembed d ings ĠB IB REF 22 Ġ. ĠNamed Ġentities Ġannot ated Ġin Ġthe ĠSnap Capt ions Ġdataset Ġinclude Ġmany Ġof Ġnew Ġand Ġemerging Ġentities , Ġand Ġthey Ġare Ġfound Ġin Ġvarious Ġsurface Ġforms Ġ( var ious Ġnick names , Ġty pos , Ġetc .) ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , ĠSnap Capt ions Ġis Ġthe Ġonly Ġdataset Ġthat Ġcontains Ġnatural Ġimage - ca ption Ġpairs Ġwith Ġexpert - annot ated Ġnamed Ġentities . </s> Bas elines </s> Task : Ġgiven Ġa Ġcaption Ġand Ġa Ġpaired Ġimage Ġ( if Ġused ), Ġthe Ġgoal Ġis Ġto Ġlabel Ġevery Ġtoken Ġin Ġa Ġcaption Ġin ĠB IO Ġscheme Ġ( B : Ġbeginning , ĠI : Ġinside , ĠO : Ġoutside ) ĠB IB REF 27 Ġ. ĠWe Ġreport Ġthe Ġperformance Ġof Ġthe Ġfollowing Ġstate - of - the - art ĠN ER Ġmodels Ġas Ġbas elines , Ġas Ġwell Ġas Ġseveral Ġconfigurations Ġof Ġour Ġproposed Ġapproach Ġto Ġexamine Ġcontributions Ġof Ġeach Ġcomponent Ġ( W : Ġword , ĠC : Ġchar , ĠV : Ġvisual ). </s> Bi - L ST M / CR F Ġ( W Ġonly ): Ġonly Ġtakes Ġword Ġtoken Ġembed d ings Ġ( Stan ford ĠGl o VE ) Ġas Ġinput . ĠThe Ġrest Ġof Ġthe Ġarchitecture Ġis Ġkept Ġthe Ġsame . </s> Bi - L ST M / CR F Ġ+ ĠBi - Char L ST M Ġ( C Ġonly ): Ġonly Ġtakes Ġa Ġcharacter Ġsequence Ġof Ġeach Ġword Ġtoken Ġas Ġinput . Ġ( No Ġword Ġembed d ings ) </s> Bi - L ST M / CR F Ġ+ ĠBi - Char L ST M Ġ( W + C ) ĠB IB REF 0 Ġ: Ġtakes Ġas Ġinput Ġboth Ġword Ġembed d ings Ġand Ġcharacter Ġembed d ings Ġextracted Ġfrom Ġa ĠBi - Char L ST M . ĠEntity ĠL ST M Ġtakes Ġconc aten ated Ġvectors Ġof Ġword Ġand Ġcharacter Ġembed d ings Ġas Ġinput Ġtokens . </s> Bi - L ST M / CR F Ġ+ ĠChar CNN Ġ( W + C ) ĠB IB REF 1 Ġ: Ġuses Ġcharacter Ġembed d ings Ġextracted Ġfrom Ġa ĠCNN Ġinstead . </s> Bi - L ST M / CR F Ġ+ ĠChar CNN Ġ( W + C ) Ġ+ ĠMulti - task ĠB IB REF 8 Ġ: Ġtrains Ġthe Ġmodel Ġto Ġperform Ġboth Ġrecognition Ġ( into Ġmultiple Ġentity Ġtypes ) Ġas Ġwell Ġas Ġsegment ation Ġ( binary ) Ġtasks . </s> ( prop osed ) ĠBi - L ST M / CR F Ġ+ ĠBi - Char L ST M Ġwith Ġmod ality Ġattention Ġ( W + C ): Ġuses Ġthe Ġmod ality Ġattention Ġto Ġmerge Ġword Ġand Ġcharacter Ġembed d ings . </s> ( prop osed ) ĠBi - L ST M / CR F Ġ+ ĠBi - Char L ST M Ġ+ ĠIn ception Ġ( W + C + V ): Ġtakes Ġas Ġinput Ġvisual Ġcontexts Ġextracted Ġfrom ĠIn ception Net Ġas Ġwell , Ġconc aten ated Ġwith Ġword Ġand Ġchar Ġvectors . </s> ( prop osed ) ĠBi - L ST M / CR F Ġ+ ĠBi - Char L ST M Ġ+ ĠIn ception Ġwith Ġmod ality Ġattention Ġ( W + C + V ): Ġuses Ġthe Ġmod ality Ġattention Ġto Ġmerge Ġword , Ġcharacter , Ġand Ġvisual Ġembed d ings Ġas Ġinput Ġto Ġentity ĠL ST M . </s> Results : ĠSnap Capt ions ĠDat as et </s> Table ĠT AB REF 6 Ġshows Ġthe ĠN ER Ġperformance Ġon Ġthe ĠSnap ĠCapt ions Ġdataset . ĠWe Ġreport Ġboth Ġentity Ġtypes Ġrecognition Ġ( PER , ĠLOC , ĠOR G , ĠM ISC ) Ġand Ġnamed Ġentity Ġsegment ation Ġ( named Ġentity Ġor Ġnot ) Ġresults . </s> Parameters : ĠWe Ġtune Ġthe Ġparameters Ġof Ġeach Ġmodel Ġwith Ġthe Ġfollowing Ġsearch Ġspace Ġ( bold Ġindicate Ġthe Ġchoice Ġfor Ġour Ġfinal Ġmodel ): Ġcharacter Ġembed d ings Ġdimension : Ġ{ 25 , Ġ50 , Ġ100 , Ġ150 , Ġ200 , Ġ300 }, Ġword Ġembed d ings Ġsize : Ġ{ 25 , Ġ50 , Ġ100 , Ġ150 , Ġ200 , Ġ300 }, ĠL ST M Ġhidden Ġstates : Ġ{ 25 , Ġ50 , Ġ100 , Ġ150 , Ġ200 , Ġ300 }, Ġand ĠIN LINE FORM 0 Ġdimension : Ġ{ 25 , Ġ50 , Ġ100 , Ġ150 , Ġ200 , Ġ300 }. ĠWe Ġoptimize Ġthe Ġparameters Ġwith ĠAd ag rad ĠB IB REF 28 Ġwith Ġbatch Ġsize Ġ10 , Ġlearning Ġrate Ġ0 . 02 , Ġe ps ilon ĠIN LINE FORM 1 Ġ, Ġand Ġdecay Ġ0 . 0 . </s> Main ĠResults : ĠWhen Ġvisual Ġcontext Ġis Ġavailable Ġ( W + C + V ), Ġwe Ġsee Ġthat Ġthe Ġmodel Ġperformance Ġgreatly Ġimproves Ġover Ġthe Ġtextual Ġmodels Ġ( W + C ), Ġshowing Ġthat Ġvisual Ġcontexts Ġare Ġcomplimentary Ġto Ġtextual Ġinformation Ġin Ġnamed Ġentity Ġrecognition Ġtasks . ĠIn Ġaddition , Ġit Ġcan Ġbe Ġseen Ġthat Ġthe Ġmod ality Ġattention Ġmodule Ġfurther Ġimproves Ġthe Ġentity Ġtype Ġrecognition Ġperformance Ġfor Ġ( W + C + V ). ĠThis Ġresult Ġindicates Ġthat Ġthe Ġmod ality Ġattention Ġis Ġable Ġto Ġfocus Ġon Ġthe Ġmost Ġeffective Ġmod ality Ġ( visual , Ġwords , Ġor Ġcharacters ) Ġadaptive Ġto Ġeach Ġsample Ġto Ġmaximize Ġinformation Ġgain . ĠNote Ġthat Ġour Ġtext - only Ġmodel Ġ( W + C ) Ġwith Ġthe Ġmod ality Ġattention Ġmodule Ġalso Ġsignificantly Ġoutper form Ġthe Ġstate - of - the - art Ġbas elines ĠB IB REF 8 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 0 Ġthat Ġuse Ġthe Ġsame Ġtextual Ġmod alities Ġ( W + C ), Ġshowing Ġthe Ġeffectiveness Ġof Ġthe Ġmod ality Ġattention Ġmodule Ġfor Ġtextual Ġmodels Ġas Ġwell . </s> Error ĠAnalysis : ĠTable ĠT AB REF 17 Ġshows Ġexample Ġcases Ġwhere Ġincorporation Ġof Ġvisual Ġcontexts Ġaffects Ġprediction Ġof Ġnamed Ġentities . ĠFor Ġexample , Ġthe Ġtoken Ġ` c urry ' Ġin Ġthe Ġcaption ĠâĢ ľ The Ġcurry 's Ġ" Ġis Ġpoly sem ous Ġand Ġmay Ġrefer Ġto Ġeither Ġa Ġtype Ġof Ġfood Ġor Ġa Ġfamous Ġbasketball Ġplayer Ġ` Stephen ĠCurry ', Ġand Ġthe Ġsurrounding Ġtextual Ġcontexts Ġdo Ġnot Ġprovide Ġenough Ġinformation Ġto Ġdis amb ig uate Ġit . ĠOn Ġthe Ġother Ġhand , Ġvisual Ġcontexts Ġ( visual Ġtags : Ġ` par ade ', Ġ` urban Ġarea ', Ġ... ) Ġprovide Ġsimilarities Ġto Ġthe Ġtoken 's Ġdistribution al Ġsemantics Ġfrom Ġother Ġtraining Ġexamples Ġ( sn aps Ġfrom ĠâĢ ľ NBA ĠChampionship ĠParade ĠStory "), Ġand Ġthus Ġthe Ġmodel Ġsuccessfully Ġpredicts Ġthe Ġtoken Ġas Ġa Ġnamed Ġentity . ĠSimilarly , Ġwhile Ġthe Ġtext - only Ġmodel Ġerrone ously Ġpredicts Ġ` Apple ' Ġin Ġthe Ġcaption ĠâĢ ľ Grand ma Ġw Ġdat Ġlit ĠApple ĠCris p " Ġas Ġan Ġorganization Ġ( Apple ĠInc .), Ġthe Ġvisual Ġcontexts Ġ( desc ribing Ġobjects Ġrelated Ġto Ġfood ) Ġhelp Ġdis amb ig uate Ġthe Ġtoken , Ġmaking Ġthe Ġmodel Ġpredict Ġit Ġcorrectly Ġas Ġa Ġnon - named Ġentity Ġ( a Ġfruit ). ĠTrend ing Ġentities Ġ( mus icians Ġor ĠDJs Ġsuch Ġas Ġ` C ID ', Ġ` D uke ĠDum ont ', Ġ` Marsh m ello ', Ġetc .) Ġare Ġalso Ġrecognized Ġcorrectly Ġwith Ġstrengthened Ġcontexts Ġfrom Ġvisual Ġinformation Ġ( desc ribing Ġconcert Ġscenes ) Ġdespite Ġlack Ġof Ġsurrounding Ġtextual Ġcontexts . ĠA Ġfew Ġcases Ġwhere Ġvisual Ġcontexts Ġharmed Ġthe Ġperformance Ġmostly Ġinclude Ġvisual Ġtags Ġthat Ġare Ġunrelated Ġto Ġa Ġtoken Ġor Ġits Ġsurrounding Ġtextual Ġcontexts . </s> Visual ization Ġof ĠMod ality ĠAttention : ĠFigure ĠFIG REF 19 Ġvisual izes Ġthe Ġmod ality Ġattention Ġmodule Ġat Ġeach Ġdecoding Ġstep Ġ( each Ġcolumn ), Ġwhere Ġamplified Ġmod ality Ġis Ġrepresented Ġwith Ġdarker Ġcolor , Ġand Ġatten uated Ġmod ality Ġis Ġrepresented Ġwith Ġlighter Ġcolor . </s> For Ġthe Ġimage - a ided Ġmodel Ġ( W + C + V ; Ġupper Ġrow Ġin ĠFigure ĠFIG REF 19 Ġ), Ġwe Ġconfirm Ġthat Ġthe Ġmod ality Ġattention Ġsuccessfully Ġatten uates Ġirrelevant Ġsignals Ġ( self ies , Ġetc .) Ġand Ġampl ifies Ġrelevant Ġmod ality - based Ġcontexts Ġin Ġprediction Ġof Ġa Ġgiven Ġtoken . ĠIn Ġthe Ġexample Ġof ĠâĢ ľ dis ney Ġword Ġessential Ġ= Ġcoffee " Ġwith Ġvisual Ġtags Ġselfie , Ġphone , Ġperson , Ġthe Ġmod ality Ġattention Ġsuccessfully Ġatten uates Ġdistracting Ġvisual Ġsignals Ġand Ġfocuses Ġon Ġtextual Ġmod alities , Ġconsequently Ġmaking Ġcorrect Ġpredictions . ĠThe Ġnamed Ġentities Ġin Ġthe Ġexamples Ġof ĠâĢ ľ Beaut iful Ġnight Ġatop ĠThe ĠSpace ĠNeed le " Ġand ĠâĢ ľ Spl ash ĠMountain " Ġare Ġchallenging Ġto Ġpredict Ġbecause Ġthey Ġare Ġcomposed Ġof Ġcommon Ġnoun s Ġ( space , Ġneedle , Ġsplash , Ġmountain ), Ġand Ġthus Ġthey Ġoften Ġneed Ġadditional Ġcontexts Ġto Ġcorrectly Ġpredict . ĠIn Ġthe Ġtraining Ġdata , Ġvisual Ġcontexts Ġmake Ġstronger Ġindicators Ġfor Ġthese Ġnamed Ġentities Ġ( space Ġneedle , Ġsplash Ġmountain ), Ġand Ġthe Ġmod ality Ġattention Ġmodule Ġsuccessfully Ġattends Ġmore Ġto Ġstronger Ġsignals . </s> For Ġtext - only Ġmodel Ġ( W + C ), Ġwe Ġobserve Ġthat Ġperformance Ġgains Ġmostly Ġcome Ġfrom Ġthe Ġmod ality Ġattention Ġmodule Ġbetter Ġhandling Ġtokens Ġunseen Ġduring Ġtraining Ġor Ġunknown Ġtokens Ġfrom Ġthe Ġpre - trained Ġword Ġembed d ings Ġmatrix . ĠFor Ġexample , Ġwhile ĠWa R ri O o O rs Ġand ĠK ool er ĠM atic Ġare Ġmissing Ġtokens Ġin Ġthe Ġword Ġembed d ings Ġmatrix , Ġit Ġsuccessfully Ġampl ifies Ġcharacter - based Ġcontexts Ġ( capital ized Ġfirst Ġletters , Ġsimilarity Ġto Ġknown Ġentities Ġ` Golden ĠState ĠWarriors ') Ġand Ġsupp resses Ġword - based Ġcontexts Ġ( word Ġembed d ings Ġfor Ġunknown Ġtokens Ġ` Wa R ri O o O rs '), Ġleading Ġto Ġcorrect Ġpredictions . ĠThis Ġresult Ġis Ġsignificant Ġbecause Ġit Ġshows Ġperformance Ġof Ġthe Ġmodel , Ġwith Ġan Ġalmost Ġidentical Ġarchitecture , Ġcan Ġstill Ġimprove Ġwithout Ġhaving Ġto Ġscale Ġthe Ġword Ġembed d ings Ġmatrix Ġindefinitely . </s> Figure ĠFIG REF 19 Ġ( b ) Ġshows Ġthe Ġcases Ġwhere Ġthe Ġmod ality Ġattention Ġled Ġto Ġincorrect Ġpredictions . ĠFor Ġexample , Ġthe Ġmodel Ġpredicts Ġmissing Ġtokens ĠH UU UGE Ġand ĠSh ampoo er Ġincorrectly Ġas Ġnamed Ġentities Ġby Ġampl ifying Ġmisleading Ġcharacter - based Ġcontexts Ġ( capital ized Ġfirst Ġletters ) Ġor Ġvisual Ġcontexts Ġ( concert Ġscenes , Ġassociated Ġcontexts Ġof Ġwhich Ġoften Ġinclude Ġnamed Ġentities Ġin Ġthe Ġtraining Ġdataset ). </s> S ensitivity Ġto ĠWord ĠEmb edd ings ĠVoc abulary ĠSize : ĠIn Ġorder Ġto Ġisolate Ġthe Ġeffectiveness Ġof Ġthe Ġmod ality Ġattention Ġmodule Ġon Ġtextual Ġmodels Ġin Ġhandling Ġmissing Ġtokens , Ġwe Ġreport Ġthe Ġperformance Ġwith Ġvarying Ġword Ġembed d ings Ġvocabulary Ġsizes Ġin ĠTable ĠT AB REF 20 Ġ. ĠBy Ġincreasing Ġthe Ġnumber Ġof Ġmissing Ġtokens Ġartificially Ġby Ġrandomly Ġremoving Ġwords Ġfrom Ġthe Ġword Ġembed d ings Ġmatrix Ġ( original Ġvoc ab Ġsize : Ġ400 K ), Ġwe Ġobserve Ġthat Ġwhile Ġthe Ġoverall Ġperformance Ġde grades , Ġthe Ġmod ality Ġattention Ġmodule Ġis Ġable Ġto Ġsuppress Ġthe Ġpe formance Ġdegradation . ĠNote Ġalso Ġthat Ġthe Ġperformance Ġgap Ġgenerally Ġgets Ġbigger Ġas Ġwe Ġdecrease Ġthe Ġvocabulary Ġsize Ġof Ġthe Ġword Ġembed d ings Ġmatrix . ĠThis Ġresult Ġis Ġsignificant Ġin Ġthat Ġthe Ġmod ality Ġattention Ġis Ġable Ġto Ġimprove Ġthe Ġmodel Ġmore Ġrobust Ġto Ġmissing Ġtokens Ġwithout Ġhaving Ġto Ġtrain Ġan Ġindefinitely Ġlarge Ġword Ġembed d ings Ġmatrix Ġfor Ġarbitrarily Ġnoisy Ġsocial Ġmedia Ġtext Ġdatasets . </s> Con clusions </s> We Ġproposed Ġa Ġnew Ġmultim od al ĠN ER Ġ( M NER : Ġimage Ġ+ Ġtext ) Ġtask Ġon Ġshort Ġsocial Ġmedia Ġposts . ĠWe Ġdemonstrated Ġfor Ġthe Ġfirst Ġtime Ġan Ġeffective ĠM NER Ġsystem , Ġwhere Ġvisual Ġinformation Ġis Ġcombined Ġwith Ġtextual Ġinformation Ġto Ġoutper form Ġtraditional Ġtext - based ĠN ER Ġbas elines . ĠOur Ġwork Ġcan Ġbe Ġapplied Ġto Ġmy ri ads Ġof Ġsocial Ġmedia Ġposts Ġor Ġother Ġarticles Ġacross Ġmultiple Ġplatforms Ġwhich Ġoften Ġinclude Ġboth Ġtext Ġand Ġaccompanying Ġimages . ĠIn Ġaddition , Ġwe Ġproposed Ġthe Ġmod ality Ġattention Ġmodule , Ġa Ġnew Ġneural Ġmechanism Ġwhich Ġlearns Ġoptimal Ġintegration Ġof Ġdifferent Ġmodes Ġof Ġcorrelated Ġinformation . ĠIn Ġessence , Ġthe Ġmod ality Ġattention Ġlearns Ġto Ġatten uate Ġirrelevant Ġor Ġunin form ative Ġmod al Ġinformation Ġwhile Ġampl ifying Ġthe Ġprimary Ġmod ality Ġto Ġextract Ġbetter Ġoverall Ġrepresentations . ĠWe Ġshowed Ġthat Ġthe Ġmod ality Ġattention Ġbased Ġmodel Ġoutper forms Ġother Ġstate - of - the - art Ġbas elines Ġwhen Ġtext Ġwas Ġthe Ġonly Ġmod ality Ġavailable , Ġby Ġbetter Ġcombining Ġword Ġand Ġcharacter Ġlevel Ġinformation .