Yes
BOOLEAN
Yes
BOOLEAN
Practical evaluation of GTD is currently only possible on synthetic data. We construct a range of datasets designed for image captioning evaluation. We call this diagnostic evaluation benchmark ShapeWorldICE (ShapeWorld for Image Captioning Evaluation). We illustrate the evaluation of specific image captioning models on ShapeWorldICE. We empirically demonstrate that the existing metrics BLEU and SPICE do not capture true caption-image agreement in all scenarios, while the GTD framework allows a fine-grained investigation of how well existing models cope with varied visual situations and linguistic constructions.
In this work, we develop the evaluation datasets within the ShapeWorld framework. ShapeWorld is a controlled data generation framework consisting of abstract colored shapes (see Figure FIGREF1 for an example). We use ShapeWorld to generate training and evaluation data for two major reasons. ShapeWorld supports customized data generation according to user specification, which enables a variety of model inspections in terms of language construction, visual complexity and reasoning ability. Another benefit is that each training and test instance generated in ShapeWorld is returned as a triplet of $<$image, caption, world model$>$. The world model stores information about the underlying microworld used to generate an image and a descriptive caption, internally represented as a list of entities with their attributes, such as shape, color, position. During data generation, ShapeWorld randomly samples a world model from a set of available entities and attributes. The generated world model is then used to realize a corresponding instance consisting of image and caption. The world model gives the actual semantic information contained in an image, which allows evaluation of caption truthfulness.
1912.08960
Introduction </s> Autom atically Ġgenerating Ġtext Ġto Ġdescribe Ġthe Ġcontent Ġof Ġimages , Ġalso Ġknown Ġas Ġimage Ġcaption ing , Ġis Ġa Ġmultim od al Ġtask Ġof Ġconsiderable Ġinterest Ġin Ġboth Ġthe Ġcomputer Ġvision Ġand Ġthe ĠN LP Ġcommunities . ĠImage Ġcaption ing Ġcan Ġbe Ġframed Ġas Ġa Ġtranslation Ġtask Ġfrom Ġan Ġimage Ġto Ġa Ġdescriptive Ġnatural Ġlanguage Ġstatement . ĠMany Ġexisting Ġcaption ing Ġmodels ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 Ġfollow Ġthe Ġtypical Ġenc oder - dec oder Ġframework Ġwhere Ġa Ġconv olution al Ġnetwork Ġis Ġused Ġto Ġcond ense Ġimages Ġinto Ġvisual Ġfeature Ġrepresentations , Ġcombined Ġwith Ġa Ġrecurrent Ġnetwork Ġfor Ġlanguage Ġgeneration . ĠWhile Ġthese Ġmodels Ġdemonstrate Ġpromising Ġresults , Ġquant ifying Ġimage Ġcaption ing Ġperformance Ġremains Ġa Ġchallenging Ġproblem , Ġin Ġa Ġsimilar Ġway Ġto Ġother Ġgener ative Ġtasks ĠB IB REF 4 , ĠB IB REF 5 . </s> E val uating Ġcandidate Ġcapt ions Ġfor Ġhuman Ġpreference Ġis Ġslow Ġand Ġlabor ious . ĠTo Ġalleviate Ġthis Ġproblem , Ġmany Ġautomatic Ġevaluation Ġmetrics Ġhave Ġbeen Ġproposed , Ġsuch Ġas ĠB LE U ĠB IB REF 6 , ĠMET E OR ĠB IB REF 7 , ĠR OU GE ĠB IB REF 8 Ġand ĠC ID Er ĠB IB REF 9 . ĠThese Ġn - gram - based Ġmetrics Ġevaluate Ġcaption ing Ġperformance Ġbased Ġon Ġsurface Ġsimilarity Ġbetween Ġa Ġcandidate Ġcaption Ġand Ġreference Ġstatements . ĠA Ġmore Ġrecent Ġevaluation Ġmetric Ġfor Ġimage Ġcaption ing Ġis ĠSP ICE ĠB IB REF 10 , Ġwhich Ġtakes Ġinto Ġaccount Ġsemantic Ġpropos itional Ġcontent Ġof Ġgenerated Ġcapt ions Ġby Ġscoring Ġa Ġcaption Ġbased Ġupon Ġa Ġgraph - based Ġsemantic Ġrepresentation Ġtransformed Ġfrom Ġreference Ġcapt ions . </s> The Ġrationale Ġbehind Ġthese Ġevaluation Ġmetrics Ġis Ġthat Ġhuman Ġreference Ġcapt ions Ġserve Ġas Ġan Ġapproximate Ġtarget Ġand Ġcomparing Ġmodel Ġoutputs Ġto Ġthis Ġtarget Ġis Ġa Ġproxy Ġfor Ġhow Ġwell Ġa Ġsystem Ġperforms . ĠThus , Ġa Ġcandidate Ġcaption Ġis Ġnot Ġdirectly Ġevaluated Ġwith Ġrespect Ġto Ġimage Ġcontent , Ġbut Ġcompared Ġto Ġa Ġset Ġof Ġhuman Ġstatements Ġabout Ġthat Ġimage . </s> However , Ġin Ġimage Ġcaption ing , Ġvisual Ġscenes Ġwith Ġmultiple Ġobjects Ġand Ġrelations Ġcorrespond Ġto Ġa Ġdiversity Ġof Ġvalid Ġdescriptions . ĠConsider Ġthe Ġexample Ġimage Ġand Ġcapt ions Ġfrom Ġthe ĠShape World Ġframework ĠB IB REF 11 Ġshown Ġin ĠFigure ĠFIG REF 1 . ĠThe Ġfirst Ġthree Ġcapt ions Ġare Ġtrue Ġstatements Ġabout Ġthe Ġimage Ġand Ġexpress Ġrelevant Ġideas , Ġbut Ġdescribe Ġdifferent Ġobjects , Ġattributes Ġand Ġspatial Ġrelationships , Ġwhile Ġthe Ġfourth Ġcaption Ġis Ġwrong Ġdespite Ġreferring Ġto Ġthe Ġsame Ġobjects Ġas Ġin Ġthe Ġthird Ġcaption . ĠThis Ġcasts Ġdoubt Ġon Ġthe Ġsu fficiency Ġof Ġusing Ġa Ġset Ġof Ġreference Ġcapt ions Ġto Ġapproximate Ġthe Ġcontent Ġof Ġan Ġimage . ĠWe Ġargue Ġthat , Ġwhile Ġexisting Ġmetrics Ġhave Ġundeniably Ġbeen Ġuseful Ġfor Ġreal - world Ġcaption ing Ġevaluation , Ġtheir Ġfocus Ġon Ġapproximate Ġsurface Ġcomparison Ġlimits Ġdeeper Ġinsights Ġinto Ġthe Ġlearning Ġprocess Ġand Ġeventual Ġbehavior Ġof Ġcaption ing Ġmodels . </s> To Ġaddress Ġthis Ġproblem , Ġwe Ġpropose Ġa Ġset Ġof Ġprincipled Ġevaluation Ġcriteria Ġwhich Ġevaluate Ġimage Ġcaption ing Ġmodels Ġfor Ġgram matical ity , Ġtruth fulness Ġand Ġdiversity Ġ( GT D ). ĠThese Ġcriteria Ġcorrespond Ġto Ġnecessary Ġrequirements Ġfor Ġimage Ġcaption ing Ġsystems : Ġ( a ) Ġthat Ġthe Ġoutput Ġis Ġgram matical , Ġ( b ) Ġthat Ġthe Ġoutput Ġstatement Ġis Ġtrue Ġwith Ġrespect Ġto Ġthe Ġimage , Ġand Ġ( c ) Ġthat Ġoutputs Ġare Ġdiverse Ġand Ġmirror Ġthe Ġvariability Ġof Ġtraining Ġcapt ions . </s> P ractical Ġevaluation Ġof ĠGT D Ġis Ġcurrently Ġonly Ġpossible Ġon Ġsynthetic Ġdata . ĠWe Ġconstruct Ġa Ġrange Ġof Ġdatasets Ġdesigned Ġfor Ġimage Ġcaption ing Ġevaluation . ĠWe Ġcall Ġthis Ġdiagnostic Ġevaluation Ġbenchmark ĠShape World ICE Ġ( Shape World Ġfor ĠImage ĠCaption ing ĠEvaluation ). ĠWe Ġillustrate Ġthe Ġevaluation Ġof Ġspecific Ġimage Ġcaption ing Ġmodels Ġon ĠShape World ICE . ĠWe Ġempir ically Ġdemonstrate Ġthat Ġthe Ġexisting Ġmetrics ĠB LE U Ġand ĠSP ICE Ġdo Ġnot Ġcapture Ġtrue Ġcaption - image Ġagreement Ġin Ġall Ġscenarios , Ġwhile Ġthe ĠGT D Ġframework Ġallows Ġa Ġfine - gr ained Ġinvestigation Ġof Ġhow Ġwell Ġexisting Ġmodels Ġcope Ġwith Ġvaried Ġvisual Ġsituations Ġand Ġlinguistic Ġconstruct ions . </s> We Ġbelieve Ġthat Ġas Ġa Ġsupplementary Ġevaluation Ġmethod Ġto Ġreal - world Ġmetrics , Ġthe ĠGT D Ġframework Ġprovides Ġevaluation Ġinsights Ġthat Ġare Ġsufficiently Ġinteresting Ġto Ġmotivate Ġfuture Ġwork . </s> Related Ġwork Ġ: :: ĠEx isting Ġevaluation Ġof Ġimage Ġcaption ing </s> As Ġa Ġnatural Ġlanguage Ġgeneration Ġtask , Ġimage Ġcaption ing Ġfrequently Ġuses Ġevaluation Ġmetrics Ġsuch Ġas ĠB LE U ĠB IB REF 6 , ĠMET E OR ĠB IB REF 7 , ĠR OU GE ĠB IB REF 8 Ġand ĠC ID Er ĠB IB REF 9 . ĠThese Ġmetrics Ġuse Ġn - gram Ġsimilarity Ġbetween Ġthe Ġcandidate Ġcaption Ġand Ġreference Ġcapt ions Ġto Ġapproximate Ġthe Ġcorrelation Ġbetween Ġa Ġcandidate Ġcaption Ġand Ġthe Ġassociated Ġground Ġtruth . ĠSP ICE ĠB IB REF 10 Ġis Ġa Ġmore Ġrecent Ġmetric Ġspecifically Ġdesigned Ġfor Ġimage Ġcaption ing . ĠFor ĠSP ICE , Ġboth Ġthe Ġcandidate Ġcaption Ġand Ġreference Ġcapt ions Ġare Ġparsed Ġto Ġscene Ġgraphs , Ġand Ġthe Ġagreement Ġbetween Ġtu ples Ġextracted Ġfrom Ġthese Ġscene Ġgraphs Ġis Ġexamined . ĠSP ICE Ġmore Ġclosely Ġrelates Ġto Ġour Ġtruth fulness Ġevaluation Ġthan Ġthe Ġother Ġmetrics , Ġbut Ġit Ġstill Ġuses Ġoverlap Ġcomparison Ġto Ġreference Ġcapt ions Ġas Ġa Ġproxy Ġto Ġground Ġtruth . ĠIn Ġcontrast , Ġour Ġtruth fulness Ġmetric Ġdirectly Ġevaluates Ġa Ġcandidate Ġcaption Ġagainst Ġa Ġmodel Ġof Ġthe Ġactual Ġvisual Ġcontent . </s> Many Ġresearchers Ġhave Ġpointed Ġout Ġproblems Ġwith Ġexisting Ġreference - based Ġmetrics Ġincluding Ġlow Ġcorrelations Ġwith Ġhuman Ġjudgment ĠB IB REF 12 , ĠB IB REF 10 , ĠB IB REF 13 Ġand Ġstrong Ġbas elines Ġusing Ġnearest - ne igh bor Ġmethods ĠB IB REF 14 Ġor Ġrelying Ġsolely Ġon Ġobject Ġdetection ĠB IB REF 15 . ĠFundamental Ġconcerns Ġhave Ġbeen Ġraised Ġwith Ġrespect Ġto ĠB LE U , Ġincluding Ġvariability Ġin Ġparameter ization Ġand Ġprecise Ġscore Ġcalculation Ġleading Ġto Ġsignificantly Ġdifferent Ġresults ĠB IB REF 16 . ĠIts Ġvalidity Ġas Ġa Ġmetric Ġfor Ġtasks Ġother Ġthan Ġmachine Ġtranslation Ġhas Ġbeen Ġquestioned ĠB IB REF 17 , Ġparticularly Ġfor Ġtasks Ġfor Ġwhich Ġthe Ġoutput Ġcontent Ġis Ġnot Ġnarrowly Ġconstrained , Ġlike Ġdialogue ĠB IB REF 18 . </s> Some Ġrecent Ġwork Ġfocuses Ġon Ġincreasing Ġthe Ġdiversity Ġof Ġgenerated Ġcapt ions , Ġfor Ġwhich Ġvarious Ġmeasures Ġare Ġproposed . ĠDev lin Ġet Ġal . ĠB IB REF 19 Ġexplored Ġthe Ġconcept Ġof Ġcaption Ġdiversity Ġby Ġevaluating Ġperformance Ġon Ġcomposition ally Ġnovel Ġimages . Ġvan ĠMil ten burg Ġet Ġal ĠB IB REF 20 Ġframed Ġimage Ġcaption ing Ġas Ġa Ġword Ġrecall Ġtask Ġand Ġproposed Ġseveral Ġmetrics , Ġpredominantly Ġfocusing Ġon Ġdiversity Ġat Ġthe Ġword Ġlevel . ĠHowever , Ġthis Ġdirection Ġis Ġstill Ġrelatively Ġnew Ġand Ġlacks Ġstandardized Ġbenchmarks Ġand Ġmetrics . </s> Related Ġwork Ġ: :: ĠSynt hetic Ġdatasets </s> Recently , Ġmany Ġsynthetic Ġdatasets Ġhave Ġbeen Ġproposed Ġas Ġdiagnostic Ġtools Ġfor Ġdeep Ġlearning Ġmodels , Ġsuch Ġas ĠCLE VR ĠB IB REF 21 Ġfor Ġvisual Ġquestion Ġanswering Ġ( V Q A ), Ġthe Ġb Ab I Ġtasks ĠB IB REF 22 Ġfor Ġtext Ġunderstanding Ġand Ġreasoning , Ġand ĠShape World ĠB IB REF 11 Ġfor Ġvisually Ġgrounded Ġlanguage Ġunderstanding . ĠThe Ġprimary Ġmotivation Ġis Ġto Ġreduce Ġcomplexity Ġwhich Ġis Ġconsidered Ġirrelevant Ġto Ġthe Ġevaluation Ġfocus , Ġto Ġenable Ġbetter Ġcontrol Ġover Ġthe Ġdata , Ġand Ġto Ġprovide Ġmore Ġdetailed Ġinsights Ġinto Ġstrengths Ġand Ġlimitations Ġof Ġexisting Ġmodels . </s> In Ġthis Ġwork , Ġwe Ġdevelop Ġthe Ġevaluation Ġdatasets Ġwithin Ġthe ĠShape World Ġframework . ĠShape World Ġis Ġa Ġcontrolled Ġdata Ġgeneration Ġframework Ġconsisting Ġof Ġabstract Ġcolored Ġshapes Ġ( see ĠFigure ĠFIG REF 1 Ġfor Ġan Ġexample ). ĠWe Ġuse ĠShape World Ġto Ġgenerate Ġtraining Ġand Ġevaluation Ġdata Ġfor Ġtwo Ġmajor Ġreasons . ĠShape World Ġsupports Ġcustomized Ġdata Ġgeneration Ġaccording Ġto Ġuser Ġspecification , Ġwhich Ġenables Ġa Ġvariety Ġof Ġmodel Ġinspections Ġin Ġterms Ġof Ġlanguage Ġconstruction , Ġvisual Ġcomplexity Ġand Ġreasoning Ġability . ĠAnother Ġbenefit Ġis Ġthat Ġeach Ġtraining Ġand Ġtest Ġinstance Ġgenerated Ġin ĠShape World Ġis Ġreturned Ġas Ġa Ġtriple t Ġof Ġ$ < $ image , Ġcaption , Ġworld Ġmodel $ > $. ĠThe Ġworld Ġmodel Ġstores Ġinformation Ġabout Ġthe Ġunderlying Ġmicrow orld Ġused Ġto Ġgenerate Ġan Ġimage Ġand Ġa Ġdescriptive Ġcaption , Ġinternally Ġrepresented Ġas Ġa Ġlist Ġof Ġentities Ġwith Ġtheir Ġattributes , Ġsuch Ġas Ġshape , Ġcolor , Ġposition . ĠDuring Ġdata Ġgeneration , ĠShape World Ġrandomly Ġsamples Ġa Ġworld Ġmodel Ġfrom Ġa Ġset Ġof Ġavailable Ġentities Ġand Ġattributes . ĠThe Ġgenerated Ġworld Ġmodel Ġis Ġthen Ġused Ġto Ġrealize Ġa Ġcorresponding Ġinstance Ġconsisting Ġof Ġimage Ġand Ġcaption . ĠThe Ġworld Ġmodel Ġgives Ġthe Ġactual Ġsemantic Ġinformation Ġcontained Ġin Ġan Ġimage , Ġwhich Ġallows Ġevaluation Ġof Ġcaption Ġtruth fulness . </s> GT D ĠEvaluation ĠFramework </s> In Ġthe Ġfollowing Ġwe Ġintroduce ĠGT D Ġin Ġmore Ġdetail , Ġconsider Ġit Ġas Ġan Ġevaluation Ġprotocol Ġcovering Ġnecessary Ġaspects Ġof Ġthe Ġmultif acet ed Ġcaption ing Ġtask , Ġrather Ġthan Ġa Ġspecific Ġmetric . </s> GT D ĠEvaluation ĠFramework Ġ: :: ĠGram matical ity </s> An Ġessential Ġcriterion Ġfor Ġan Ġimage Ġcaption ing Ġmodel Ġis Ġthat Ġthe Ġcapt ions Ġgenerated Ġare Ġgram matically Ġwell - formed . ĠFully Ġaccurate Ġassessment Ġof Ġgram matical ity Ġin Ġa Ġgeneral Ġcontext Ġis Ġitself Ġa Ġdifficult Ġtask , Ġbut Ġbecomes Ġmore Ġfeasible Ġin Ġa Ġvery Ġconstrained Ġcontext Ġlike Ġour Ġdiagnostic Ġlanguage Ġdata . ĠWe Ġtake Ġparse ability Ġwith Ġthe ĠEnglish ĠResource ĠGram mar ĠB IB REF 23 Ġas Ġa Ġsurrogate Ġfor Ġgram matical ity , Ġmeaning Ġthat Ġa Ġsentence Ġis Ġconsidered Ġgram matically Ġwell - formed Ġif Ġwe Ġobtain Ġa Ġparse Ġusing Ġthe ĠER G . </s> The ĠER G Ġis Ġa Ġbroad - co verage Ġgrammar Ġbased Ġon Ġthe Ġhead - driven Ġphrase Ġstructure Ġgrammar Ġ( H PS G ) Ġframework . ĠIt Ġis Ġlingu istically Ġprecise : Ġsentences Ġonly Ġparse Ġif Ġthey Ġare Ġvalid Ġaccording Ġto Ġits Ġhand - built Ġrules . ĠIt Ġis Ġdesigned Ġto Ġbe Ġgeneral - purpose : Ġverified Ġcoverage Ġis Ġaround Ġ80 % Ġfor ĠWikipedia , Ġand Ġover Ġ90 % Ġfor Ġcorpor a Ġwith Ġshorter Ġsentences Ġand Ġmore Ġlimited Ġvocabulary Ġ( for Ġdetails Ġsee ĠB IB REF 24 Ġflick inger 2011 acc uracy ). ĠSince Ġthe ĠShape World Ġtraining Ġdata ĠâĢĵ Ġthe Ġonly Ġlanguage Ġsource Ġfor Ġmodels Ġto Ġlearn Ġfrom ĠâĢĵ Ġis Ġgenerated Ġusing Ġthe Ġsame Ġgrammar , Ġthe ĠER G Ġhas Ġ$\ sim Ġ$ 100 % Ġcoverage Ġof Ġgram matical ity Ġin Ġthe Ġmodel Ġoutput Ġspace . </s> GT D ĠEvaluation ĠFramework Ġ: :: ĠTruth fulness </s> The Ġsecond Ġaspect Ġwe Ġinvestigate Ġis Ġtruth fulness , Ġthat Ġis , Ġwhether Ġa Ġcandidate Ġcaption Ġis Ġcompatible Ġwith Ġthe Ġcontent Ġof Ġthe Ġimage Ġit Ġis Ġsupposed Ġto Ġdescribe . ĠWe Ġevaluate Ġcaption Ġtruth fulness Ġon Ġthe Ġbasis Ġof Ġa Ġlingu istically - mot ivated Ġapproach Ġusing Ġformal Ġsemantics . ĠWe Ġconvert Ġthe Ġoutput Ġof Ġthe ĠER G Ġparse Ġfor Ġa Ġgram matical Ġcaption Ġto Ġa ĠDepend ency ĠMin imal ĠRec ursion ĠSem antics Ġ( DM RS ) Ġgraph Ġusing Ġthe Ġp yd m rs Ġtool ĠB IB REF 25 . ĠEach Ġconverted ĠDM RS Ġis Ġa Ġlogical Ġsemantic Ġgraph Ġrepresentation Ġcorresponding Ġto Ġthe Ġcaption . ĠWe Ġconstruct Ġa Ġlogical Ġproposition Ġfrom Ġthe ĠDM RS Ġgraph , Ġand Ġevaluate Ġit Ġagainst Ġthe Ġactual Ġworld Ġmodel Ġof Ġthe Ġcorresponding Ġimage . ĠA Ġcaption Ġcan Ġbe Ġsaid Ġto Ġagree Ġwith Ġan Ġimage Ġonly Ġif Ġthe Ġproposition Ġevaluates Ġas Ġtrue Ġon Ġthe Ġbasis Ġof Ġthe Ġworld Ġmodel . ĠBy Ġexamining Ġthe Ġlogical Ġagreement Ġbetween Ġa Ġcaption Ġrepresentation Ġand Ġa Ġworld Ġmodel , Ġwe Ġcan Ġcheck Ġwhether Ġthe Ġsemantics Ġof Ġthis Ġcaption Ġagrees Ġwith Ġthe Ġvisual Ġcontent Ġwhich Ġthe Ġworld Ġmodel Ġrepresents . ĠThus Ġwe Ġdo Ġnot Ġrely Ġon Ġa Ġset Ġof Ġcapt ions Ġas Ġa Ġsurrogate Ġfor Ġthe Ġcontent Ġof Ġan Ġimage , Ġbut Ġinstead Ġleverage Ġthe Ġfact Ġthat Ġwe Ġhave Ġthe Ġground Ġtruth , Ġthus Ġenabling Ġthe Ġevaluation Ġof Ġtrue Ġimage - ca ption Ġagreement . </s> GT D ĠEvaluation ĠFramework Ġ: :: ĠDiversity </s> While Ġgram matical ity Ġand Ġtruth fulness Ġare Ġessential Ġrequirements Ġfor Ġimage Ġcapt ions , Ġthese Ġcriteria Ġalone Ġcan Ġeasily Ġbe ĠâĢ ľ g amed âĢ Ŀ Ġby Ġspecializing Ġon Ġa Ġsmall Ġset Ġof Ġgeneric Ġstatements Ġwhich Ġare Ġtrue Ġmost Ġof Ġthe Ġtime . ĠIn Ġthe Ġcontext Ġof Ġabstract Ġshapes , Ġsuch Ġcapt ions Ġinclude Ġexamples Ġlike ĠâĢ ľ There Ġis Ġa Ġshape âĢ Ŀ Ġor ĠâĢ ľ At Ġleast Ġzero Ġshapes Ġare Ġblue âĢ Ŀ Ġ( which Ġis Ġtechnically Ġtrue Ġeven Ġif Ġthere Ġis Ġno Ġblue Ġshape ). ĠThis Ġmotiv ates Ġthe Ġthird Ġfundamental Ġrequirement Ġof Ġcaption ing Ġoutput Ġto Ġbe Ġdiverse . </s> As ĠShape World ICE Ġexploits Ġa Ġlimited Ġsize Ġof Ġopen - class Ġwords , Ġwe Ġemphasize Ġthe Ġdiversity Ġin ĠShape World ICE Ġat Ġthe Ġsentence Ġlevel Ġrather Ġthan Ġthe Ġword Ġlevel . ĠSince Ġthe Ġground - truth Ġreference Ġcapt ions Ġin ĠShape World Ġare Ġrandomly Ġsampled , Ġwe Ġtake Ġthe Ġsampled Ġcapt ions Ġaccompanying Ġthe Ġtest Ġimages Ġas Ġa Ġproxy Ġfor Ġoptimal Ġcaption Ġdiversity , Ġand Ġcompare Ġit Ġwith Ġthe Ġempirical Ġoutput Ġdiversity Ġof Ġthe Ġevaluated Ġmodel Ġon Ġthese Ġtest Ġimages . ĠPract ically , Ġwe Ġlook Ġat Ġlanguage Ġconstruct ions Ġused Ġand Ġcompute Ġthe Ġcorresponding Ġdiversity Ġscore Ġas Ġthe Ġratio Ġof Ġobserved Ġnumber Ġversus Ġoptimal Ġnumber : </s> Language Ġconstruct ions Ġhere Ġcorrespond Ġto Ġreduced Ġcaption Ġrepresentations Ġwhich Ġonly Ġrecord Ġwhether Ġan Ġobject Ġis Ġdescribed Ġby Ġshape Ġ( e . g ., ĠâĢ ľ square âĢ Ŀ ), Ġcolor Ġ( e . g ., ĠâĢ ľ red Ġshape âĢ Ŀ ) Ġor Ġcolor - shape Ġcombination Ġ( e . g ., ĠâĢ ľ red Ġsquare âĢ Ŀ ). ĠSo Ġthe Ġstatement ĠâĢ ľ A Ġsquare Ġis Ġred âĢ Ŀ Ġand ĠâĢ ľ A Ġcircle Ġis Ġblue âĢ Ŀ Ġare Ġconsidered Ġthe Ġsame , Ġwhile ĠâĢ ľ A Ġshape Ġis Ġred âĢ Ŀ Ġis Ġdifferent . </s> Exper imental ĠSetup Ġ: :: ĠDat as ets </s> We Ġdevelop Ġa Ġvariety Ġof ĠShape World ICE Ġdatasets , Ġwith Ġa Ġsimilar Ġidea Ġto Ġthe ĠâĢ ľ skill Ġtasks âĢ Ŀ Ġin Ġthe Ġb Ab I Ġframework ĠB IB REF 22 . ĠTable ĠT AB REF 4 Ġgives Ġan Ġoverview Ġfor Ġdifferent ĠShape World ICE Ġdatasets Ġwe Ġuse Ġin Ġthis Ġpaper . ĠWe Ġconsider Ġthree Ġdifferent Ġtypes Ġof Ġcaption ing Ġtasks , Ġeach Ġof Ġwhich Ġfocuses Ġon Ġa Ġdistinct Ġaspect Ġof Ġreasoning Ġabilities . ĠEx ist ential Ġdescriptions Ġexamine Ġwhether Ġa Ġcertain Ġobject Ġis Ġpresent Ġin Ġan Ġimage . ĠSp atial Ġdescriptions Ġidentify Ġspatial Ġrelationships Ġamong Ġvisual Ġobjects . ĠQuant ification Ġdescriptions Ġinvolve Ġcount - based Ġand Ġratio - based Ġstatements , Ġwith Ġan Ġexplicit Ġfocus Ġon Ġinspecting Ġmodels Ġfor Ġtheir Ġcounting Ġability . ĠWe Ġdevelop Ġtwo Ġvariants Ġfor Ġeach Ġtype Ġof Ġdataset Ġto Ġenable Ġdifferent Ġlevels Ġof Ġvisual Ġcomplexity Ġor Ġspecific Ġaspects Ġof Ġthe Ġsame Ġreasoning Ġtype . ĠAll Ġthe Ġtraining Ġand Ġtest Ġcapt ions Ġsampled Ġin Ġthis Ġwork Ġare Ġin ĠEnglish . </s> Each Ġdataset Ġvariant Ġconsists Ġof Ġaround Ġ200 k Ġtraining Ġinstances Ġand Ġ4 , 09 6 Ġvalidation Ġinstances , Ġplus Ġ4 , 09 6 Ġtest Ġinstances . ĠEach Ġtraining Ġinstance Ġconsists Ġof Ġan Ġimage Ġand Ġa Ġreference Ġcaption . ĠAt Ġtest Ġtime , Ġonly Ġthe Ġtest Ġimages Ġare Ġavailable Ġto Ġthe Ġevaluated Ġmodels . ĠUnder lying Ġworld Ġmodels Ġare Ġkept Ġfrom Ġthe Ġmodels Ġand Ġare Ġused Ġfor Ġlater ĠGT D Ġevaluation . ĠFor Ġeach Ġtest Ġinstance , Ġwe Ġsample Ġten Ġreference Ġcapt ions Ġof Ġthe Ġsame Ġdistribution Ġas Ġthe Ġtraining Ġcapt ions Ġto Ġenable Ġthe Ġcomparison Ġof Ġour Ġproposed Ġmetrics Ġto ĠB LE U Ġand ĠSP ICE . ĠWe Ġfine - t une Ġour Ġmodel Ġhyper param eters Ġbased Ġon Ġthe Ġperformance Ġon Ġthe Ġvalidation Ġset . ĠAll Ġreported Ġresults Ġare Ġmeasured Ġon Ġthe Ġtest Ġsplit Ġwith Ġthe Ġparameters Ġyielding Ġthe Ġbest Ġvalidation Ġperformance . </s> Exper imental ĠSetup Ġ: :: ĠModels </s> We Ġexperiment Ġwith Ġtwo Ġimage Ġcaption ing Ġmodels : Ġthe ĠShow & Tell Ġmodel ĠB IB REF 0 Ġand Ġthe ĠL RC N 1 u Ġmodel ĠB IB REF 1 . ĠBoth Ġmodels Ġfollow Ġthe Ġbasic Ġenc oder - dec oder Ġarchitecture Ġdesign Ġthat Ġuses Ġa ĠCNN Ġenc oder Ġto Ġcond ense Ġthe Ġvisual Ġinformation Ġinto Ġan Ġimage Ġembed ding , Ġwhich Ġin Ġturn Ġconditions Ġan ĠL ST M Ġdec oder Ġto Ġgenerate Ġa Ġnatural Ġlanguage Ġcaption . ĠThe Ġmain Ġdifference Ġbetween Ġthe Ġtwo Ġmodels Ġis Ġthe Ġway Ġthey Ġcondition Ġthe Ġdec oder . ĠThe ĠShow & Tell Ġmodel Ġfeeds Ġthe Ġimage Ġembed ding Ġas Ġthe ĠâĢ ľ pred ec essor Ġword Ġembed ding âĢ Ŀ Ġto Ġthe Ġfirst Ġproduced Ġword , Ġwhile Ġthe ĠL RC N 1 u Ġmodel Ġconc aten ates Ġthe Ġimage Ġfeatures Ġwith Ġthe Ġembedded Ġprevious Ġword Ġas Ġthe Ġinput Ġto Ġthe Ġsequence Ġmodel Ġat Ġeach Ġtime Ġstep . </s> We Ġfollow Ġthe Ġcommon Ġpractice Ġin Ġimage Ġcaption ing Ġto Ġuse Ġa ĠCNN Ġcomponent Ġpret rained Ġon Ġobject Ġdetection Ġand Ġfine - t une Ġits Ġparameters Ġon Ġthe Ġimage Ġcaption ing Ġtask . ĠThe Ġenc oder Ġand Ġdec oder Ġcomponents Ġare Ġjointly Ġoptimized Ġwith Ġrespect Ġto Ġthe Ġstandard Ġcross - ent ropy Ġsequence Ġloss Ġon Ġthe Ġrespective ĠShape World ICE Ġdataset . ĠFor Ġall Ġour Ġexperiments , Ġwe Ġtrain Ġmodels Ġend - to - end Ġfor Ġa Ġfixed Ġnumber Ġof Ġ100 k Ġiterations Ġwith Ġa Ġbatch Ġsize Ġof Ġ64 . ĠWe Ġuse ĠAdam Ġoptimization ĠB IB REF 26 Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 001 . ĠWord Ġembed d ings Ġare Ġrandomly Ġinitialized Ġand Ġjointly Ġtrained Ġduring Ġthe Ġtraining . </s> Results </s> We Ġtrain Ġand Ġevaluate Ġthe ĠShow & Tell Ġand ĠL RC N 1 u Ġmodels Ġon Ġthe ĠShape World ICE Ġdatasets . ĠHere Ġwe Ġdiscuss Ġin Ġdetail Ġthe Ġdiagnostic Ġresults Ġof Ġthese Ġexperiments . ĠDuring Ġtraining , Ġwe Ġperiodically Ġrecord Ġmodel Ġoutput Ġon Ġthe Ġtest Ġimages , Ġto Ġbe Ġable Ġto Ġanalyze Ġthe Ġdevelopment Ġof Ġour Ġevaluation Ġmetrics Ġthroughout Ġthe Ġprocess . ĠWe Ġalso Ġcompute ĠB LE U - 4 Ġscores Ġand ĠSP ICE Ġscores Ġof Ġgenerated Ġcapt ions Ġfor Ġcomparison , Ġusing Ġ10 Ġreference Ġcapt ions Ġper Ġtest Ġimage . </s> L RC N 1 u Ġexhibits Ġclearly Ġsuperior Ġperformance Ġin Ġterms Ġof Ġtruth fulness . ĠWe Ġstart Ġoff Ġby Ġcomparing Ġperformance Ġof Ġthe ĠShow & Tell Ġmodel Ġand Ġthe ĠL RC N 1 u Ġmodel , Ġsee ĠFigure ĠFIG REF 8 . ĠWhile Ġboth Ġmodels Ġlearn Ġto Ġproduce Ġgram matical Ġsentences Ġearly Ġon , Ġit Ġcan Ġbe Ġseen Ġthat ĠL RC N 1 u Ġis Ġclearly Ġsuperior Ġin Ġterms Ġof Ġtruth fulness , Ġachieving Ġ100 % Ġhalfway Ġthrough Ġtraining , Ġwhereas ĠShow & Tell Ġonly Ġslowly Ġreaches Ġaround Ġ90 % Ġby Ġthe Ġend Ġof Ġ100 k Ġiterations . ĠThis Ġindicates Ġthat Ġincorporating Ġvisual Ġfeatures Ġat Ġevery Ġgeneration Ġstep Ġis Ġbeneficial Ġfor Ġproducing Ġtrue Ġcapt ions . ĠThe Ġdiversity Ġratios Ġof Ġcapt ions Ġgenerated Ġby Ġtwo Ġmodels Ġboth Ġincrease Ġsubstantially Ġas Ġthe Ġtraining Ġprogresses , Ġwith ĠL RC N 1 u Ġexhibiting Ġa Ġslightly Ġgreater Ġcaption Ġdiversity Ġat Ġthe Ġend Ġof Ġtraining . </s> We Ġobserved Ġsimilar Ġresults Ġon Ġother ĠShape World ICE Ġdatasets Ġthat Ġwe Ġexperimented Ġwith , Ġvalid ating Ġthe Ġsuperiority Ġof ĠL RC N 1 u Ġover ĠShow & Tell Ġon ĠShape World ICE . ĠConsequently , Ġwe Ġdecided Ġto Ġfocus Ġon Ġthe ĠL RC N 1 u Ġarchitecture Ġin Ġsubsequent Ġevaluations , Ġwhere Ġwe Ġreport Ġdetailed Ġresults Ġwith Ġrespect Ġto Ġthe ĠGT D Ġframework Ġon Ġa Ġvariety Ġof Ġdatasets . </s> Cor relation Ġbetween Ġthe ĠB LE U / SP ICE Ġscores Ġand Ġthe Ġground Ġtruth . ĠFrom Ġthe Ġlearning Ġcurves Ġshown Ġin ĠFigure ĠFIG REF 9 , Ġwe Ġfind Ġlow Ġor Ġno Ġcorrelation Ġbetween Ġthe ĠB LE U / SP ICE Ġscores Ġand Ġcaption Ġtruth fulness . </s> On ĠEx ist ential - One Shape , Ġthe ĠB LE U Ġcurve Ġfollows Ġthe Ġtrend Ġof Ġthe Ġtruth fulness Ġcurve Ġin Ġgeneral , Ġindicating Ġthat ĠB LE U Ġis Ġable Ġto Ġcapture Ġcaption Ġtruth fulness Ġwell Ġin Ġthis Ġsimple Ġscenario . ĠHowever , Ġwhile ĠB LE U Ġreports Ġequivalent Ġmodel Ġperformance Ġon ĠEx ist ential - Multi Sh apes Ġand ĠSp atial - Multi Sh apes , Ġthe Ġtruth fulness Ġmetric Ġdemonstrates Ġvery Ġdifferent Ġresults . ĠThe ĠB LE U Ġscore Ġfor Ġgenerated ĠEx ist ential - Multi Sh apes Ġcapt ions Ġincreases Ġrapidly Ġat Ġthe Ġbeginning Ġof Ġtraining Ġand Ġthen Ġplate aus Ġdespite Ġthe Ġcontinuous Ġincrease Ġin Ġtruth fulness Ġratio . ĠCapt ions Ġgenerated Ġon ĠSp atial - Multi Sh apes Ġattain Ġa Ġrelatively Ġhigh ĠB LE U Ġscore Ġfrom Ġan Ġearly Ġstage Ġof Ġtraining , Ġbut Ġexhibit Ġlow Ġagreement Ġ($ < $ 0 . 6 Ġtruth fulness Ġratio ) Ġwith Ġground - truth Ġvisual Ġscenes . ĠIn Ġthe Ġcase Ġof ĠSp atial - Multi Sh apes , Ġspatial Ġdescript ors Ġfor Ġtwo Ġobjects Ġare Ġchosen Ġfrom Ġa Ġfixed Ġset Ġ( âĢ ľ above âĢ Ŀ , ĠâĢ ľ below âĢ Ŀ , ĠâĢ ľ to Ġthe Ġleft Ġof âĢ Ŀ Ġand ĠâĢ ľ to Ġthe Ġright Ġof âĢ Ŀ ). ĠIt Ġis Ġvery Ġlikely Ġfor Ġa Ġgenerated Ġspatial Ġdescriptor Ġto Ġmatch Ġone Ġof Ġthe Ġdescript ors Ġmentioned Ġin Ġreference Ġcapt ions . ĠIn Ġthis Ġparticular Ġcase , Ġthe Ġmodel Ġis Ġapt Ġto Ġinfer Ġa Ġcaption Ġwhich Ġhas Ġhigh Ġn - gram Ġoverl aps Ġwith Ġreference Ġcapt ions , Ġresulting Ġin Ġa Ġrelatively Ġhigh ĠB LE U Ġscore . ĠThus Ġan Ġincreased ĠB LE U Ġscore Ġdoes Ġnot Ġnecessarily Ġindicate Ġimproved Ġperformance . </s> While Ġthe Ġtruth fulness Ġand ĠB LE U Ġscores Ġin ĠFigure ĠFIG REF 9 Ġboth Ġincrease Ġrapidly Ġearly Ġon Ġand Ġthen Ġstay Ġstable Ġat Ġa Ġhigh Ġrate Ġafter Ġtraining Ġfor Ġ20 k Ġiterations , Ġthe ĠSP ICE Ġcurve Ġinstead Ġshows Ġa Ġdownward Ġtrend Ġin Ġthe Ġlater Ġstage Ġof Ġtraining . ĠWe Ġexamined Ġthe Ġoutput ĠSP ICE Ġscore Ġfor Ġeach Ġtest Ġinstance . ĠSP ICE Ġreports Ġa Ġprecision Ġscore Ġof Ġ1 . 0 Ġfor Ġmost Ġtest Ġinstances Ġafter Ġ20 k Ġiterations , Ġwhich Ġis Ġconsistent Ġwith Ġthe Ġtruth fulness Ġand ĠB LE U Ġscores . ĠHowever , ĠSP ICE Ġforms Ġthe Ġreference Ġscene Ġgraph Ġas Ġthe Ġunion Ġof Ġthe Ġscene Ġgraphs Ġextracted Ġfrom Ġindividual Ġreference Ġcapt ions , Ġthus Ġintroducing Ġredund ancies . ĠSP ICE Ġuses Ġthe ĠF 1 Ġscore Ġof Ġscene Ġgraph Ġmatching Ġbetween Ġthe Ġcandidate Ġand Ġreference Ġand Ġhence Ġis Ġlowered Ġby Ġimperfect Ġrecall . </s> Comp aring ĠSP ICE Ġcurves Ġfor Ġthree Ġdatasets Ġshown Ġin ĠFigure ĠFIG REF 9 - FIG REF 9 , Ġthey Ġsuggest Ġan Ġincrease Ġin Ġtask Ġcomplexity , Ġbut Ġthey Ġdo Ġnot Ġreflect Ġthe Ġsuccess ively Ġclosing Ġgap Ġof Ġcaption Ġtruth fulness Ġscores Ġbetween Ġtwo ĠEx ist ential Ġdatasets , Ġor Ġthe Ġsubstantial Ġdifference Ġin Ġcaption Ġtruth fulness Ġbetween Ġcapt ions Ġon ĠEx ist ential - Multi Sh apes Ġand ĠSp atial - Multi Sh apes . </s> In Ġthe Ġremainder Ġof Ġthe Ġpaper Ġwe Ġdiscuss Ġin Ġdetail Ġthe Ġdiagnostic Ġresults Ġof Ġthe ĠL RC N 1 u Ġmodel Ġdemonstrated Ġby Ġthe ĠGT D Ġevaluation Ġframework . </s> Perfect Ġgram matical ity Ġfor Ġall Ġcaption Ġtypes . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 15 , Ġgenerated Ġcapt ions Ġfor Ġall Ġtypes Ġof ĠShape World ICE Ġdatasets Ġattain Ġquasi - perfect Ġgram matical ity Ġscores Ġin Ġfewer Ġthan Ġ5 , 000 Ġiterations , Ġsuggesting Ġthat Ġthe Ġmodel Ġquickly Ġlearns Ġto Ġgenerate Ġgram matically Ġwell - formed Ġsentences . </s> Failure Ġto Ġlearn Ġcomplex Ġspatial Ġrelationships . ĠWhile ĠCNN s Ġcan Ġproduce Ġrich Ġvisual Ġrepresentations Ġthat Ġcan Ġbe Ġused Ġfor Ġa Ġvariety Ġof Ġvision Ġtasks ĠB IB REF 27 , Ġit Ġremains Ġan Ġopen Ġquestion Ġwhether Ġthese Ġcondensed Ġvisual Ġrepresentations Ġare Ġrich Ġenough Ġfor Ġmultim od al Ġtasks Ġthat Ġrequire Ġhigher - level Ġabilities Ġof Ġscene Ġunderstanding Ġand Ġvisual Ġreasoning . ĠFrom ĠFigure ĠFIG REF 16 , Ġwe Ġcan Ġsee Ġthat Ġwhile Ġthe Ġmodel Ġperforms Ġrather Ġwell Ġon ĠEx ist ential Ġdatasets , Ġit Ġexhibits Ġa Ġworse Ġperformance Ġon ĠSp atial Ġdata . ĠThe Ġcaption Ġagreement Ġratio Ġin Ġthe Ġsimple ĠSp atial - Two Sh apes Ġscenario Ġis Ġrelatively Ġhigh , Ġbut Ġdrops Ġsignificantly Ġon ĠSp atial - Multi Sh apes , Ġdemonstrating Ġthe Ġdeficiencies Ġof Ġthe Ġmodel Ġin Ġlearning Ġspatial Ġrelationships Ġfrom Ġcomplex Ġvisual Ġscenes . </s> The Ġcounting Ġtask Ġis Ġnon - t riv ial . ĠCount ing Ġhas Ġlong Ġbeen Ġconsidered Ġto Ġbe Ġa Ġchallenging Ġtask Ġin Ġmultim od al Ġreasoning ĠB IB REF 28 , ĠB IB REF 29 . ĠTo Ġexplore Ġhow Ġwell Ġthe ĠL RC N 1 u Ġmodel Ġcop es Ġwith Ġcounting Ġtasks , Ġwe Ġgenerated Ġtwo ĠQuant ification Ġdatasets . ĠThe ĠQuant - Count Ġcapt ions Ġdescribe Ġthe Ġnumber Ġof Ġobjects Ġwith Ġcertain Ġattributes Ġthat Ġappear Ġin Ġan Ġimage Ġ( e . g . ĠâĢ ľ Exactly Ġfour Ġshapes Ġare Ġcrosses âĢ Ŀ ), Ġwhile Ġthe ĠQuant - Rat io Ġcapt ions Ġdescribe Ġthe Ġratio Ġof Ġcertain Ġobjects Ġ( e . g . ĠâĢ ľ A Ġthird Ġof Ġthe Ġshapes Ġare Ġblue Ġsquares âĢ Ŀ ). </s> From ĠFigure ĠFIG REF 16 , Ġwe Ġnotice Ġthat Ġthe ĠL RC N 1 u Ġmodel Ġperforms Ġpoorly Ġon Ġthese Ġdatasets Ġin Ġterms Ġof Ġtruth fulness , Ġreflected Ġin Ġthe Ġ0 . 50 Ġand Ġ0 . 46 Ġscores Ġachieved Ġby Ġthe Ġmodel Ġon Ġthe ĠQuant - Count Ġand ĠQuant - Rat io Ġtasks Ġrespectively . ĠThe Ġlearning Ġcurve Ġfor ĠQuant - Rat io Ġexhibits Ġa Ġmore Ġgradual Ġrise Ġas Ġthe Ġtraining Ġprogresses , Ġsuggesting Ġa Ġgreater Ġcomplexity Ġfor Ġthe Ġratio - based Ġtask . </s> Ca ption Ġdiversity Ġbenefits Ġfrom Ġvaried Ġlanguage Ġconstruct ions Ġin Ġthe Ġtraining Ġdata . ĠThe Ġdiversity Ġratios Ġof Ġgenerated Ġcapt ions Ġfor Ġdifferent ĠShape World ICE Ġdatasets Ġare Ġillustrated Ġin ĠFigure ĠFIG REF 17 . ĠWe Ġcan Ġsee Ġthat Ġthe Ġdiversity Ġof Ġinferred Ġcapt ions Ġis Ġlargely Ġsensitive Ġto Ġthe Ġcaption Ġvariability Ġin Ġthe Ġdataset Ġitself . ĠFor Ġsimple Ġdatasets Ġ( such Ġas ĠEx ist ential - One Shape ) Ġwhere Ġlanguage Ġconstruct ions Ġin Ġthe Ġtraining Ġset Ġare Ġless Ġdiverse , Ġthe Ġoutput Ġcapt ions Ġtend Ġto Ġhave Ġuniform Ġsentence Ġstructures . ĠThe Ġhigh Ġdiversity Ġratios Ġof Ġgenerated ĠSp atial Ġand ĠQuant ification Ġcapt ions Ġsuggest Ġthat Ġcaption Ġdiversity Ġbenefits Ġfrom Ġheter ogeneous Ġlanguage Ġconstruct ions Ġin Ġcomplex Ġdatasets . </s> Disc ussions Ġand ĠCon clusions </s> E val uation Ġmetrics Ġare Ġrequired Ġas Ġa Ġproxy Ġfor Ġperformance Ġin Ġreal Ġapplications . ĠAs Ġsuch , Ġthey Ġshould , Ġas Ġfar Ġas Ġpossible , Ġallow Ġmeasurement Ġof Ġfundamental Ġaspects Ġof Ġthe Ġperformance Ġof Ġmodels Ġon Ġtasks . ĠIn Ġthis Ġwork , Ġwe Ġpropose Ġthe ĠGT D Ġevaluation Ġframework Ġas Ġa Ġsupplement Ġto Ġstandard Ġimage Ġcaption ing Ġevaluation Ġwhich Ġexplicitly Ġfocuses Ġon Ġgram matical ity , Ġtruth fulness Ġand Ġdiversity . ĠWe Ġdeveloped Ġthe ĠShape World ICE Ġevaluation Ġsuite Ġto Ġallow Ġin - depth Ġand Ġfine - gr ained Ġinspection Ġof Ġmodel Ġbehaviors . ĠWe Ġhave Ġempir ically Ġverified Ġthat ĠGT D Ġcaptures Ġdifferent Ġaspects Ġof Ġperformance Ġto Ġexisting Ġmetrics Ġby Ġevaluating Ġimage Ġcaption ing Ġmodels Ġon Ġthe ĠShape World ICE Ġsuite . ĠWe Ġhope Ġthat Ġthis Ġframework Ġwill Ġshed Ġlight Ġon Ġimportant Ġaspects Ġof Ġmodel Ġbehaviour Ġand Ġthat Ġthis Ġwill Ġhelp Ġguide Ġfuture Ġresearch Ġefforts . </s> While Ġperforming Ġthe Ġevaluation Ġexperiments Ġon Ġthe ĠL RC N 1 u Ġmodel , Ġwe Ġnoticed Ġthat Ġcaption Ġagreement Ġdoes Ġnot Ġalways Ġimprove Ġas Ġthe Ġtraining Ġloss Ġdecreases . ĠIdeally , Ġthe Ġtraining Ġobjective Ġshould Ġbe Ġin Ġaccordance Ġwith Ġhow Ġa Ġmodel Ġis Ġeventually Ġevaluated . ĠIn Ġfuture Ġwork , Ġwe Ġplan Ġto Ġinvestigate Ġthe Ġfeasibility Ġof Ġdeliberately Ġencoding Ġthe ĠGT D Ġsignal Ġin Ġthe Ġtraining Ġprocess , Ġfor Ġinstance , Ġby Ġimplementing Ġa ĠGT D - aware Ġloss . ĠWe Ġalso Ġplan Ġto Ġextend Ġthe Ġexisting ĠShape World ICE Ġbenchmark Ġto Ġinclude Ġmore Ġlinguistic Ġconstruct ions Ġ( such Ġas Ġrelative Ġclauses , Ġcompound Ġsentences Ġand Ġcore ference ). ĠBy Ġdoing Ġso , Ġwe Ġhope Ġto Ġreveal Ġhow Ġwell Ġexisting Ġimage Ġcaption ing Ġmodels Ġcope Ġwith Ġcomplex Ġgeneration Ġtasks . </s> Acknowled gments </s> We Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġconstructive Ġfeedback . ĠH X Ġis Ġgrateful Ġfor Ġbeing Ġsupported Ġby Ġthe ĠC SC ĠCambridge ĠScholarship . ĠTS Ġis Ġsupported Ġin Ġpart Ġby Ġthe ĠEPS RC ĠCentre Ġfor ĠDoctor al ĠTraining Ġin ĠData ĠScience , Ġfunded Ġby Ġthe ĠEPS RC Ġ( gr ant ĠEP / L 01 64 27 / 1 ) Ġand Ġthe ĠUniversity Ġof ĠEdinburgh . ĠAK Ġis Ġgrateful Ġfor Ġbeing Ġsupported Ġby Ġa ĠQualcomm ĠResearch ĠStudents hip Ġand Ġan ĠEPS RC ĠDoctor al ĠTraining ĠStudents hip .
Introduction Automatically generating text to describe the content of images, also known as image captioning, is a multimodal task of considerable interest in both the computer vision and the NLP communities. Image captioning can be framed as a translation task from an image to a descriptive natural language statement. Many existing captioning models BIBREF0, BIBREF1, BIBREF2, BIBREF3 follow the typical encoder-decoder framework where a convolutional network is used to condense images into visual feature representations, combined with a recurrent network for language generation. While these models demonstrate promising results, quantifying image captioning performance remains a challenging problem, in a similar way to other generative tasks BIBREF4, BIBREF5. Evaluating candidate captions for human preference is slow and laborious. To alleviate this problem, many automatic evaluation metrics have been proposed, such as BLEU BIBREF6, METEOR BIBREF7, ROUGE BIBREF8 and CIDEr BIBREF9. These n-gram-based metrics evaluate captioning performance based on surface similarity between a candidate caption and reference statements. A more recent evaluation metric for image captioning is SPICE BIBREF10, which takes into account semantic propositional content of generated captions by scoring a caption based upon a graph-based semantic representation transformed from reference captions. The rationale behind these evaluation metrics is that human reference captions serve as an approximate target and comparing model outputs to this target is a proxy for how well a system performs. Thus, a candidate caption is not directly evaluated with respect to image content, but compared to a set of human statements about that image. However, in image captioning, visual scenes with multiple objects and relations correspond to a diversity of valid descriptions. Consider the example image and captions from the ShapeWorld framework BIBREF11 shown in Figure FIGREF1. The first three captions are true statements about the image and express relevant ideas, but describe different objects, attributes and spatial relationships, while the fourth caption is wrong despite referring to the same objects as in the third caption. This casts doubt on the sufficiency of using a set of reference captions to approximate the content of an image. We argue that, while existing metrics have undeniably been useful for real-world captioning evaluation, their focus on approximate surface comparison limits deeper insights into the learning process and eventual behavior of captioning models. To address this problem, we propose a set of principled evaluation criteria which evaluate image captioning models for grammaticality, truthfulness and diversity (GTD). These criteria correspond to necessary requirements for image captioning systems: (a) that the output is grammatical, (b) that the output statement is true with respect to the image, and (c) that outputs are diverse and mirror the variability of training captions. Practical evaluation of GTD is currently only possible on synthetic data. We construct a range of datasets designed for image captioning evaluation. We call this diagnostic evaluation benchmark ShapeWorldICE (ShapeWorld for Image Captioning Evaluation). We illustrate the evaluation of specific image captioning models on ShapeWorldICE. We empirically demonstrate that the existing metrics BLEU and SPICE do not capture true caption-image agreement in all scenarios, while the GTD framework allows a fine-grained investigation of how well existing models cope with varied visual situations and linguistic constructions. We believe that as a supplementary evaluation method to real-world metrics, the GTD framework provides evaluation insights that are sufficiently interesting to motivate future work. Related work ::: Existing evaluation of image captioning As a natural language generation task, image captioning frequently uses evaluation metrics such as BLEU BIBREF6, METEOR BIBREF7, ROUGE BIBREF8 and CIDEr BIBREF9. These metrics use n-gram similarity between the candidate caption and reference captions to approximate the correlation between a candidate caption and the associated ground truth. SPICE BIBREF10 is a more recent metric specifically designed for image captioning. For SPICE, both the candidate caption and reference captions are parsed to scene graphs, and the agreement between tuples extracted from these scene graphs is examined. SPICE more closely relates to our truthfulness evaluation than the other metrics, but it still uses overlap comparison to reference captions as a proxy to ground truth. In contrast, our truthfulness metric directly evaluates a candidate caption against a model of the actual visual content. Many researchers have pointed out problems with existing reference-based metrics including low correlations with human judgment BIBREF12, BIBREF10, BIBREF13 and strong baselines using nearest-neighbor methods BIBREF14 or relying solely on object detection BIBREF15. Fundamental concerns have been raised with respect to BLEU, including variability in parameterization and precise score calculation leading to significantly different results BIBREF16. Its validity as a metric for tasks other than machine translation has been questioned BIBREF17, particularly for tasks for which the output content is not narrowly constrained, like dialogue BIBREF18. Some recent work focuses on increasing the diversity of generated captions, for which various measures are proposed. Devlin et al. BIBREF19 explored the concept of caption diversity by evaluating performance on compositionally novel images. van Miltenburg et al BIBREF20 framed image captioning as a word recall task and proposed several metrics, predominantly focusing on diversity at the word level. However, this direction is still relatively new and lacks standardized benchmarks and metrics. Related work ::: Synthetic datasets Recently, many synthetic datasets have been proposed as diagnostic tools for deep learning models, such as CLEVR BIBREF21 for visual question answering (VQA), the bAbI tasks BIBREF22 for text understanding and reasoning, and ShapeWorld BIBREF11 for visually grounded language understanding. The primary motivation is to reduce complexity which is considered irrelevant to the evaluation focus, to enable better control over the data, and to provide more detailed insights into strengths and limitations of existing models. In this work, we develop the evaluation datasets within the ShapeWorld framework. ShapeWorld is a controlled data generation framework consisting of abstract colored shapes (see Figure FIGREF1 for an example). We use ShapeWorld to generate training and evaluation data for two major reasons. ShapeWorld supports customized data generation according to user specification, which enables a variety of model inspections in terms of language construction, visual complexity and reasoning ability. Another benefit is that each training and test instance generated in ShapeWorld is returned as a triplet of $<$image, caption, world model$>$. The world model stores information about the underlying microworld used to generate an image and a descriptive caption, internally represented as a list of entities with their attributes, such as shape, color, position. During data generation, ShapeWorld randomly samples a world model from a set of available entities and attributes. The generated world model is then used to realize a corresponding instance consisting of image and caption. The world model gives the actual semantic information contained in an image, which allows evaluation of caption truthfulness. GTD Evaluation Framework In the following we introduce GTD in more detail, consider it as an evaluation protocol covering necessary aspects of the multifaceted captioning task, rather than a specific metric. GTD Evaluation Framework ::: Grammaticality An essential criterion for an image captioning model is that the captions generated are grammatically well-formed. Fully accurate assessment of grammaticality in a general context is itself a difficult task, but becomes more feasible in a very constrained context like our diagnostic language data. We take parseability with the English Resource Grammar BIBREF23 as a surrogate for grammaticality, meaning that a sentence is considered grammatically well-formed if we obtain a parse using the ERG. The ERG is a broad-coverage grammar based on the head-driven phrase structure grammar (HPSG) framework. It is linguistically precise: sentences only parse if they are valid according to its hand-built rules. It is designed to be general-purpose: verified coverage is around 80% for Wikipedia, and over 90% for corpora with shorter sentences and more limited vocabulary (for details see BIBREF24 flickinger2011accuracy). Since the ShapeWorld training data – the only language source for models to learn from – is generated using the same grammar, the ERG has $\sim $100% coverage of grammaticality in the model output space. GTD Evaluation Framework ::: Truthfulness The second aspect we investigate is truthfulness, that is, whether a candidate caption is compatible with the content of the image it is supposed to describe. We evaluate caption truthfulness on the basis of a linguistically-motivated approach using formal semantics. We convert the output of the ERG parse for a grammatical caption to a Dependency Minimal Recursion Semantics (DMRS) graph using the pydmrs tool BIBREF25. Each converted DMRS is a logical semantic graph representation corresponding to the caption. We construct a logical proposition from the DMRS graph, and evaluate it against the actual world model of the corresponding image. A caption can be said to agree with an image only if the proposition evaluates as true on the basis of the world model. By examining the logical agreement between a caption representation and a world model, we can check whether the semantics of this caption agrees with the visual content which the world model represents. Thus we do not rely on a set of captions as a surrogate for the content of an image, but instead leverage the fact that we have the ground truth, thus enabling the evaluation of true image-caption agreement. GTD Evaluation Framework ::: Diversity While grammaticality and truthfulness are essential requirements for image captions, these criteria alone can easily be “gamed” by specializing on a small set of generic statements which are true most of the time. In the context of abstract shapes, such captions include examples like “There is a shape” or “At least zero shapes are blue” (which is technically true even if there is no blue shape). This motivates the third fundamental requirement of captioning output to be diverse. As ShapeWorldICE exploits a limited size of open-class words, we emphasize the diversity in ShapeWorldICE at the sentence level rather than the word level. Since the ground-truth reference captions in ShapeWorld are randomly sampled, we take the sampled captions accompanying the test images as a proxy for optimal caption diversity, and compare it with the empirical output diversity of the evaluated model on these test images. Practically, we look at language constructions used and compute the corresponding diversity score as the ratio of observed number versus optimal number: Language constructions here correspond to reduced caption representations which only record whether an object is described by shape (e.g., “square”), color (e.g., “red shape”) or color-shape combination (e.g., “red square”). So the statement “A square is red” and “A circle is blue” are considered the same, while “A shape is red” is different. Experimental Setup ::: Datasets We develop a variety of ShapeWorldICE datasets, with a similar idea to the “skill tasks” in the bAbI framework BIBREF22. Table TABREF4 gives an overview for different ShapeWorldICE datasets we use in this paper. We consider three different types of captioning tasks, each of which focuses on a distinct aspect of reasoning abilities. Existential descriptions examine whether a certain object is present in an image. Spatial descriptions identify spatial relationships among visual objects. Quantification descriptions involve count-based and ratio-based statements, with an explicit focus on inspecting models for their counting ability. We develop two variants for each type of dataset to enable different levels of visual complexity or specific aspects of the same reasoning type. All the training and test captions sampled in this work are in English. Each dataset variant consists of around 200k training instances and 4,096 validation instances, plus 4,096 test instances. Each training instance consists of an image and a reference caption. At test time, only the test images are available to the evaluated models. Underlying world models are kept from the models and are used for later GTD evaluation. For each test instance, we sample ten reference captions of the same distribution as the training captions to enable the comparison of our proposed metrics to BLEU and SPICE. We fine-tune our model hyperparameters based on the performance on the validation set. All reported results are measured on the test split with the parameters yielding the best validation performance. Experimental Setup ::: Models We experiment with two image captioning models: the Show&Tell model BIBREF0 and the LRCN1u model BIBREF1. Both models follow the basic encoder-decoder architecture design that uses a CNN encoder to condense the visual information into an image embedding, which in turn conditions an LSTM decoder to generate a natural language caption. The main difference between the two models is the way they condition the decoder. The Show&Tell model feeds the image embedding as the “predecessor word embedding” to the first produced word, while the LRCN1u model concatenates the image features with the embedded previous word as the input to the sequence model at each time step. We follow the common practice in image captioning to use a CNN component pretrained on object detection and fine-tune its parameters on the image captioning task. The encoder and decoder components are jointly optimized with respect to the standard cross-entropy sequence loss on the respective ShapeWorldICE dataset. For all our experiments, we train models end-to-end for a fixed number of 100k iterations with a batch size of 64. We use Adam optimization BIBREF26 with a learning rate of 0.001. Word embeddings are randomly initialized and jointly trained during the training. Results We train and evaluate the Show&Tell and LRCN1u models on the ShapeWorldICE datasets. Here we discuss in detail the diagnostic results of these experiments. During training, we periodically record model output on the test images, to be able to analyze the development of our evaluation metrics throughout the process. We also compute BLEU-4 scores and SPICE scores of generated captions for comparison, using 10 reference captions per test image. LRCN1u exhibits clearly superior performance in terms of truthfulness. We start off by comparing performance of the Show&Tell model and the LRCN1u model, see Figure FIGREF8. While both models learn to produce grammatical sentences early on, it can be seen that LRCN1u is clearly superior in terms of truthfulness, achieving 100% halfway through training, whereas Show&Tell only slowly reaches around 90% by the end of 100k iterations. This indicates that incorporating visual features at every generation step is beneficial for producing true captions. The diversity ratios of captions generated by two models both increase substantially as the training progresses, with LRCN1u exhibiting a slightly greater caption diversity at the end of training. We observed similar results on other ShapeWorldICE datasets that we experimented with, validating the superiority of LRCN1u over Show&Tell on ShapeWorldICE. Consequently, we decided to focus on the LRCN1u architecture in subsequent evaluations, where we report detailed results with respect to the GTD framework on a variety of datasets. Correlation between the BLEU/SPICE scores and the ground truth. From the learning curves shown in Figure FIGREF9, we find low or no correlation between the BLEU/SPICE scores and caption truthfulness. On Existential-OneShape, the BLEU curve follows the trend of the truthfulness curve in general, indicating that BLEU is able to capture caption truthfulness well in this simple scenario. However, while BLEU reports equivalent model performance on Existential-MultiShapes and Spatial-MultiShapes, the truthfulness metric demonstrates very different results. The BLEU score for generated Existential-MultiShapes captions increases rapidly at the beginning of training and then plateaus despite the continuous increase in truthfulness ratio. Captions generated on Spatial-MultiShapes attain a relatively high BLEU score from an early stage of training, but exhibit low agreement ($<$0.6 truthfulness ratio) with ground-truth visual scenes. In the case of Spatial-MultiShapes, spatial descriptors for two objects are chosen from a fixed set (“above”, “below”, “to the left of” and “to the right of”). It is very likely for a generated spatial descriptor to match one of the descriptors mentioned in reference captions. In this particular case, the model is apt to infer a caption which has high n-gram overlaps with reference captions, resulting in a relatively high BLEU score. Thus an increased BLEU score does not necessarily indicate improved performance. While the truthfulness and BLEU scores in Figure FIGREF9 both increase rapidly early on and then stay stable at a high rate after training for 20k iterations, the SPICE curve instead shows a downward trend in the later stage of training. We examined the output SPICE score for each test instance. SPICE reports a precision score of 1.0 for most test instances after 20k iterations, which is consistent with the truthfulness and BLEU scores. However, SPICE forms the reference scene graph as the union of the scene graphs extracted from individual reference captions, thus introducing redundancies. SPICE uses the F1 score of scene graph matching between the candidate and reference and hence is lowered by imperfect recall. Comparing SPICE curves for three datasets shown in Figure FIGREF9-FIGREF9, they suggest an increase in task complexity, but they do not reflect the successively closing gap of caption truthfulness scores between two Existential datasets, or the substantial difference in caption truthfulness between captions on Existential-MultiShapes and Spatial-MultiShapes. In the remainder of the paper we discuss in detail the diagnostic results of the LRCN1u model demonstrated by the GTD evaluation framework. Perfect grammaticality for all caption types. As shown in Figure FIGREF15, generated captions for all types of ShapeWorldICE datasets attain quasi-perfect grammaticality scores in fewer than 5,000 iterations, suggesting that the model quickly learns to generate grammatically well-formed sentences. Failure to learn complex spatial relationships. While CNNs can produce rich visual representations that can be used for a variety of vision tasks BIBREF27, it remains an open question whether these condensed visual representations are rich enough for multimodal tasks that require higher-level abilities of scene understanding and visual reasoning. From Figure FIGREF16, we can see that while the model performs rather well on Existential datasets, it exhibits a worse performance on Spatial data. The caption agreement ratio in the simple Spatial-TwoShapes scenario is relatively high, but drops significantly on Spatial-MultiShapes, demonstrating the deficiencies of the model in learning spatial relationships from complex visual scenes. The counting task is non-trivial. Counting has long been considered to be a challenging task in multimodal reasoning BIBREF28, BIBREF29. To explore how well the LRCN1u model copes with counting tasks, we generated two Quantification datasets. The Quant-Count captions describe the number of objects with certain attributes that appear in an image (e.g. “Exactly four shapes are crosses”), while the Quant-Ratio captions describe the ratio of certain objects (e.g. “A third of the shapes are blue squares”). From Figure FIGREF16, we notice that the LRCN1u model performs poorly on these datasets in terms of truthfulness, reflected in the 0.50 and 0.46 scores achieved by the model on the Quant-Count and Quant-Ratio tasks respectively. The learning curve for Quant-Ratio exhibits a more gradual rise as the training progresses, suggesting a greater complexity for the ratio-based task. Caption diversity benefits from varied language constructions in the training data. The diversity ratios of generated captions for different ShapeWorldICE datasets are illustrated in Figure FIGREF17. We can see that the diversity of inferred captions is largely sensitive to the caption variability in the dataset itself. For simple datasets (such as Existential-OneShape) where language constructions in the training set are less diverse, the output captions tend to have uniform sentence structures. The high diversity ratios of generated Spatial and Quantification captions suggest that caption diversity benefits from heterogeneous language constructions in complex datasets. Discussions and Conclusions Evaluation metrics are required as a proxy for performance in real applications. As such, they should, as far as possible, allow measurement of fundamental aspects of the performance of models on tasks. In this work, we propose the GTD evaluation framework as a supplement to standard image captioning evaluation which explicitly focuses on grammaticality, truthfulness and diversity. We developed the ShapeWorldICE evaluation suite to allow in-depth and fine-grained inspection of model behaviors. We have empirically verified that GTD captures different aspects of performance to existing metrics by evaluating image captioning models on the ShapeWorldICE suite. We hope that this framework will shed light on important aspects of model behaviour and that this will help guide future research efforts. While performing the evaluation experiments on the LRCN1u model, we noticed that caption agreement does not always improve as the training loss decreases. Ideally, the training objective should be in accordance with how a model is eventually evaluated. In future work, we plan to investigate the feasibility of deliberately encoding the GTD signal in the training process, for instance, by implementing a GTD-aware loss. We also plan to extend the existing ShapeWorldICE benchmark to include more linguistic constructions (such as relative clauses, compound sentences and coreference). By doing so, we hope to reveal how well existing image captioning models cope with complex generation tasks. Acknowledgments We thank the anonymous reviewers for their constructive feedback. HX is grateful for being supported by the CSC Cambridge Scholarship. TS is supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded by the EPSRC (grant EP/L016427/1) and the University of Edinburgh. AK is grateful for being supported by a Qualcomm Research Studentship and an EPSRC Doctoral Training Studentship.
Are the images from a specific domain?
50e80cfa84200717921840fddcf3b051a9216ad8
Are Ġthe Ġimages Ġfrom Ġa Ġspecific Ġdomain ?
Yes
<s> Are Ġthe Ġimages Ġfrom Ġa Ġspecific Ġdomain ? </s> Introduction </s> Autom atically Ġgenerating Ġtext Ġto Ġdescribe Ġthe Ġcontent Ġof Ġimages , Ġalso Ġknown Ġas Ġimage Ġcaption ing , Ġis Ġa Ġmultim od al Ġtask Ġof Ġconsiderable Ġinterest Ġin Ġboth Ġthe Ġcomputer Ġvision Ġand Ġthe ĠN LP Ġcommunities . ĠImage Ġcaption ing Ġcan Ġbe Ġframed Ġas Ġa Ġtranslation Ġtask Ġfrom Ġan Ġimage Ġto Ġa Ġdescriptive Ġnatural Ġlanguage Ġstatement . ĠMany Ġexisting Ġcaption ing Ġmodels ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 Ġfollow Ġthe Ġtypical Ġenc oder - dec oder Ġframework Ġwhere Ġa Ġconv olution al Ġnetwork Ġis Ġused Ġto Ġcond ense Ġimages Ġinto Ġvisual Ġfeature Ġrepresentations , Ġcombined Ġwith Ġa Ġrecurrent Ġnetwork Ġfor Ġlanguage Ġgeneration . ĠWhile Ġthese Ġmodels Ġdemonstrate Ġpromising Ġresults , Ġquant ifying Ġimage Ġcaption ing Ġperformance Ġremains Ġa Ġchallenging Ġproblem , Ġin Ġa Ġsimilar Ġway Ġto Ġother Ġgener ative Ġtasks ĠB IB REF 4 , ĠB IB REF 5 . </s> E val uating Ġcandidate Ġcapt ions Ġfor Ġhuman Ġpreference Ġis Ġslow Ġand Ġlabor ious . ĠTo Ġalleviate Ġthis Ġproblem , Ġmany Ġautomatic Ġevaluation Ġmetrics Ġhave Ġbeen Ġproposed , Ġsuch Ġas ĠB LE U ĠB IB REF 6 , ĠMET E OR ĠB IB REF 7 , ĠR OU GE ĠB IB REF 8 Ġand ĠC ID Er ĠB IB REF 9 . ĠThese Ġn - gram - based Ġmetrics Ġevaluate Ġcaption ing Ġperformance Ġbased Ġon Ġsurface Ġsimilarity Ġbetween Ġa Ġcandidate Ġcaption Ġand Ġreference Ġstatements . ĠA Ġmore Ġrecent Ġevaluation Ġmetric Ġfor Ġimage Ġcaption ing Ġis ĠSP ICE ĠB IB REF 10 , Ġwhich Ġtakes Ġinto Ġaccount Ġsemantic Ġpropos itional Ġcontent Ġof Ġgenerated Ġcapt ions Ġby Ġscoring Ġa Ġcaption Ġbased Ġupon Ġa Ġgraph - based Ġsemantic Ġrepresentation Ġtransformed Ġfrom Ġreference Ġcapt ions . </s> The Ġrationale Ġbehind Ġthese Ġevaluation Ġmetrics Ġis Ġthat Ġhuman Ġreference Ġcapt ions Ġserve Ġas Ġan Ġapproximate Ġtarget Ġand Ġcomparing Ġmodel Ġoutputs Ġto Ġthis Ġtarget Ġis Ġa Ġproxy Ġfor Ġhow Ġwell Ġa Ġsystem Ġperforms . ĠThus , Ġa Ġcandidate Ġcaption Ġis Ġnot Ġdirectly Ġevaluated Ġwith Ġrespect Ġto Ġimage Ġcontent , Ġbut Ġcompared Ġto Ġa Ġset Ġof Ġhuman Ġstatements Ġabout Ġthat Ġimage . </s> However , Ġin Ġimage Ġcaption ing , Ġvisual Ġscenes Ġwith Ġmultiple Ġobjects Ġand Ġrelations Ġcorrespond Ġto Ġa Ġdiversity Ġof Ġvalid Ġdescriptions . ĠConsider Ġthe Ġexample Ġimage Ġand Ġcapt ions Ġfrom Ġthe ĠShape World Ġframework ĠB IB REF 11 Ġshown Ġin ĠFigure ĠFIG REF 1 . ĠThe Ġfirst Ġthree Ġcapt ions Ġare Ġtrue Ġstatements Ġabout Ġthe Ġimage Ġand Ġexpress Ġrelevant Ġideas , Ġbut Ġdescribe Ġdifferent Ġobjects , Ġattributes Ġand Ġspatial Ġrelationships , Ġwhile Ġthe Ġfourth Ġcaption Ġis Ġwrong Ġdespite Ġreferring Ġto Ġthe Ġsame Ġobjects Ġas Ġin Ġthe Ġthird Ġcaption . ĠThis Ġcasts Ġdoubt Ġon Ġthe Ġsu fficiency Ġof Ġusing Ġa Ġset Ġof Ġreference Ġcapt ions Ġto Ġapproximate Ġthe Ġcontent Ġof Ġan Ġimage . ĠWe Ġargue Ġthat , Ġwhile Ġexisting Ġmetrics Ġhave Ġundeniably Ġbeen Ġuseful Ġfor Ġreal - world Ġcaption ing Ġevaluation , Ġtheir Ġfocus Ġon Ġapproximate Ġsurface Ġcomparison Ġlimits Ġdeeper Ġinsights Ġinto Ġthe Ġlearning Ġprocess Ġand Ġeventual Ġbehavior Ġof Ġcaption ing Ġmodels . </s> To Ġaddress Ġthis Ġproblem , Ġwe Ġpropose Ġa Ġset Ġof Ġprincipled Ġevaluation Ġcriteria Ġwhich Ġevaluate Ġimage Ġcaption ing Ġmodels Ġfor Ġgram matical ity , Ġtruth fulness Ġand Ġdiversity Ġ( GT D ). ĠThese Ġcriteria Ġcorrespond Ġto Ġnecessary Ġrequirements Ġfor Ġimage Ġcaption ing Ġsystems : Ġ( a ) Ġthat Ġthe Ġoutput Ġis Ġgram matical , Ġ( b ) Ġthat Ġthe Ġoutput Ġstatement Ġis Ġtrue Ġwith Ġrespect Ġto Ġthe Ġimage , Ġand Ġ( c ) Ġthat Ġoutputs Ġare Ġdiverse Ġand Ġmirror Ġthe Ġvariability Ġof Ġtraining Ġcapt ions . </s> P ractical Ġevaluation Ġof ĠGT D Ġis Ġcurrently Ġonly Ġpossible Ġon Ġsynthetic Ġdata . ĠWe Ġconstruct Ġa Ġrange Ġof Ġdatasets Ġdesigned Ġfor Ġimage Ġcaption ing Ġevaluation . ĠWe Ġcall Ġthis Ġdiagnostic Ġevaluation Ġbenchmark ĠShape World ICE Ġ( Shape World Ġfor ĠImage ĠCaption ing ĠEvaluation ). ĠWe Ġillustrate Ġthe Ġevaluation Ġof Ġspecific Ġimage Ġcaption ing Ġmodels Ġon ĠShape World ICE . ĠWe Ġempir ically Ġdemonstrate Ġthat Ġthe Ġexisting Ġmetrics ĠB LE U Ġand ĠSP ICE Ġdo Ġnot Ġcapture Ġtrue Ġcaption - image Ġagreement Ġin Ġall Ġscenarios , Ġwhile Ġthe ĠGT D Ġframework Ġallows Ġa Ġfine - gr ained Ġinvestigation Ġof Ġhow Ġwell Ġexisting Ġmodels Ġcope Ġwith Ġvaried Ġvisual Ġsituations Ġand Ġlinguistic Ġconstruct ions . </s> We Ġbelieve Ġthat Ġas Ġa Ġsupplementary Ġevaluation Ġmethod Ġto Ġreal - world Ġmetrics , Ġthe ĠGT D Ġframework Ġprovides Ġevaluation Ġinsights Ġthat Ġare Ġsufficiently Ġinteresting Ġto Ġmotivate Ġfuture Ġwork . </s> Related Ġwork Ġ: :: ĠEx isting Ġevaluation Ġof Ġimage Ġcaption ing </s> As Ġa Ġnatural Ġlanguage Ġgeneration Ġtask , Ġimage Ġcaption ing Ġfrequently Ġuses Ġevaluation Ġmetrics Ġsuch Ġas ĠB LE U ĠB IB REF 6 , ĠMET E OR ĠB IB REF 7 , ĠR OU GE ĠB IB REF 8 Ġand ĠC ID Er ĠB IB REF 9 . ĠThese Ġmetrics Ġuse Ġn - gram Ġsimilarity Ġbetween Ġthe Ġcandidate Ġcaption Ġand Ġreference Ġcapt ions Ġto Ġapproximate Ġthe Ġcorrelation Ġbetween Ġa Ġcandidate Ġcaption Ġand Ġthe Ġassociated Ġground Ġtruth . ĠSP ICE ĠB IB REF 10 Ġis Ġa Ġmore Ġrecent Ġmetric Ġspecifically Ġdesigned Ġfor Ġimage Ġcaption ing . ĠFor ĠSP ICE , Ġboth Ġthe Ġcandidate Ġcaption Ġand Ġreference Ġcapt ions Ġare Ġparsed Ġto Ġscene Ġgraphs , Ġand Ġthe Ġagreement Ġbetween Ġtu ples Ġextracted Ġfrom Ġthese Ġscene Ġgraphs Ġis Ġexamined . ĠSP ICE Ġmore Ġclosely Ġrelates Ġto Ġour Ġtruth fulness Ġevaluation Ġthan Ġthe Ġother Ġmetrics , Ġbut Ġit Ġstill Ġuses Ġoverlap Ġcomparison Ġto Ġreference Ġcapt ions Ġas Ġa Ġproxy Ġto Ġground Ġtruth . ĠIn Ġcontrast , Ġour Ġtruth fulness Ġmetric Ġdirectly Ġevaluates Ġa Ġcandidate Ġcaption Ġagainst Ġa Ġmodel Ġof Ġthe Ġactual Ġvisual Ġcontent . </s> Many Ġresearchers Ġhave Ġpointed Ġout Ġproblems Ġwith Ġexisting Ġreference - based Ġmetrics Ġincluding Ġlow Ġcorrelations Ġwith Ġhuman Ġjudgment ĠB IB REF 12 , ĠB IB REF 10 , ĠB IB REF 13 Ġand Ġstrong Ġbas elines Ġusing Ġnearest - ne igh bor Ġmethods ĠB IB REF 14 Ġor Ġrelying Ġsolely Ġon Ġobject Ġdetection ĠB IB REF 15 . ĠFundamental Ġconcerns Ġhave Ġbeen Ġraised Ġwith Ġrespect Ġto ĠB LE U , Ġincluding Ġvariability Ġin Ġparameter ization Ġand Ġprecise Ġscore Ġcalculation Ġleading Ġto Ġsignificantly Ġdifferent Ġresults ĠB IB REF 16 . ĠIts Ġvalidity Ġas Ġa Ġmetric Ġfor Ġtasks Ġother Ġthan Ġmachine Ġtranslation Ġhas Ġbeen Ġquestioned ĠB IB REF 17 , Ġparticularly Ġfor Ġtasks Ġfor Ġwhich Ġthe Ġoutput Ġcontent Ġis Ġnot Ġnarrowly Ġconstrained , Ġlike Ġdialogue ĠB IB REF 18 . </s> Some Ġrecent Ġwork Ġfocuses Ġon Ġincreasing Ġthe Ġdiversity Ġof Ġgenerated Ġcapt ions , Ġfor Ġwhich Ġvarious Ġmeasures Ġare Ġproposed . ĠDev lin Ġet Ġal . ĠB IB REF 19 Ġexplored Ġthe Ġconcept Ġof Ġcaption Ġdiversity Ġby Ġevaluating Ġperformance Ġon Ġcomposition ally Ġnovel Ġimages . Ġvan ĠMil ten burg Ġet Ġal ĠB IB REF 20 Ġframed Ġimage Ġcaption ing Ġas Ġa Ġword Ġrecall Ġtask Ġand Ġproposed Ġseveral Ġmetrics , Ġpredominantly Ġfocusing Ġon Ġdiversity Ġat Ġthe Ġword Ġlevel . ĠHowever , Ġthis Ġdirection Ġis Ġstill Ġrelatively Ġnew Ġand Ġlacks Ġstandardized Ġbenchmarks Ġand Ġmetrics . </s> Related Ġwork Ġ: :: ĠSynt hetic Ġdatasets </s> Recently , Ġmany Ġsynthetic Ġdatasets Ġhave Ġbeen Ġproposed Ġas Ġdiagnostic Ġtools Ġfor Ġdeep Ġlearning Ġmodels , Ġsuch Ġas ĠCLE VR ĠB IB REF 21 Ġfor Ġvisual Ġquestion Ġanswering Ġ( V Q A ), Ġthe Ġb Ab I Ġtasks ĠB IB REF 22 Ġfor Ġtext Ġunderstanding Ġand Ġreasoning , Ġand ĠShape World ĠB IB REF 11 Ġfor Ġvisually Ġgrounded Ġlanguage Ġunderstanding . ĠThe Ġprimary Ġmotivation Ġis Ġto Ġreduce Ġcomplexity Ġwhich Ġis Ġconsidered Ġirrelevant Ġto Ġthe Ġevaluation Ġfocus , Ġto Ġenable Ġbetter Ġcontrol Ġover Ġthe Ġdata , Ġand Ġto Ġprovide Ġmore Ġdetailed Ġinsights Ġinto Ġstrengths Ġand Ġlimitations Ġof Ġexisting Ġmodels . </s> In Ġthis Ġwork , Ġwe Ġdevelop Ġthe Ġevaluation Ġdatasets Ġwithin Ġthe ĠShape World Ġframework . ĠShape World Ġis Ġa Ġcontrolled Ġdata Ġgeneration Ġframework Ġconsisting Ġof Ġabstract Ġcolored Ġshapes Ġ( see ĠFigure ĠFIG REF 1 Ġfor Ġan Ġexample ). ĠWe Ġuse ĠShape World Ġto Ġgenerate Ġtraining Ġand Ġevaluation Ġdata Ġfor Ġtwo Ġmajor Ġreasons . ĠShape World Ġsupports Ġcustomized Ġdata Ġgeneration Ġaccording Ġto Ġuser Ġspecification , Ġwhich Ġenables Ġa Ġvariety Ġof Ġmodel Ġinspections Ġin Ġterms Ġof Ġlanguage Ġconstruction , Ġvisual Ġcomplexity Ġand Ġreasoning Ġability . ĠAnother Ġbenefit Ġis Ġthat Ġeach Ġtraining Ġand Ġtest Ġinstance Ġgenerated Ġin ĠShape World Ġis Ġreturned Ġas Ġa Ġtriple t Ġof Ġ$ < $ image , Ġcaption , Ġworld Ġmodel $ > $. ĠThe Ġworld Ġmodel Ġstores Ġinformation Ġabout Ġthe Ġunderlying Ġmicrow orld Ġused Ġto Ġgenerate Ġan Ġimage Ġand Ġa Ġdescriptive Ġcaption , Ġinternally Ġrepresented Ġas Ġa Ġlist Ġof Ġentities Ġwith Ġtheir Ġattributes , Ġsuch Ġas Ġshape , Ġcolor , Ġposition . ĠDuring Ġdata Ġgeneration , ĠShape World Ġrandomly Ġsamples Ġa Ġworld Ġmodel Ġfrom Ġa Ġset Ġof Ġavailable Ġentities Ġand Ġattributes . ĠThe Ġgenerated Ġworld Ġmodel Ġis Ġthen Ġused Ġto Ġrealize Ġa Ġcorresponding Ġinstance Ġconsisting Ġof Ġimage Ġand Ġcaption . ĠThe Ġworld Ġmodel Ġgives Ġthe Ġactual Ġsemantic Ġinformation Ġcontained Ġin Ġan Ġimage , Ġwhich Ġallows Ġevaluation Ġof Ġcaption Ġtruth fulness . </s> GT D ĠEvaluation ĠFramework </s> In Ġthe Ġfollowing Ġwe Ġintroduce ĠGT D Ġin Ġmore Ġdetail , Ġconsider Ġit Ġas Ġan Ġevaluation Ġprotocol Ġcovering Ġnecessary Ġaspects Ġof Ġthe Ġmultif acet ed Ġcaption ing Ġtask , Ġrather Ġthan Ġa Ġspecific Ġmetric . </s> GT D ĠEvaluation ĠFramework Ġ: :: ĠGram matical ity </s> An Ġessential Ġcriterion Ġfor Ġan Ġimage Ġcaption ing Ġmodel Ġis Ġthat Ġthe Ġcapt ions Ġgenerated Ġare Ġgram matically Ġwell - formed . ĠFully Ġaccurate Ġassessment Ġof Ġgram matical ity Ġin Ġa Ġgeneral Ġcontext Ġis Ġitself Ġa Ġdifficult Ġtask , Ġbut Ġbecomes Ġmore Ġfeasible Ġin Ġa Ġvery Ġconstrained Ġcontext Ġlike Ġour Ġdiagnostic Ġlanguage Ġdata . ĠWe Ġtake Ġparse ability Ġwith Ġthe ĠEnglish ĠResource ĠGram mar ĠB IB REF 23 Ġas Ġa Ġsurrogate Ġfor Ġgram matical ity , Ġmeaning Ġthat Ġa Ġsentence Ġis Ġconsidered Ġgram matically Ġwell - formed Ġif Ġwe Ġobtain Ġa Ġparse Ġusing Ġthe ĠER G . </s> The ĠER G Ġis Ġa Ġbroad - co verage Ġgrammar Ġbased Ġon Ġthe Ġhead - driven Ġphrase Ġstructure Ġgrammar Ġ( H PS G ) Ġframework . ĠIt Ġis Ġlingu istically Ġprecise : Ġsentences Ġonly Ġparse Ġif Ġthey Ġare Ġvalid Ġaccording Ġto Ġits Ġhand - built Ġrules . ĠIt Ġis Ġdesigned Ġto Ġbe Ġgeneral - purpose : Ġverified Ġcoverage Ġis Ġaround Ġ80 % Ġfor ĠWikipedia , Ġand Ġover Ġ90 % Ġfor Ġcorpor a Ġwith Ġshorter Ġsentences Ġand Ġmore Ġlimited Ġvocabulary Ġ( for Ġdetails Ġsee ĠB IB REF 24 Ġflick inger 2011 acc uracy ). ĠSince Ġthe ĠShape World Ġtraining Ġdata ĠâĢĵ Ġthe Ġonly Ġlanguage Ġsource Ġfor Ġmodels Ġto Ġlearn Ġfrom ĠâĢĵ Ġis Ġgenerated Ġusing Ġthe Ġsame Ġgrammar , Ġthe ĠER G Ġhas Ġ$\ sim Ġ$ 100 % Ġcoverage Ġof Ġgram matical ity Ġin Ġthe Ġmodel Ġoutput Ġspace . </s> GT D ĠEvaluation ĠFramework Ġ: :: ĠTruth fulness </s> The Ġsecond Ġaspect Ġwe Ġinvestigate Ġis Ġtruth fulness , Ġthat Ġis , Ġwhether Ġa Ġcandidate Ġcaption Ġis Ġcompatible Ġwith Ġthe Ġcontent Ġof Ġthe Ġimage Ġit Ġis Ġsupposed Ġto Ġdescribe . ĠWe Ġevaluate Ġcaption Ġtruth fulness Ġon Ġthe Ġbasis Ġof Ġa Ġlingu istically - mot ivated Ġapproach Ġusing Ġformal Ġsemantics . ĠWe Ġconvert Ġthe Ġoutput Ġof Ġthe ĠER G Ġparse Ġfor Ġa Ġgram matical Ġcaption Ġto Ġa ĠDepend ency ĠMin imal ĠRec ursion ĠSem antics Ġ( DM RS ) Ġgraph Ġusing Ġthe Ġp yd m rs Ġtool ĠB IB REF 25 . ĠEach Ġconverted ĠDM RS Ġis Ġa Ġlogical Ġsemantic Ġgraph Ġrepresentation Ġcorresponding Ġto Ġthe Ġcaption . ĠWe Ġconstruct Ġa Ġlogical Ġproposition Ġfrom Ġthe ĠDM RS Ġgraph , Ġand Ġevaluate Ġit Ġagainst Ġthe Ġactual Ġworld Ġmodel Ġof Ġthe Ġcorresponding Ġimage . ĠA Ġcaption Ġcan Ġbe Ġsaid Ġto Ġagree Ġwith Ġan Ġimage Ġonly Ġif Ġthe Ġproposition Ġevaluates Ġas Ġtrue Ġon Ġthe Ġbasis Ġof Ġthe Ġworld Ġmodel . ĠBy Ġexamining Ġthe Ġlogical Ġagreement Ġbetween Ġa Ġcaption Ġrepresentation Ġand Ġa Ġworld Ġmodel , Ġwe Ġcan Ġcheck Ġwhether Ġthe Ġsemantics Ġof Ġthis Ġcaption Ġagrees Ġwith Ġthe Ġvisual Ġcontent Ġwhich Ġthe Ġworld Ġmodel Ġrepresents . ĠThus Ġwe Ġdo Ġnot Ġrely Ġon Ġa Ġset Ġof Ġcapt ions Ġas Ġa Ġsurrogate Ġfor Ġthe Ġcontent Ġof Ġan Ġimage , Ġbut Ġinstead Ġleverage Ġthe Ġfact Ġthat Ġwe Ġhave Ġthe Ġground Ġtruth , Ġthus Ġenabling Ġthe Ġevaluation Ġof Ġtrue Ġimage - ca ption Ġagreement . </s> GT D ĠEvaluation ĠFramework Ġ: :: ĠDiversity </s> While Ġgram matical ity Ġand Ġtruth fulness Ġare Ġessential Ġrequirements Ġfor Ġimage Ġcapt ions , Ġthese Ġcriteria Ġalone Ġcan Ġeasily Ġbe ĠâĢ ľ g amed âĢ Ŀ Ġby Ġspecializing Ġon Ġa Ġsmall Ġset Ġof Ġgeneric Ġstatements Ġwhich Ġare Ġtrue Ġmost Ġof Ġthe Ġtime . ĠIn Ġthe Ġcontext Ġof Ġabstract Ġshapes , Ġsuch Ġcapt ions Ġinclude Ġexamples Ġlike ĠâĢ ľ There Ġis Ġa Ġshape âĢ Ŀ Ġor ĠâĢ ľ At Ġleast Ġzero Ġshapes Ġare Ġblue âĢ Ŀ Ġ( which Ġis Ġtechnically Ġtrue Ġeven Ġif Ġthere Ġis Ġno Ġblue Ġshape ). ĠThis Ġmotiv ates Ġthe Ġthird Ġfundamental Ġrequirement Ġof Ġcaption ing Ġoutput Ġto Ġbe Ġdiverse . </s> As ĠShape World ICE Ġexploits Ġa Ġlimited Ġsize Ġof Ġopen - class Ġwords , Ġwe Ġemphasize Ġthe Ġdiversity Ġin ĠShape World ICE Ġat Ġthe Ġsentence Ġlevel Ġrather Ġthan Ġthe Ġword Ġlevel . ĠSince Ġthe Ġground - truth Ġreference Ġcapt ions Ġin ĠShape World Ġare Ġrandomly Ġsampled , Ġwe Ġtake Ġthe Ġsampled Ġcapt ions Ġaccompanying Ġthe Ġtest Ġimages Ġas Ġa Ġproxy Ġfor Ġoptimal Ġcaption Ġdiversity , Ġand Ġcompare Ġit Ġwith Ġthe Ġempirical Ġoutput Ġdiversity Ġof Ġthe Ġevaluated Ġmodel Ġon Ġthese Ġtest Ġimages . ĠPract ically , Ġwe Ġlook Ġat Ġlanguage Ġconstruct ions Ġused Ġand Ġcompute Ġthe Ġcorresponding Ġdiversity Ġscore Ġas Ġthe Ġratio Ġof Ġobserved Ġnumber Ġversus Ġoptimal Ġnumber : </s> Language Ġconstruct ions Ġhere Ġcorrespond Ġto Ġreduced Ġcaption Ġrepresentations Ġwhich Ġonly Ġrecord Ġwhether Ġan Ġobject Ġis Ġdescribed Ġby Ġshape Ġ( e . g ., ĠâĢ ľ square âĢ Ŀ ), Ġcolor Ġ( e . g ., ĠâĢ ľ red Ġshape âĢ Ŀ ) Ġor Ġcolor - shape Ġcombination Ġ( e . g ., ĠâĢ ľ red Ġsquare âĢ Ŀ ). ĠSo Ġthe Ġstatement ĠâĢ ľ A Ġsquare Ġis Ġred âĢ Ŀ Ġand ĠâĢ ľ A Ġcircle Ġis Ġblue âĢ Ŀ Ġare Ġconsidered Ġthe Ġsame , Ġwhile ĠâĢ ľ A Ġshape Ġis Ġred âĢ Ŀ Ġis Ġdifferent . </s> Exper imental ĠSetup Ġ: :: ĠDat as ets </s> We Ġdevelop Ġa Ġvariety Ġof ĠShape World ICE Ġdatasets , Ġwith Ġa Ġsimilar Ġidea Ġto Ġthe ĠâĢ ľ skill Ġtasks âĢ Ŀ Ġin Ġthe Ġb Ab I Ġframework ĠB IB REF 22 . ĠTable ĠT AB REF 4 Ġgives Ġan Ġoverview Ġfor Ġdifferent ĠShape World ICE Ġdatasets Ġwe Ġuse Ġin Ġthis Ġpaper . ĠWe Ġconsider Ġthree Ġdifferent Ġtypes Ġof Ġcaption ing Ġtasks , Ġeach Ġof Ġwhich Ġfocuses Ġon Ġa Ġdistinct Ġaspect Ġof Ġreasoning Ġabilities . ĠEx ist ential Ġdescriptions Ġexamine Ġwhether Ġa Ġcertain Ġobject Ġis Ġpresent Ġin Ġan Ġimage . ĠSp atial Ġdescriptions Ġidentify Ġspatial Ġrelationships Ġamong Ġvisual Ġobjects . ĠQuant ification Ġdescriptions Ġinvolve Ġcount - based Ġand Ġratio - based Ġstatements , Ġwith Ġan Ġexplicit Ġfocus Ġon Ġinspecting Ġmodels Ġfor Ġtheir Ġcounting Ġability . ĠWe Ġdevelop Ġtwo Ġvariants Ġfor Ġeach Ġtype Ġof Ġdataset Ġto Ġenable Ġdifferent Ġlevels Ġof Ġvisual Ġcomplexity Ġor Ġspecific Ġaspects Ġof Ġthe Ġsame Ġreasoning Ġtype . ĠAll Ġthe Ġtraining Ġand Ġtest Ġcapt ions Ġsampled Ġin Ġthis Ġwork Ġare Ġin ĠEnglish . </s> Each Ġdataset Ġvariant Ġconsists Ġof Ġaround Ġ200 k Ġtraining Ġinstances Ġand Ġ4 , 09 6 Ġvalidation Ġinstances , Ġplus Ġ4 , 09 6 Ġtest Ġinstances . ĠEach Ġtraining Ġinstance Ġconsists Ġof Ġan Ġimage Ġand Ġa Ġreference Ġcaption . ĠAt Ġtest Ġtime , Ġonly Ġthe Ġtest Ġimages Ġare Ġavailable Ġto Ġthe Ġevaluated Ġmodels . ĠUnder lying Ġworld Ġmodels Ġare Ġkept Ġfrom Ġthe Ġmodels Ġand Ġare Ġused Ġfor Ġlater ĠGT D Ġevaluation . ĠFor Ġeach Ġtest Ġinstance , Ġwe Ġsample Ġten Ġreference Ġcapt ions Ġof Ġthe Ġsame Ġdistribution Ġas Ġthe Ġtraining Ġcapt ions Ġto Ġenable Ġthe Ġcomparison Ġof Ġour Ġproposed Ġmetrics Ġto ĠB LE U Ġand ĠSP ICE . ĠWe Ġfine - t une Ġour Ġmodel Ġhyper param eters Ġbased Ġon Ġthe Ġperformance Ġon Ġthe Ġvalidation Ġset . ĠAll Ġreported Ġresults Ġare Ġmeasured Ġon Ġthe Ġtest Ġsplit Ġwith Ġthe Ġparameters Ġyielding Ġthe Ġbest Ġvalidation Ġperformance . </s> Exper imental ĠSetup Ġ: :: ĠModels </s> We Ġexperiment Ġwith Ġtwo Ġimage Ġcaption ing Ġmodels : Ġthe ĠShow & Tell Ġmodel ĠB IB REF 0 Ġand Ġthe ĠL RC N 1 u Ġmodel ĠB IB REF 1 . ĠBoth Ġmodels Ġfollow Ġthe Ġbasic Ġenc oder - dec oder Ġarchitecture Ġdesign Ġthat Ġuses Ġa ĠCNN Ġenc oder Ġto Ġcond ense Ġthe Ġvisual Ġinformation Ġinto Ġan Ġimage Ġembed ding , Ġwhich Ġin Ġturn Ġconditions Ġan ĠL ST M Ġdec oder Ġto Ġgenerate Ġa Ġnatural Ġlanguage Ġcaption . ĠThe Ġmain Ġdifference Ġbetween Ġthe Ġtwo Ġmodels Ġis Ġthe Ġway Ġthey Ġcondition Ġthe Ġdec oder . ĠThe ĠShow & Tell Ġmodel Ġfeeds Ġthe Ġimage Ġembed ding Ġas Ġthe ĠâĢ ľ pred ec essor Ġword Ġembed ding âĢ Ŀ Ġto Ġthe Ġfirst Ġproduced Ġword , Ġwhile Ġthe ĠL RC N 1 u Ġmodel Ġconc aten ates Ġthe Ġimage Ġfeatures Ġwith Ġthe Ġembedded Ġprevious Ġword Ġas Ġthe Ġinput Ġto Ġthe Ġsequence Ġmodel Ġat Ġeach Ġtime Ġstep . </s> We Ġfollow Ġthe Ġcommon Ġpractice Ġin Ġimage Ġcaption ing Ġto Ġuse Ġa ĠCNN Ġcomponent Ġpret rained Ġon Ġobject Ġdetection Ġand Ġfine - t une Ġits Ġparameters Ġon Ġthe Ġimage Ġcaption ing Ġtask . ĠThe Ġenc oder Ġand Ġdec oder Ġcomponents Ġare Ġjointly Ġoptimized Ġwith Ġrespect Ġto Ġthe Ġstandard Ġcross - ent ropy Ġsequence Ġloss Ġon Ġthe Ġrespective ĠShape World ICE Ġdataset . ĠFor Ġall Ġour Ġexperiments , Ġwe Ġtrain Ġmodels Ġend - to - end Ġfor Ġa Ġfixed Ġnumber Ġof Ġ100 k Ġiterations Ġwith Ġa Ġbatch Ġsize Ġof Ġ64 . ĠWe Ġuse ĠAdam Ġoptimization ĠB IB REF 26 Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 001 . ĠWord Ġembed d ings Ġare Ġrandomly Ġinitialized Ġand Ġjointly Ġtrained Ġduring Ġthe Ġtraining . </s> Results </s> We Ġtrain Ġand Ġevaluate Ġthe ĠShow & Tell Ġand ĠL RC N 1 u Ġmodels Ġon Ġthe ĠShape World ICE Ġdatasets . ĠHere Ġwe Ġdiscuss Ġin Ġdetail Ġthe Ġdiagnostic Ġresults Ġof Ġthese Ġexperiments . ĠDuring Ġtraining , Ġwe Ġperiodically Ġrecord Ġmodel Ġoutput Ġon Ġthe Ġtest Ġimages , Ġto Ġbe Ġable Ġto Ġanalyze Ġthe Ġdevelopment Ġof Ġour Ġevaluation Ġmetrics Ġthroughout Ġthe Ġprocess . ĠWe Ġalso Ġcompute ĠB LE U - 4 Ġscores Ġand ĠSP ICE Ġscores Ġof Ġgenerated Ġcapt ions Ġfor Ġcomparison , Ġusing Ġ10 Ġreference Ġcapt ions Ġper Ġtest Ġimage . </s> L RC N 1 u Ġexhibits Ġclearly Ġsuperior Ġperformance Ġin Ġterms Ġof Ġtruth fulness . ĠWe Ġstart Ġoff Ġby Ġcomparing Ġperformance Ġof Ġthe ĠShow & Tell Ġmodel Ġand Ġthe ĠL RC N 1 u Ġmodel , Ġsee ĠFigure ĠFIG REF 8 . ĠWhile Ġboth Ġmodels Ġlearn Ġto Ġproduce Ġgram matical Ġsentences Ġearly Ġon , Ġit Ġcan Ġbe Ġseen Ġthat ĠL RC N 1 u Ġis Ġclearly Ġsuperior Ġin Ġterms Ġof Ġtruth fulness , Ġachieving Ġ100 % Ġhalfway Ġthrough Ġtraining , Ġwhereas ĠShow & Tell Ġonly Ġslowly Ġreaches Ġaround Ġ90 % Ġby Ġthe Ġend Ġof Ġ100 k Ġiterations . ĠThis Ġindicates Ġthat Ġincorporating Ġvisual Ġfeatures Ġat Ġevery Ġgeneration Ġstep Ġis Ġbeneficial Ġfor Ġproducing Ġtrue Ġcapt ions . ĠThe Ġdiversity Ġratios Ġof Ġcapt ions Ġgenerated Ġby Ġtwo Ġmodels Ġboth Ġincrease Ġsubstantially Ġas Ġthe Ġtraining Ġprogresses , Ġwith ĠL RC N 1 u Ġexhibiting Ġa Ġslightly Ġgreater Ġcaption Ġdiversity Ġat Ġthe Ġend Ġof Ġtraining . </s> We Ġobserved Ġsimilar Ġresults Ġon Ġother ĠShape World ICE Ġdatasets Ġthat Ġwe Ġexperimented Ġwith , Ġvalid ating Ġthe Ġsuperiority Ġof ĠL RC N 1 u Ġover ĠShow & Tell Ġon ĠShape World ICE . ĠConsequently , Ġwe Ġdecided Ġto Ġfocus Ġon Ġthe ĠL RC N 1 u Ġarchitecture Ġin Ġsubsequent Ġevaluations , Ġwhere Ġwe Ġreport Ġdetailed Ġresults Ġwith Ġrespect Ġto Ġthe ĠGT D Ġframework Ġon Ġa Ġvariety Ġof Ġdatasets . </s> Cor relation Ġbetween Ġthe ĠB LE U / SP ICE Ġscores Ġand Ġthe Ġground Ġtruth . ĠFrom Ġthe Ġlearning Ġcurves Ġshown Ġin ĠFigure ĠFIG REF 9 , Ġwe Ġfind Ġlow Ġor Ġno Ġcorrelation Ġbetween Ġthe ĠB LE U / SP ICE Ġscores Ġand Ġcaption Ġtruth fulness . </s> On ĠEx ist ential - One Shape , Ġthe ĠB LE U Ġcurve Ġfollows Ġthe Ġtrend Ġof Ġthe Ġtruth fulness Ġcurve Ġin Ġgeneral , Ġindicating Ġthat ĠB LE U Ġis Ġable Ġto Ġcapture Ġcaption Ġtruth fulness Ġwell Ġin Ġthis Ġsimple Ġscenario . ĠHowever , Ġwhile ĠB LE U Ġreports Ġequivalent Ġmodel Ġperformance Ġon ĠEx ist ential - Multi Sh apes Ġand ĠSp atial - Multi Sh apes , Ġthe Ġtruth fulness Ġmetric Ġdemonstrates Ġvery Ġdifferent Ġresults . ĠThe ĠB LE U Ġscore Ġfor Ġgenerated ĠEx ist ential - Multi Sh apes Ġcapt ions Ġincreases Ġrapidly Ġat Ġthe Ġbeginning Ġof Ġtraining Ġand Ġthen Ġplate aus Ġdespite Ġthe Ġcontinuous Ġincrease Ġin Ġtruth fulness Ġratio . ĠCapt ions Ġgenerated Ġon ĠSp atial - Multi Sh apes Ġattain Ġa Ġrelatively Ġhigh ĠB LE U Ġscore Ġfrom Ġan Ġearly Ġstage Ġof Ġtraining , Ġbut Ġexhibit Ġlow Ġagreement Ġ($ < $ 0 . 6 Ġtruth fulness Ġratio ) Ġwith Ġground - truth Ġvisual Ġscenes . ĠIn Ġthe Ġcase Ġof ĠSp atial - Multi Sh apes , Ġspatial Ġdescript ors Ġfor Ġtwo Ġobjects Ġare Ġchosen Ġfrom Ġa Ġfixed Ġset Ġ( âĢ ľ above âĢ Ŀ , ĠâĢ ľ below âĢ Ŀ , ĠâĢ ľ to Ġthe Ġleft Ġof âĢ Ŀ Ġand ĠâĢ ľ to Ġthe Ġright Ġof âĢ Ŀ ). ĠIt Ġis Ġvery Ġlikely Ġfor Ġa Ġgenerated Ġspatial Ġdescriptor Ġto Ġmatch Ġone Ġof Ġthe Ġdescript ors Ġmentioned Ġin Ġreference Ġcapt ions . ĠIn Ġthis Ġparticular Ġcase , Ġthe Ġmodel Ġis Ġapt Ġto Ġinfer Ġa Ġcaption Ġwhich Ġhas Ġhigh Ġn - gram Ġoverl aps Ġwith Ġreference Ġcapt ions , Ġresulting Ġin Ġa Ġrelatively Ġhigh ĠB LE U Ġscore . ĠThus Ġan Ġincreased ĠB LE U Ġscore Ġdoes Ġnot Ġnecessarily Ġindicate Ġimproved Ġperformance . </s> While Ġthe Ġtruth fulness Ġand ĠB LE U Ġscores Ġin ĠFigure ĠFIG REF 9 Ġboth Ġincrease Ġrapidly Ġearly Ġon Ġand Ġthen Ġstay Ġstable Ġat Ġa Ġhigh Ġrate Ġafter Ġtraining Ġfor Ġ20 k Ġiterations , Ġthe ĠSP ICE Ġcurve Ġinstead Ġshows Ġa Ġdownward Ġtrend Ġin Ġthe Ġlater Ġstage Ġof Ġtraining . ĠWe Ġexamined Ġthe Ġoutput ĠSP ICE Ġscore Ġfor Ġeach Ġtest Ġinstance . ĠSP ICE Ġreports Ġa Ġprecision Ġscore Ġof Ġ1 . 0 Ġfor Ġmost Ġtest Ġinstances Ġafter Ġ20 k Ġiterations , Ġwhich Ġis Ġconsistent Ġwith Ġthe Ġtruth fulness Ġand ĠB LE U Ġscores . ĠHowever , ĠSP ICE Ġforms Ġthe Ġreference Ġscene Ġgraph Ġas Ġthe Ġunion Ġof Ġthe Ġscene Ġgraphs Ġextracted Ġfrom Ġindividual Ġreference Ġcapt ions , Ġthus Ġintroducing Ġredund ancies . ĠSP ICE Ġuses Ġthe ĠF 1 Ġscore Ġof Ġscene Ġgraph Ġmatching Ġbetween Ġthe Ġcandidate Ġand Ġreference Ġand Ġhence Ġis Ġlowered Ġby Ġimperfect Ġrecall . </s> Comp aring ĠSP ICE Ġcurves Ġfor Ġthree Ġdatasets Ġshown Ġin ĠFigure ĠFIG REF 9 - FIG REF 9 , Ġthey Ġsuggest Ġan Ġincrease Ġin Ġtask Ġcomplexity , Ġbut Ġthey Ġdo Ġnot Ġreflect Ġthe Ġsuccess ively Ġclosing Ġgap Ġof Ġcaption Ġtruth fulness Ġscores Ġbetween Ġtwo ĠEx ist ential Ġdatasets , Ġor Ġthe Ġsubstantial Ġdifference Ġin Ġcaption Ġtruth fulness Ġbetween Ġcapt ions Ġon ĠEx ist ential - Multi Sh apes Ġand ĠSp atial - Multi Sh apes . </s> In Ġthe Ġremainder Ġof Ġthe Ġpaper Ġwe Ġdiscuss Ġin Ġdetail Ġthe Ġdiagnostic Ġresults Ġof Ġthe ĠL RC N 1 u Ġmodel Ġdemonstrated Ġby Ġthe ĠGT D Ġevaluation Ġframework . </s> Perfect Ġgram matical ity Ġfor Ġall Ġcaption Ġtypes . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 15 , Ġgenerated Ġcapt ions Ġfor Ġall Ġtypes Ġof ĠShape World ICE Ġdatasets Ġattain Ġquasi - perfect Ġgram matical ity Ġscores Ġin Ġfewer Ġthan Ġ5 , 000 Ġiterations , Ġsuggesting Ġthat Ġthe Ġmodel Ġquickly Ġlearns Ġto Ġgenerate Ġgram matically Ġwell - formed Ġsentences . </s> Failure Ġto Ġlearn Ġcomplex Ġspatial Ġrelationships . ĠWhile ĠCNN s Ġcan Ġproduce Ġrich Ġvisual Ġrepresentations Ġthat Ġcan Ġbe Ġused Ġfor Ġa Ġvariety Ġof Ġvision Ġtasks ĠB IB REF 27 , Ġit Ġremains Ġan Ġopen Ġquestion Ġwhether Ġthese Ġcondensed Ġvisual Ġrepresentations Ġare Ġrich Ġenough Ġfor Ġmultim od al Ġtasks Ġthat Ġrequire Ġhigher - level Ġabilities Ġof Ġscene Ġunderstanding Ġand Ġvisual Ġreasoning . ĠFrom ĠFigure ĠFIG REF 16 , Ġwe Ġcan Ġsee Ġthat Ġwhile Ġthe Ġmodel Ġperforms Ġrather Ġwell Ġon ĠEx ist ential Ġdatasets , Ġit Ġexhibits Ġa Ġworse Ġperformance Ġon ĠSp atial Ġdata . ĠThe Ġcaption Ġagreement Ġratio Ġin Ġthe Ġsimple ĠSp atial - Two Sh apes Ġscenario Ġis Ġrelatively Ġhigh , Ġbut Ġdrops Ġsignificantly Ġon ĠSp atial - Multi Sh apes , Ġdemonstrating Ġthe Ġdeficiencies Ġof Ġthe Ġmodel Ġin Ġlearning Ġspatial Ġrelationships Ġfrom Ġcomplex Ġvisual Ġscenes . </s> The Ġcounting Ġtask Ġis Ġnon - t riv ial . ĠCount ing Ġhas Ġlong Ġbeen Ġconsidered Ġto Ġbe Ġa Ġchallenging Ġtask Ġin Ġmultim od al Ġreasoning ĠB IB REF 28 , ĠB IB REF 29 . ĠTo Ġexplore Ġhow Ġwell Ġthe ĠL RC N 1 u Ġmodel Ġcop es Ġwith Ġcounting Ġtasks , Ġwe Ġgenerated Ġtwo ĠQuant ification Ġdatasets . ĠThe ĠQuant - Count Ġcapt ions Ġdescribe Ġthe Ġnumber Ġof Ġobjects Ġwith Ġcertain Ġattributes Ġthat Ġappear Ġin Ġan Ġimage Ġ( e . g . ĠâĢ ľ Exactly Ġfour Ġshapes Ġare Ġcrosses âĢ Ŀ ), Ġwhile Ġthe ĠQuant - Rat io Ġcapt ions Ġdescribe Ġthe Ġratio Ġof Ġcertain Ġobjects Ġ( e . g . ĠâĢ ľ A Ġthird Ġof Ġthe Ġshapes Ġare Ġblue Ġsquares âĢ Ŀ ). </s> From ĠFigure ĠFIG REF 16 , Ġwe Ġnotice Ġthat Ġthe ĠL RC N 1 u Ġmodel Ġperforms Ġpoorly Ġon Ġthese Ġdatasets Ġin Ġterms Ġof Ġtruth fulness , Ġreflected Ġin Ġthe Ġ0 . 50 Ġand Ġ0 . 46 Ġscores Ġachieved Ġby Ġthe Ġmodel Ġon Ġthe ĠQuant - Count Ġand ĠQuant - Rat io Ġtasks Ġrespectively . ĠThe Ġlearning Ġcurve Ġfor ĠQuant - Rat io Ġexhibits Ġa Ġmore Ġgradual Ġrise Ġas Ġthe Ġtraining Ġprogresses , Ġsuggesting Ġa Ġgreater Ġcomplexity Ġfor Ġthe Ġratio - based Ġtask . </s> Ca ption Ġdiversity Ġbenefits Ġfrom Ġvaried Ġlanguage Ġconstruct ions Ġin Ġthe Ġtraining Ġdata . ĠThe Ġdiversity Ġratios Ġof Ġgenerated Ġcapt ions Ġfor Ġdifferent ĠShape World ICE Ġdatasets Ġare Ġillustrated Ġin ĠFigure ĠFIG REF 17 . ĠWe Ġcan Ġsee Ġthat Ġthe Ġdiversity Ġof Ġinferred Ġcapt ions Ġis Ġlargely Ġsensitive Ġto Ġthe Ġcaption Ġvariability Ġin Ġthe Ġdataset Ġitself . ĠFor Ġsimple Ġdatasets Ġ( such Ġas ĠEx ist ential - One Shape ) Ġwhere Ġlanguage Ġconstruct ions Ġin Ġthe Ġtraining Ġset Ġare Ġless Ġdiverse , Ġthe Ġoutput Ġcapt ions Ġtend Ġto Ġhave Ġuniform Ġsentence Ġstructures . ĠThe Ġhigh Ġdiversity Ġratios Ġof Ġgenerated ĠSp atial Ġand ĠQuant ification Ġcapt ions Ġsuggest Ġthat Ġcaption Ġdiversity Ġbenefits Ġfrom Ġheter ogeneous Ġlanguage Ġconstruct ions Ġin Ġcomplex Ġdatasets . </s> Disc ussions Ġand ĠCon clusions </s> E val uation Ġmetrics Ġare Ġrequired Ġas Ġa Ġproxy Ġfor Ġperformance Ġin Ġreal Ġapplications . ĠAs Ġsuch , Ġthey Ġshould , Ġas Ġfar Ġas Ġpossible , Ġallow Ġmeasurement Ġof Ġfundamental Ġaspects Ġof Ġthe Ġperformance Ġof Ġmodels Ġon Ġtasks . ĠIn Ġthis Ġwork , Ġwe Ġpropose Ġthe ĠGT D Ġevaluation Ġframework Ġas Ġa Ġsupplement Ġto Ġstandard Ġimage Ġcaption ing Ġevaluation Ġwhich Ġexplicitly Ġfocuses Ġon Ġgram matical ity , Ġtruth fulness Ġand Ġdiversity . ĠWe Ġdeveloped Ġthe ĠShape World ICE Ġevaluation Ġsuite Ġto Ġallow Ġin - depth Ġand Ġfine - gr ained Ġinspection Ġof Ġmodel Ġbehaviors . ĠWe Ġhave Ġempir ically Ġverified Ġthat ĠGT D Ġcaptures Ġdifferent Ġaspects Ġof Ġperformance Ġto Ġexisting Ġmetrics Ġby Ġevaluating Ġimage Ġcaption ing Ġmodels Ġon Ġthe ĠShape World ICE Ġsuite . ĠWe Ġhope Ġthat Ġthis Ġframework Ġwill Ġshed Ġlight Ġon Ġimportant Ġaspects Ġof Ġmodel Ġbehaviour Ġand Ġthat Ġthis Ġwill Ġhelp Ġguide Ġfuture Ġresearch Ġefforts . </s> While Ġperforming Ġthe Ġevaluation Ġexperiments Ġon Ġthe ĠL RC N 1 u Ġmodel , Ġwe Ġnoticed Ġthat Ġcaption Ġagreement Ġdoes Ġnot Ġalways Ġimprove Ġas Ġthe Ġtraining Ġloss Ġdecreases . ĠIdeally , Ġthe Ġtraining Ġobjective Ġshould Ġbe Ġin Ġaccordance Ġwith Ġhow Ġa Ġmodel Ġis Ġeventually Ġevaluated . ĠIn Ġfuture Ġwork , Ġwe Ġplan Ġto Ġinvestigate Ġthe Ġfeasibility Ġof Ġdeliberately Ġencoding Ġthe ĠGT D Ġsignal Ġin Ġthe Ġtraining Ġprocess , Ġfor Ġinstance , Ġby Ġimplementing Ġa ĠGT D - aware Ġloss . ĠWe Ġalso Ġplan Ġto Ġextend Ġthe Ġexisting ĠShape World ICE Ġbenchmark Ġto Ġinclude Ġmore Ġlinguistic Ġconstruct ions Ġ( such Ġas Ġrelative Ġclauses , Ġcompound Ġsentences Ġand Ġcore ference ). ĠBy Ġdoing Ġso , Ġwe Ġhope Ġto Ġreveal Ġhow Ġwell Ġexisting Ġimage Ġcaption ing Ġmodels Ġcope Ġwith Ġcomplex Ġgeneration Ġtasks . </s> Acknowled gments </s> We Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġconstructive Ġfeedback . ĠH X Ġis Ġgrateful Ġfor Ġbeing Ġsupported Ġby Ġthe ĠC SC ĠCambridge ĠScholarship . ĠTS Ġis Ġsupported Ġin Ġpart Ġby Ġthe ĠEPS RC ĠCentre Ġfor ĠDoctor al ĠTraining Ġin ĠData ĠScience , Ġfunded Ġby Ġthe ĠEPS RC Ġ( gr ant ĠEP / L 01 64 27 / 1 ) Ġand Ġthe ĠUniversity Ġof ĠEdinburgh . ĠAK Ġis Ġgrateful Ġfor Ġbeing Ġsupported Ġby Ġa ĠQualcomm ĠResearch ĠStudents hip Ġand Ġan ĠEPS RC ĠDoctor al ĠTraining ĠStudents hip .