we provide the decoder with the context of the whole sequence and the content of the current image (i.e. global and local information) to generate the corresponding text that will contribute to the overall story
EXTRACTIVE
The encoder takes the images in order, one at every timestep $t$ . At time $t=5$ , we obtain the context vector through $h_e^{(t)}$ (represented by $\mathbf {Z}$ ). This vector is used to initialize each decoder's hidden state while the first input to each decoder is its corresponding image embedding $e(I_i)$ . Each decoder generates a sequence of words $\lbrace p_1,...,p_{n}\rbrace $ for each image in the sequence. 
EXTRACTIVE
The decoder is the second LSTM network that uses the information obtained from the encoder to generate the sequence's story. The first input $x_0$ to the decoder is the image for which the text is being generated. The last hidden state from the encoder $h_e^{(t)}$ is used to initialize the first hidden state of the decoder $h_d^{(0)}$ . With this strategy, we provide the decoder with the context of the whole sequence and the content of the current image (i.e. global and local information) to generate the corresponding text that will contribute to the overall story.
Our proposed architecture is presented in Figure 3 . For each image in the sequence, we obtain its representation $\lbrace e(I_1),...,e(I_5)\rbrace $ using Inception v3. The encoder takes the images in order, one at every timestep $t$ . At time $t=5$ , we obtain the context vector through $h_e^{(t)}$ (represented by $\mathbf {Z}$ ). This vector is used to initialize each decoder's hidden state while the first input to each decoder is its corresponding image embedding $e(I_i)$ . Each decoder generates a sequence of words $\lbrace p_1,...,p_{n}\rbrace $ for each image in the sequence. The final story is the concatenation of the output of the 5 decoders.
1806.00738
Introduction </s> Over Ġthe Ġpast Ġfew Ġyears , Ġgenerating Ġtext Ġfrom Ġimages Ġand Ġvideos Ġhas Ġgained Ġa Ġlot Ġof Ġattention Ġin Ġthe ĠComputer ĠVision Ġand ĠNatural ĠLanguage ĠProcessing Ġcommunities Ġand Ġseveral Ġrelated Ġtasks Ġhave Ġbeen Ġproposed , Ġsuch Ġas Ġimage Ġlabeling , Ġimage Ġand Ġvideo Ġdescription Ġand Ġvisual Ġquestion Ġanswering . ĠIn Ġparticular , Ġprominent Ġresults Ġhave Ġbeen Ġachieved Ġin Ġimage Ġdescription Ġwith Ġvarious Ġdeep Ġneural Ġnetwork Ġarchitectures , Ġe . g . ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 0 Ġ. ĠHowever , Ġthe Ġneed Ġof Ġgenerating Ġmore Ġnarrative Ġtexts Ġfrom Ġimages Ġwhich Ġmay Ġreflect Ġexperiences , Ġrather Ġthan Ġjust Ġlisting Ġobjects Ġand Ġtheir Ġattributes , Ġhas Ġgiven Ġrise Ġto Ġtasks Ġsuch Ġas Ġvisual Ġstorytelling ĠB IB REF 4 Ġ. ĠThis Ġtask Ġis Ġabout Ġgenerating Ġa Ġstory Ġfrom Ġa Ġsequence Ġof Ġimages . ĠFigure Ġ1 Ġshows Ġthe Ġdifference Ġbetween Ġdescriptions Ġof Ġimages Ġin Ġisolation Ġand Ġstories Ġfor Ġimages Ġin Ġsequence . </s> In Ġthis Ġpaper , Ġwe Ġdescribe Ġthe Ġdeep Ġneural Ġnetwork Ġarchitecture Ġwe Ġused Ġfor Ġthe ĠVisual ĠStory telling ĠChallenge Ġ2018 . ĠThe Ġproblem Ġto Ġsolve Ġin Ġthis Ġchallenge Ġcan Ġbe Ġstated Ġas Ġfollows : ĠGiven Ġa Ġsequence Ġof Ġ5 Ġimages , Ġthe Ġsystem Ġshould Ġoutput Ġa Ġstory Ġrelated Ġto Ġthe Ġcontent Ġand Ġevents Ġin Ġthe Ġimages . ĠOur Ġarchitecture Ġis Ġan Ġextension Ġof Ġthe Ġimage Ġdescription Ġarchitecture Ġpresented Ġby ĠB IB REF 0 Ġ. ĠWe Ġsubmitted Ġthe Ġgenerated Ġstories Ġto Ġthe Ġinternal Ġtrack Ġof Ġthe ĠVisual ĠStory telling Ġ( V IST ) ĠChallenge , Ġwhich Ġwere Ġevaluated Ġusing Ġthe ĠMET E OR Ġmetric ĠB IB REF 5 Ġas Ġwell Ġas Ġhuman Ġratings . </s> Previous Ġwork </s> The Ġwork Ġby ĠB IB REF 6 Ġpresented Ġprobably Ġthe Ġfirst Ġsystem Ġfor Ġgenerating Ġstories Ġfrom Ġan Ġalbum Ġof Ġimages . ĠThis Ġearly Ġapproach Ġinvolved Ġthe Ġuse Ġof Ġthe ĠNYC Ġand ĠDisney Ġdatasets Ġmined Ġfrom Ġblog Ġposts Ġby Ġthe Ġauthors . </s> The Ġvisual Ġstorytelling Ġtask Ġand Ġdataset Ġwere Ġintroduced Ġby ĠB IB REF 4 Ġ. ĠThis Ġwas Ġthe Ġfirst Ġdataset Ġspecifically Ġcreated Ġfor Ġvisual Ġstorytelling . ĠThey Ġproposed Ġa Ġbaseline Ġapproach Ġwhich Ġconsists Ġof Ġa Ġsequence Ġto Ġsequence Ġmodel , Ġwhere Ġthe Ġenc oder Ġtakes Ġthe Ġsequence Ġof Ġimages Ġas Ġinput Ġand Ġthe Ġdec oder Ġtakes Ġthe Ġlast Ġstate Ġof Ġthe Ġenc oder Ġas Ġits Ġfirst Ġstate Ġto Ġgenerate Ġthe Ġstory . ĠSince Ġthis Ġmodel Ġproduces Ġstories Ġwith Ġgeneric Ġphrases , Ġthey Ġused Ġdecode - time Ġhe ur istics Ġto Ġimprove Ġthe Ġgenerated Ġstories . </s> BI B REF 7 Ġpresented Ġa Ġmulti - task Ġmodel Ġthat Ġperforms Ġalbum Ġsummar ization Ġand Ġstory Ġgeneration . ĠEven Ġthough Ġthe Ġmodel Ġachieved Ġstate - of - the - art Ġscores Ġon Ġthe ĠV IST Ġdataset Ġwith Ġdifferent Ġmetrics , Ġsome Ġof Ġthe Ġsample Ġstories Ġpresented Ġin Ġthe Ġpaper Ġare Ġinco herent . </s> Model </s> Our Ġmodel Ġextends Ġthe Ġimage Ġdescription Ġmodel Ġby ĠB IB REF 0 Ġ, Ġwhich Ġconsists Ġof Ġan Ġenc oder - dec oder Ġarchitecture . ĠThe Ġenc oder Ġis Ġa ĠConv olution al ĠNeural ĠNetwork Ġ( CNN ) Ġand Ġthe Ġdec oder Ġis Ġa ĠLong ĠShort - Term ĠMemory Ġ( L ST M ) Ġnetwork , Ġas Ġpresented Ġin ĠFigure Ġ2 Ġ. ĠThe Ġimage Ġis Ġpassed Ġthrough Ġthe Ġenc oder Ġgenerating Ġthe Ġimage Ġrepresentation Ġthat Ġis Ġused Ġby Ġthe Ġdec oder Ġto Ġknow Ġthe Ġcontent Ġof Ġthe Ġimage Ġand Ġgenerate Ġthe Ġdescription Ġword Ġby Ġword . ĠIn Ġthe Ġfollowing , Ġwe Ġdescribe Ġhow Ġwe Ġextended Ġthis Ġmodel Ġfor Ġthe Ġvisual Ġstorytelling Ġtask . </s> Enc oder </s> The Ġmodel 's Ġfirst Ġcomponent Ġis Ġa ĠRec urrent ĠNeural ĠNetwork Ġ( R NN ), Ġmore Ġprecisely Ġan ĠL ST M Ġthat Ġsummarizes Ġthe Ġsequence Ġof Ġimages . ĠAt Ġevery Ġtim est ep Ġ$ t $ Ġthe Ġnetwork Ġtakes Ġas Ġinput Ġan Ġimage Ġ$ I _ i $ Ġwhere Ġ$ i \ in Ġ\ l brace Ġ1 , 2 , 3 , 4 , 5 \ r brace Ġ$ Ġfrom Ġthe Ġsequence . ĠAt Ġtime Ġ$ t = 5 $ Ġ, Ġthe ĠL ST M Ġhas Ġencoded Ġthe Ġ5 Ġimages Ġand Ġprovides Ġthe Ġsequence 's Ġcontext Ġthrough Ġits Ġlast Ġhidden Ġstate Ġden oted Ġby Ġ$ h _ e ^{ ( t )} $ Ġ. ĠThe Ġrepresentation Ġof Ġthe Ġimages Ġwas Ġobtained Ġthrough ĠIn ception ĠV 3 . </s> Dec oder </s> The Ġdec oder Ġis Ġthe Ġsecond ĠL ST M Ġnetwork Ġthat Ġuses Ġthe Ġinformation Ġobtained Ġfrom Ġthe Ġenc oder Ġto Ġgenerate Ġthe Ġsequence 's Ġstory . ĠThe Ġfirst Ġinput Ġ$ x _ 0 $ Ġto Ġthe Ġdec oder Ġis Ġthe Ġimage Ġfor Ġwhich Ġthe Ġtext Ġis Ġbeing Ġgenerated . ĠThe Ġlast Ġhidden Ġstate Ġfrom Ġthe Ġenc oder Ġ$ h _ e ^{ ( t )} $ Ġis Ġused Ġto Ġinitialize Ġthe Ġfirst Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġ$ h _ d ^{ ( 0 )} $ Ġ. ĠWith Ġthis Ġstrategy , Ġwe Ġprovide Ġthe Ġdec oder Ġwith Ġthe Ġcontext Ġof Ġthe Ġwhole Ġsequence Ġand Ġthe Ġcontent Ġof Ġthe Ġcurrent Ġimage Ġ( i . e . Ġglobal Ġand Ġlocal Ġinformation ) Ġto Ġgenerate Ġthe Ġcorresponding Ġtext Ġthat Ġwill Ġcontribute Ġto Ġthe Ġoverall Ġstory . </s> Our Ġmodel Ġcontains Ġfive Ġindependent Ġdec od ers , Ġone Ġfor Ġeach Ġimage Ġin Ġthe Ġsequence . ĠAll Ġthe Ġ5 Ġdec od ers Ġuse Ġthe Ġlast Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġ( i . e . Ġthe Ġcontext ) Ġas Ġits Ġfirst Ġhidden Ġstate Ġand Ġtake Ġthe Ġcorresponding Ġimage Ġembed ding Ġas Ġits Ġfirst Ġinput . ĠIn Ġthis Ġway , Ġthe Ġfirst Ġdec oder Ġgenerates Ġthe Ġsequence Ġof Ġwords Ġfor Ġthe Ġfirst Ġimage Ġin Ġthe Ġsequence , Ġthe Ġsecond Ġdec oder Ġfor Ġthe Ġsecond Ġimage Ġin Ġthe Ġsequence , Ġand Ġso Ġon . ĠThis Ġallows Ġeach Ġdec oder Ġto Ġlearn Ġa Ġspecific Ġlanguage Ġmodel Ġfor Ġeach Ġposition Ġof Ġthe Ġsequence . ĠFor Ġinstance , Ġthe Ġfirst Ġdec oder Ġwill Ġlearn Ġthe Ġopening Ġsentences Ġof Ġthe Ġstory Ġwhile Ġthe Ġlast Ġdec oder Ġthe Ġclosing Ġsentences . ĠThe Ġword Ġembed d ings Ġwere Ġcomputed Ġusing Ġword 2 vec ĠB IB REF 8 Ġ. </s> Our Ġproposed Ġarchitecture Ġis Ġpresented Ġin ĠFigure Ġ3 Ġ. ĠFor Ġeach Ġimage Ġin Ġthe Ġsequence , Ġwe Ġobtain Ġits Ġrepresentation Ġ$\ l brace Ġe ( I _ 1 ), ... , e ( I _ 5 )\ r brace Ġ$ Ġusing ĠIn ception Ġv 3 . ĠThe Ġenc oder Ġtakes Ġthe Ġimages Ġin Ġorder , Ġone Ġat Ġevery Ġtim est ep Ġ$ t $ Ġ. ĠAt Ġtime Ġ$ t = 5 $ Ġ, Ġwe Ġobtain Ġthe Ġcontext Ġvector Ġthrough Ġ$ h _ e ^{ ( t )} $ Ġ( represented Ġby Ġ$\ math bf Ġ{ Z } $ Ġ). ĠThis Ġvector Ġis Ġused Ġto Ġinitialize Ġeach Ġdec oder 's Ġhidden Ġstate Ġwhile Ġthe Ġfirst Ġinput Ġto Ġeach Ġdec oder Ġis Ġits Ġcorresponding Ġimage Ġembed ding Ġ$ e ( I _ i ) $ Ġ. ĠEach Ġdec oder Ġgenerates Ġa Ġsequence Ġof Ġwords Ġ$\ l brace Ġp _ 1 ,... , p _{ n }\ r brace Ġ$ Ġfor Ġeach Ġimage Ġin Ġthe Ġsequence . ĠThe Ġfinal Ġstory Ġis Ġthe Ġconc aten ation Ġof Ġthe Ġoutput Ġof Ġthe Ġ5 Ġdec od ers . </s> Method ology </s> The Ġgenerated Ġstories Ġwere Ġevaluated Ġusing Ġboth Ġautomatic Ġmetrics Ġand Ġhuman Ġratings . ĠThe Ġautomatic Ġevaluation Ġwas Ġperformed Ġby Ġcomputing Ġthe ĠMET E OR Ġmetric ĠB IB REF 5 Ġon Ġa Ġpublic Ġtest Ġset Ġand Ġa Ġhidden Ġtest Ġset . ĠThe Ġformer Ġis Ġa Ġset Ġof Ġ$ 1 , 9 38 $ Ġimage Ġsequences Ġand Ġstories Ġtaken Ġfrom Ġthe Ġtest Ġset Ġof Ġthe ĠV IST Ġdataset ĠB IB REF 4 Ġ. ĠThe Ġlatter Ġconsists Ġof Ġnew Ġstories Ġgenerated Ġby Ġhumans Ġfrom Ġa Ġsubset Ġof Ġimage Ġsequences Ġof Ġthe Ġpublic Ġtest Ġset . </s> Human Ġratings Ġof Ġthe Ġstories Ġwere Ġcollected Ġfrom Ġcrowd Ġworkers Ġin ĠAmazon ĠMechanical ĠTurk . ĠOnly Ġ200 Ġstories Ġwere Ġselected Ġfrom Ġthe Ġhidden Ġtest Ġset Ġfor Ġthis Ġevaluation . ĠThe Ġcrowd Ġworkers Ġevaluated Ġeach Ġstory Ġon Ġa ĠL ik ert Ġscale Ġwith Ġrespect Ġto Ġ6 Ġaspects : Ġa ) Ġthe Ġstory Ġis Ġfocused , Ġb ) Ġthe Ġstory Ġhas Ġgood Ġstructure Ġand Ġco herence , Ġc ) Ġwould Ġyou Ġshare Ġthis Ġstory , Ġd ) Ġdo Ġyou Ġthink Ġthis Ġstory Ġwas Ġwritten Ġby Ġa Ġhuman , Ġe ) Ġthe Ġstory Ġis Ġvisually Ġgrounded Ġand Ġf ) Ġthe Ġstory Ġis Ġdetailed . ĠThe Ġcrowd Ġworkers Ġwere Ġalso Ġasked Ġto Ġevaluate Ġstories Ġgenerated Ġby Ġhumans Ġfor Ġcomparison Ġpurposes . </s> Results </s> Table Ġ1 Ġshows Ġthe ĠMET E OR Ġscores Ġby Ġour Ġmodel Ġin Ġthe Ġpublic Ġand Ġhidden Ġtest Ġset Ġof Ġthe ĠVisual ĠStory telling ĠChallenge Ġ2018 . ĠTable Ġ2 Ġpresents Ġresults Ġof Ġthe Ġhuman Ġevaluation . ĠOur Ġmodel Ġachieved Ġcompetitive ĠMET E OR Ġscores Ġin Ġboth Ġtest Ġsets Ġand Ġperformed Ġwell Ġin Ġthe Ġhuman Ġevaluation . </s> An Ġevaluation Ġover Ġthe Ġcomplete ĠV IST Ġtest Ġset Ġwas Ġalso Ġperformed Ġand Ġthe Ġresults Ġare Ġshown Ġin ĠTable Ġ3 Ġ. </s> Our Ġmodel Ġobtained Ġthe Ġhighest Ġscores Ġwith Ġthe ĠMET E OR Ġand ĠB LE U - 3 Ġmetrics Ġbut Ġl agged Ġbehind Ġthe Ġmodel Ġby ĠB IB REF 7 Ġwith Ġthe ĠR OU GE Ġand ĠC ID Er Ġmetrics . </s> Figure Ġ4 Ġshows Ġsome Ġsample Ġstories Ġgenerated Ġby Ġour Ġmodel Ġfrom Ġthe Ġpublic Ġtest Ġset Ġof Ġthe ĠVisual ĠStory telling ĠChallenge Ġ2018 . ĠAlthough Ġsome Ġof Ġthe Ġgenerated Ġstories Ġare Ġgram matically Ġcorrect Ġand Ġcoherent , Ġthey Ġtend Ġto Ġcontain Ġrepetitive Ġphrases Ġor Ġideas . ĠWe Ġcan Ġalso Ġobserve Ġthat Ġsome Ġstories Ġare Ġnot Ġnearly Ġrelated Ġto Ġthe Ġactual Ġcontent Ġof Ġthe Ġimages Ġor Ġinclude Ġgeneric Ġphrases Ġlike ĠThis Ġis Ġa Ġpicture Ġof Ġa Ġstore . ĠThese Ġlimitations Ġof Ġour Ġmodel Ġreflected Ġon Ġthe Ġratings Ġof Ġthe Ġvisually Ġgrounded Ġand Ġdetailed Ġaspects Ġof Ġthe Ġhuman Ġevaluation . </s> Con clusions Ġand ĠFuture ĠWork </s> Our Ġvisual Ġstory te ller Ġincorporates Ġa Ġcontext Ġenc oder Ġand Ġmultiple Ġindependent Ġdec od ers Ġto Ġthe Ġimage Ġdescription Ġarchitecture Ġby ĠB IB REF 0 Ġto Ġgenerate Ġstories Ġfrom Ġimage Ġsequences . ĠHaving Ġan Ġindependent Ġdec oder Ġfor Ġeach Ġposition Ġof Ġthe Ġimage Ġsequence , Ġallowed Ġour Ġvisual Ġstory te ller Ġto Ġbuild Ġmore Ġspecific Ġlanguage Ġmodels Ġusing Ġthe Ġcontext Ġvector Ġas Ġits Ġfirst Ġstate Ġand Ġthe Ġimage Ġembed ding Ġas Ġits Ġfirst Ġinput . ĠIn Ġthe Ġinternal Ġtrack Ġof Ġthe ĠVisual ĠStory telling ĠChallenge Ġ2018 , Ġwe Ġobtained Ġcompetitive ĠMET E OR Ġscores Ġin Ġboth Ġthe Ġpublic Ġand Ġhidden Ġtest Ġsets Ġand Ġperformed Ġwell Ġin Ġthe Ġhuman Ġevaluation . </s> In Ġthe Ġfuture , Ġwe Ġplan Ġto Ġexplore Ġthe Ġuse Ġof Ġan Ġattention Ġmechanism Ġor Ġa Ġbid irection al ĠL ST M Ġto Ġcope Ġwith Ġrepetitive Ġphrases Ġwithin Ġthe Ġsame Ġstory . </s> Acknowled gments </s> This Ġresearch Ġis Ġsupported Ġby Ġthe ĠP API IT - UN AM Ġresearch Ġgrant ĠIA 104 016 . ĠDiana ĠGonz Ã¡ lez ĠRico Ġis Ġsupported Ġby ĠCON ACY T .
Introduction Over the past few years, generating text from images and videos has gained a lot of attention in the Computer Vision and Natural Language Processing communities and several related tasks have been proposed, such as image labeling, image and video description and visual question answering. In particular, prominent results have been achieved in image description with various deep neural network architectures, e.g. BIBREF1 , BIBREF2 , BIBREF3 , BIBREF0 . However, the need of generating more narrative texts from images which may reflect experiences, rather than just listing objects and their attributes, has given rise to tasks such as visual storytelling BIBREF4 . This task is about generating a story from a sequence of images. Figure 1 shows the difference between descriptions of images in isolation and stories for images in sequence. In this paper, we describe the deep neural network architecture we used for the Visual Storytelling Challenge 2018. The problem to solve in this challenge can be stated as follows: Given a sequence of 5 images, the system should output a story related to the content and events in the images. Our architecture is an extension of the image description architecture presented by BIBREF0 . We submitted the generated stories to the internal track of the Visual Storytelling (VIST) Challenge, which were evaluated using the METEOR metric BIBREF5 as well as human ratings. Previous work The work by BIBREF6 presented probably the first system for generating stories from an album of images. This early approach involved the use of the NYC and Disney datasets mined from blog posts by the authors. The visual storytelling task and dataset were introduced by BIBREF4 . This was the first dataset specifically created for visual storytelling. They proposed a baseline approach which consists of a sequence to sequence model, where the encoder takes the sequence of images as input and the decoder takes the last state of the encoder as its first state to generate the story. Since this model produces stories with generic phrases, they used decode-time heuristics to improve the generated stories. BIBREF7 presented a multi-task model that performs album summarization and story generation. Even though the model achieved state-of-the-art scores on the VIST dataset with different metrics, some of the sample stories presented in the paper are incoherent. Model Our model extends the image description model by BIBREF0 , which consists of an encoder-decoder architecture. The encoder is a Convolutional Neural Network (CNN) and the decoder is a Long Short-Term Memory (LSTM) network, as presented in Figure 2 . The image is passed through the encoder generating the image representation that is used by the decoder to know the content of the image and generate the description word by word. In the following, we describe how we extended this model for the visual storytelling task. Encoder The model's first component is a Recurrent Neural Network (RNN), more precisely an LSTM that summarizes the sequence of images. At every timestep $t$ the network takes as input an image $I_i$ where $i\in \lbrace 1,2,3,4,5\rbrace $ from the sequence. At time $t=5$ , the LSTM has encoded the 5 images and provides the sequence's context through its last hidden state denoted by $h_e^{(t)}$ . The representation of the images was obtained through Inception V3. Decoder The decoder is the second LSTM network that uses the information obtained from the encoder to generate the sequence's story. The first input $x_0$ to the decoder is the image for which the text is being generated. The last hidden state from the encoder $h_e^{(t)}$ is used to initialize the first hidden state of the decoder $h_d^{(0)}$ . With this strategy, we provide the decoder with the context of the whole sequence and the content of the current image (i.e. global and local information) to generate the corresponding text that will contribute to the overall story. Our model contains five independent decoders, one for each image in the sequence. All the 5 decoders use the last hidden state of the encoder (i.e. the context) as its first hidden state and take the corresponding image embedding as its first input. In this way, the first decoder generates the sequence of words for the first image in the sequence, the second decoder for the second image in the sequence, and so on. This allows each decoder to learn a specific language model for each position of the sequence. For instance, the first decoder will learn the opening sentences of the story while the last decoder the closing sentences. The word embeddings were computed using word2vec BIBREF8 . Our proposed architecture is presented in Figure 3 . For each image in the sequence, we obtain its representation $\lbrace e(I_1),...,e(I_5)\rbrace $ using Inception v3. The encoder takes the images in order, one at every timestep $t$ . At time $t=5$ , we obtain the context vector through $h_e^{(t)}$ (represented by $\mathbf {Z}$ ). This vector is used to initialize each decoder's hidden state while the first input to each decoder is its corresponding image embedding $e(I_i)$ . Each decoder generates a sequence of words $\lbrace p_1,...,p_{n}\rbrace $ for each image in the sequence. The final story is the concatenation of the output of the 5 decoders. Methodology The generated stories were evaluated using both automatic metrics and human ratings. The automatic evaluation was performed by computing the METEOR metric BIBREF5 on a public test set and a hidden test set. The former is a set of $1,938$ image sequences and stories taken from the test set of the VIST dataset BIBREF4 . The latter consists of new stories generated by humans from a subset of image sequences of the public test set. Human ratings of the stories were collected from crowd workers in Amazon Mechanical Turk. Only 200 stories were selected from the hidden test set for this evaluation. The crowd workers evaluated each story on a Likert scale with respect to 6 aspects: a) the story is focused, b) the story has good structure and coherence, c) would you share this story, d) do you think this story was written by a human, e) the story is visually grounded and f) the story is detailed. The crowd workers were also asked to evaluate stories generated by humans for comparison purposes. Results Table 1 shows the METEOR scores by our model in the public and hidden test set of the Visual Storytelling Challenge 2018. Table 2 presents results of the human evaluation. Our model achieved competitive METEOR scores in both test sets and performed well in the human evaluation. An evaluation over the complete VIST test set was also performed and the results are shown in Table 3 . Our model obtained the highest scores with the METEOR and BLEU-3 metrics but lagged behind the model by BIBREF7 with the ROUGE and CIDEr metrics. Figure 4 shows some sample stories generated by our model from the public test set of the Visual Storytelling Challenge 2018. Although some of the generated stories are grammatically correct and coherent, they tend to contain repetitive phrases or ideas. We can also observe that some stories are not nearly related to the actual content of the images or include generic phrases like This is a picture of a store. These limitations of our model reflected on the ratings of the visually grounded and detailed aspects of the human evaluation. Conclusions and Future Work Our visual storyteller incorporates a context encoder and multiple independent decoders to the image description architecture by BIBREF0 to generate stories from image sequences. Having an independent decoder for each position of the image sequence, allowed our visual storyteller to build more specific language models using the context vector as its first state and the image embedding as its first input. In the internal track of the Visual Storytelling Challenge 2018, we obtained competitive METEOR scores in both the public and hidden test sets and performed well in the human evaluation. In the future, we plan to explore the use of an attention mechanism or a bidirectional LSTM to cope with repetitive phrases within the same story. Acknowledgments This research is supported by the PAPIIT-UNAM research grant IA104016. Diana González Rico is supported by CONACYT.
How is the sequential nature of the story captured?
584af673429c7f8621c6bf83362a37048daa0e5d
How Ġis Ġthe Ġsequential Ġnature Ġof Ġthe Ġstory Ġcaptured ?
we Ġprovide Ġthe Ġdec oder Ġwith Ġthe Ġcontext Ġof Ġthe Ġwhole Ġsequence Ġand Ġthe Ġcontent Ġof Ġthe Ġcurrent Ġimage Ġ( i . e . Ġglobal Ġand Ġlocal Ġinformation ) Ġto Ġgenerate Ġthe Ġcorresponding Ġtext Ġthat Ġwill Ġcontribute Ġto Ġthe Ġoverall Ġstory
<s> How Ġis Ġthe Ġsequential Ġnature Ġof Ġthe Ġstory Ġcaptured ? </s> Introduction </s> Over Ġthe Ġpast Ġfew Ġyears , Ġgenerating Ġtext Ġfrom Ġimages Ġand Ġvideos Ġhas Ġgained Ġa Ġlot Ġof Ġattention Ġin Ġthe ĠComputer ĠVision Ġand ĠNatural ĠLanguage ĠProcessing Ġcommunities Ġand Ġseveral Ġrelated Ġtasks Ġhave Ġbeen Ġproposed , Ġsuch Ġas Ġimage Ġlabeling , Ġimage Ġand Ġvideo Ġdescription Ġand Ġvisual Ġquestion Ġanswering . ĠIn Ġparticular , Ġprominent Ġresults Ġhave Ġbeen Ġachieved Ġin Ġimage Ġdescription Ġwith Ġvarious Ġdeep Ġneural Ġnetwork Ġarchitectures , Ġe . g . ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 0 Ġ. ĠHowever , Ġthe Ġneed Ġof Ġgenerating Ġmore Ġnarrative Ġtexts Ġfrom Ġimages Ġwhich Ġmay Ġreflect Ġexperiences , Ġrather Ġthan Ġjust Ġlisting Ġobjects Ġand Ġtheir Ġattributes , Ġhas Ġgiven Ġrise Ġto Ġtasks Ġsuch Ġas Ġvisual Ġstorytelling ĠB IB REF 4 Ġ. ĠThis Ġtask Ġis Ġabout Ġgenerating Ġa Ġstory Ġfrom Ġa Ġsequence Ġof Ġimages . ĠFigure Ġ1 Ġshows Ġthe Ġdifference Ġbetween Ġdescriptions Ġof Ġimages Ġin Ġisolation Ġand Ġstories Ġfor Ġimages Ġin Ġsequence . </s> In Ġthis Ġpaper , Ġwe Ġdescribe Ġthe Ġdeep Ġneural Ġnetwork Ġarchitecture Ġwe Ġused Ġfor Ġthe ĠVisual ĠStory telling ĠChallenge Ġ2018 . ĠThe Ġproblem Ġto Ġsolve Ġin Ġthis Ġchallenge Ġcan Ġbe Ġstated Ġas Ġfollows : ĠGiven Ġa Ġsequence Ġof Ġ5 Ġimages , Ġthe Ġsystem Ġshould Ġoutput Ġa Ġstory Ġrelated Ġto Ġthe Ġcontent Ġand Ġevents Ġin Ġthe Ġimages . ĠOur Ġarchitecture Ġis Ġan Ġextension Ġof Ġthe Ġimage Ġdescription Ġarchitecture Ġpresented Ġby ĠB IB REF 0 Ġ. ĠWe Ġsubmitted Ġthe Ġgenerated Ġstories Ġto Ġthe Ġinternal Ġtrack Ġof Ġthe ĠVisual ĠStory telling Ġ( V IST ) ĠChallenge , Ġwhich Ġwere Ġevaluated Ġusing Ġthe ĠMET E OR Ġmetric ĠB IB REF 5 Ġas Ġwell Ġas Ġhuman Ġratings . </s> Previous Ġwork </s> The Ġwork Ġby ĠB IB REF 6 Ġpresented Ġprobably Ġthe Ġfirst Ġsystem Ġfor Ġgenerating Ġstories Ġfrom Ġan Ġalbum Ġof Ġimages . ĠThis Ġearly Ġapproach Ġinvolved Ġthe Ġuse Ġof Ġthe ĠNYC Ġand ĠDisney Ġdatasets Ġmined Ġfrom Ġblog Ġposts Ġby Ġthe Ġauthors . </s> The Ġvisual Ġstorytelling Ġtask Ġand Ġdataset Ġwere Ġintroduced Ġby ĠB IB REF 4 Ġ. ĠThis Ġwas Ġthe Ġfirst Ġdataset Ġspecifically Ġcreated Ġfor Ġvisual Ġstorytelling . ĠThey Ġproposed Ġa Ġbaseline Ġapproach Ġwhich Ġconsists Ġof Ġa Ġsequence Ġto Ġsequence Ġmodel , Ġwhere Ġthe Ġenc oder Ġtakes Ġthe Ġsequence Ġof Ġimages Ġas Ġinput Ġand Ġthe Ġdec oder Ġtakes Ġthe Ġlast Ġstate Ġof Ġthe Ġenc oder Ġas Ġits Ġfirst Ġstate Ġto Ġgenerate Ġthe Ġstory . ĠSince Ġthis Ġmodel Ġproduces Ġstories Ġwith Ġgeneric Ġphrases , Ġthey Ġused Ġdecode - time Ġhe ur istics Ġto Ġimprove Ġthe Ġgenerated Ġstories . </s> BI B REF 7 Ġpresented Ġa Ġmulti - task Ġmodel Ġthat Ġperforms Ġalbum Ġsummar ization Ġand Ġstory Ġgeneration . ĠEven Ġthough Ġthe Ġmodel Ġachieved Ġstate - of - the - art Ġscores Ġon Ġthe ĠV IST Ġdataset Ġwith Ġdifferent Ġmetrics , Ġsome Ġof Ġthe Ġsample Ġstories Ġpresented Ġin Ġthe Ġpaper Ġare Ġinco herent . </s> Model </s> Our Ġmodel Ġextends Ġthe Ġimage Ġdescription Ġmodel Ġby ĠB IB REF 0 Ġ, Ġwhich Ġconsists Ġof Ġan Ġenc oder - dec oder Ġarchitecture . ĠThe Ġenc oder Ġis Ġa ĠConv olution al ĠNeural ĠNetwork Ġ( CNN ) Ġand Ġthe Ġdec oder Ġis Ġa ĠLong ĠShort - Term ĠMemory Ġ( L ST M ) Ġnetwork , Ġas Ġpresented Ġin ĠFigure Ġ2 Ġ. ĠThe Ġimage Ġis Ġpassed Ġthrough Ġthe Ġenc oder Ġgenerating Ġthe Ġimage Ġrepresentation Ġthat Ġis Ġused Ġby Ġthe Ġdec oder Ġto Ġknow Ġthe Ġcontent Ġof Ġthe Ġimage Ġand Ġgenerate Ġthe Ġdescription Ġword Ġby Ġword . ĠIn Ġthe Ġfollowing , Ġwe Ġdescribe Ġhow Ġwe Ġextended Ġthis Ġmodel Ġfor Ġthe Ġvisual Ġstorytelling Ġtask . </s> Enc oder </s> The Ġmodel 's Ġfirst Ġcomponent Ġis Ġa ĠRec urrent ĠNeural ĠNetwork Ġ( R NN ), Ġmore Ġprecisely Ġan ĠL ST M Ġthat Ġsummarizes Ġthe Ġsequence Ġof Ġimages . ĠAt Ġevery Ġtim est ep Ġ$ t $ Ġthe Ġnetwork Ġtakes Ġas Ġinput Ġan Ġimage Ġ$ I _ i $ Ġwhere Ġ$ i \ in Ġ\ l brace Ġ1 , 2 , 3 , 4 , 5 \ r brace Ġ$ Ġfrom Ġthe Ġsequence . ĠAt Ġtime Ġ$ t = 5 $ Ġ, Ġthe ĠL ST M Ġhas Ġencoded Ġthe Ġ5 Ġimages Ġand Ġprovides Ġthe Ġsequence 's Ġcontext Ġthrough Ġits Ġlast Ġhidden Ġstate Ġden oted Ġby Ġ$ h _ e ^{ ( t )} $ Ġ. ĠThe Ġrepresentation Ġof Ġthe Ġimages Ġwas Ġobtained Ġthrough ĠIn ception ĠV 3 . </s> Dec oder </s> The Ġdec oder Ġis Ġthe Ġsecond ĠL ST M Ġnetwork Ġthat Ġuses Ġthe Ġinformation Ġobtained Ġfrom Ġthe Ġenc oder Ġto Ġgenerate Ġthe Ġsequence 's Ġstory . ĠThe Ġfirst Ġinput Ġ$ x _ 0 $ Ġto Ġthe Ġdec oder Ġis Ġthe Ġimage Ġfor Ġwhich Ġthe Ġtext Ġis Ġbeing Ġgenerated . ĠThe Ġlast Ġhidden Ġstate Ġfrom Ġthe Ġenc oder Ġ$ h _ e ^{ ( t )} $ Ġis Ġused Ġto Ġinitialize Ġthe Ġfirst Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġ$ h _ d ^{ ( 0 )} $ Ġ. ĠWith Ġthis Ġstrategy , Ġwe Ġprovide Ġthe Ġdec oder Ġwith Ġthe Ġcontext Ġof Ġthe Ġwhole Ġsequence Ġand Ġthe Ġcontent Ġof Ġthe Ġcurrent Ġimage Ġ( i . e . Ġglobal Ġand Ġlocal Ġinformation ) Ġto Ġgenerate Ġthe Ġcorresponding Ġtext Ġthat Ġwill Ġcontribute Ġto Ġthe Ġoverall Ġstory . </s> Our Ġmodel Ġcontains Ġfive Ġindependent Ġdec od ers , Ġone Ġfor Ġeach Ġimage Ġin Ġthe Ġsequence . ĠAll Ġthe Ġ5 Ġdec od ers Ġuse Ġthe Ġlast Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġ( i . e . Ġthe Ġcontext ) Ġas Ġits Ġfirst Ġhidden Ġstate Ġand Ġtake Ġthe Ġcorresponding Ġimage Ġembed ding Ġas Ġits Ġfirst Ġinput . ĠIn Ġthis Ġway , Ġthe Ġfirst Ġdec oder Ġgenerates Ġthe Ġsequence Ġof Ġwords Ġfor Ġthe Ġfirst Ġimage Ġin Ġthe Ġsequence , Ġthe Ġsecond Ġdec oder Ġfor Ġthe Ġsecond Ġimage Ġin Ġthe Ġsequence , Ġand Ġso Ġon . ĠThis Ġallows Ġeach Ġdec oder Ġto Ġlearn Ġa Ġspecific Ġlanguage Ġmodel Ġfor Ġeach Ġposition Ġof Ġthe Ġsequence . ĠFor Ġinstance , Ġthe Ġfirst Ġdec oder Ġwill Ġlearn Ġthe Ġopening Ġsentences Ġof Ġthe Ġstory Ġwhile Ġthe Ġlast Ġdec oder Ġthe Ġclosing Ġsentences . ĠThe Ġword Ġembed d ings Ġwere Ġcomputed Ġusing Ġword 2 vec ĠB IB REF 8 Ġ. </s> Our Ġproposed Ġarchitecture Ġis Ġpresented Ġin ĠFigure Ġ3 Ġ. ĠFor Ġeach Ġimage Ġin Ġthe Ġsequence , Ġwe Ġobtain Ġits Ġrepresentation Ġ$\ l brace Ġe ( I _ 1 ), ... , e ( I _ 5 )\ r brace Ġ$ Ġusing ĠIn ception Ġv 3 . ĠThe Ġenc oder Ġtakes Ġthe Ġimages Ġin Ġorder , Ġone Ġat Ġevery Ġtim est ep Ġ$ t $ Ġ. ĠAt Ġtime Ġ$ t = 5 $ Ġ, Ġwe Ġobtain Ġthe Ġcontext Ġvector Ġthrough Ġ$ h _ e ^{ ( t )} $ Ġ( represented Ġby Ġ$\ math bf Ġ{ Z } $ Ġ). ĠThis Ġvector Ġis Ġused Ġto Ġinitialize Ġeach Ġdec oder 's Ġhidden Ġstate Ġwhile Ġthe Ġfirst Ġinput Ġto Ġeach Ġdec oder Ġis Ġits Ġcorresponding Ġimage Ġembed ding Ġ$ e ( I _ i ) $ Ġ. ĠEach Ġdec oder Ġgenerates Ġa Ġsequence Ġof Ġwords Ġ$\ l brace Ġp _ 1 ,... , p _{ n }\ r brace Ġ$ Ġfor Ġeach Ġimage Ġin Ġthe Ġsequence . ĠThe Ġfinal Ġstory Ġis Ġthe Ġconc aten ation Ġof Ġthe Ġoutput Ġof Ġthe Ġ5 Ġdec od ers . </s> Method ology </s> The Ġgenerated Ġstories Ġwere Ġevaluated Ġusing Ġboth Ġautomatic Ġmetrics Ġand Ġhuman Ġratings . ĠThe Ġautomatic Ġevaluation Ġwas Ġperformed Ġby Ġcomputing Ġthe ĠMET E OR Ġmetric ĠB IB REF 5 Ġon Ġa Ġpublic Ġtest Ġset Ġand Ġa Ġhidden Ġtest Ġset . ĠThe Ġformer Ġis Ġa Ġset Ġof Ġ$ 1 , 9 38 $ Ġimage Ġsequences Ġand Ġstories Ġtaken Ġfrom Ġthe Ġtest Ġset Ġof Ġthe ĠV IST Ġdataset ĠB IB REF 4 Ġ. ĠThe Ġlatter Ġconsists Ġof Ġnew Ġstories Ġgenerated Ġby Ġhumans Ġfrom Ġa Ġsubset Ġof Ġimage Ġsequences Ġof Ġthe Ġpublic Ġtest Ġset . </s> Human Ġratings Ġof Ġthe Ġstories Ġwere Ġcollected Ġfrom Ġcrowd Ġworkers Ġin ĠAmazon ĠMechanical ĠTurk . ĠOnly Ġ200 Ġstories Ġwere Ġselected Ġfrom Ġthe Ġhidden Ġtest Ġset Ġfor Ġthis Ġevaluation . ĠThe Ġcrowd Ġworkers Ġevaluated Ġeach Ġstory Ġon Ġa ĠL ik ert Ġscale Ġwith Ġrespect Ġto Ġ6 Ġaspects : Ġa ) Ġthe Ġstory Ġis Ġfocused , Ġb ) Ġthe Ġstory Ġhas Ġgood Ġstructure Ġand Ġco herence , Ġc ) Ġwould Ġyou Ġshare Ġthis Ġstory , Ġd ) Ġdo Ġyou Ġthink Ġthis Ġstory Ġwas Ġwritten Ġby Ġa Ġhuman , Ġe ) Ġthe Ġstory Ġis Ġvisually Ġgrounded Ġand Ġf ) Ġthe Ġstory Ġis Ġdetailed . ĠThe Ġcrowd Ġworkers Ġwere Ġalso Ġasked Ġto Ġevaluate Ġstories Ġgenerated Ġby Ġhumans Ġfor Ġcomparison Ġpurposes . </s> Results </s> Table Ġ1 Ġshows Ġthe ĠMET E OR Ġscores Ġby Ġour Ġmodel Ġin Ġthe Ġpublic Ġand Ġhidden Ġtest Ġset Ġof Ġthe ĠVisual ĠStory telling ĠChallenge Ġ2018 . ĠTable Ġ2 Ġpresents Ġresults Ġof Ġthe Ġhuman Ġevaluation . ĠOur Ġmodel Ġachieved Ġcompetitive ĠMET E OR Ġscores Ġin Ġboth Ġtest Ġsets Ġand Ġperformed Ġwell Ġin Ġthe Ġhuman Ġevaluation . </s> An Ġevaluation Ġover Ġthe Ġcomplete ĠV IST Ġtest Ġset Ġwas Ġalso Ġperformed Ġand Ġthe Ġresults Ġare Ġshown Ġin ĠTable Ġ3 Ġ. </s> Our Ġmodel Ġobtained Ġthe Ġhighest Ġscores Ġwith Ġthe ĠMET E OR Ġand ĠB LE U - 3 Ġmetrics Ġbut Ġl agged Ġbehind Ġthe Ġmodel Ġby ĠB IB REF 7 Ġwith Ġthe ĠR OU GE Ġand ĠC ID Er Ġmetrics . </s> Figure Ġ4 Ġshows Ġsome Ġsample Ġstories Ġgenerated Ġby Ġour Ġmodel Ġfrom Ġthe Ġpublic Ġtest Ġset Ġof Ġthe ĠVisual ĠStory telling ĠChallenge Ġ2018 . ĠAlthough Ġsome Ġof Ġthe Ġgenerated Ġstories Ġare Ġgram matically Ġcorrect Ġand Ġcoherent , Ġthey Ġtend Ġto Ġcontain Ġrepetitive Ġphrases Ġor Ġideas . ĠWe Ġcan Ġalso Ġobserve Ġthat Ġsome Ġstories Ġare Ġnot Ġnearly Ġrelated Ġto Ġthe Ġactual Ġcontent Ġof Ġthe Ġimages Ġor Ġinclude Ġgeneric Ġphrases Ġlike ĠThis Ġis Ġa Ġpicture Ġof Ġa Ġstore . ĠThese Ġlimitations Ġof Ġour Ġmodel Ġreflected Ġon Ġthe Ġratings Ġof Ġthe Ġvisually Ġgrounded Ġand Ġdetailed Ġaspects Ġof Ġthe Ġhuman Ġevaluation . </s> Con clusions Ġand ĠFuture ĠWork </s> Our Ġvisual Ġstory te ller Ġincorporates Ġa Ġcontext Ġenc oder Ġand Ġmultiple Ġindependent Ġdec od ers Ġto Ġthe Ġimage Ġdescription Ġarchitecture Ġby ĠB IB REF 0 Ġto Ġgenerate Ġstories Ġfrom Ġimage Ġsequences . ĠHaving Ġan Ġindependent Ġdec oder Ġfor Ġeach Ġposition Ġof Ġthe Ġimage Ġsequence , Ġallowed Ġour Ġvisual Ġstory te ller Ġto Ġbuild Ġmore Ġspecific Ġlanguage Ġmodels Ġusing Ġthe Ġcontext Ġvector Ġas Ġits Ġfirst Ġstate Ġand Ġthe Ġimage Ġembed ding Ġas Ġits Ġfirst Ġinput . ĠIn Ġthe Ġinternal Ġtrack Ġof Ġthe ĠVisual ĠStory telling ĠChallenge Ġ2018 , Ġwe Ġobtained Ġcompetitive ĠMET E OR Ġscores Ġin Ġboth Ġthe Ġpublic Ġand Ġhidden Ġtest Ġsets Ġand Ġperformed Ġwell Ġin Ġthe Ġhuman Ġevaluation . </s> In Ġthe Ġfuture , Ġwe Ġplan Ġto Ġexplore Ġthe Ġuse Ġof Ġan Ġattention Ġmechanism Ġor Ġa Ġbid irection al ĠL ST M Ġto Ġcope Ġwith Ġrepetitive Ġphrases Ġwithin Ġthe Ġsame Ġstory . </s> Acknowled gments </s> This Ġresearch Ġis Ġsupported Ġby Ġthe ĠP API IT - UN AM Ġresearch Ġgrant ĠIA 104 016 . ĠDiana ĠGonz Ã¡ lez ĠRico Ġis Ġsupported Ġby ĠCON ACY T .