Information need of humans is essentially multimodal in nature, enabling maximum exploitation of situated context. We introduce a dataset for sequential procedural (how-to) text generation from images in cooking domain. The dataset consists of 16,441 cooking recipes with 160,479 photos associated with different steps. We setup a baseline motivated by the best performing model in terms of human evaluation for the Visual Story Telling (ViST) task. In addition, we introduce two models to incorporate high level structure learnt by a Finite State Machine (FSM) in neural sequential generation process by: (1) Scaffolding Structure in Decoder (SSiD) (2) Scaffolding Structure in Loss (SSiL). These models show an improvement in empirical as well as human evaluation. Our best performing model (SSiL) achieves a METEOR score of 0.31, which is an improvement of 0.6 over the baseline model. We also conducted human evaluation of the generated grounded recipes, which reveal that 61% found that our proposed (SSiL) model is better than the baseline model in terms of overall recipes, and 72.5% preferred our model in terms of coherence and structure. We also discuss analysis of the output highlighting key important NLP issues for prospective directions.
 Interpretation is heavily conditioned on context. Real world interactions provide this context in multiple modalities. In this paper, the context is derived from vision and language. The description of a picture changes drastically when seen in a sequential narrative context. Formally, this task is defined as: given a sequence of images I = {I 1 , I 2 , ..., I n } and pairwise associated textual descriptions, T = {T 1 , T 2 , ..., T n }; for a new sequence I , our task is to generate the corresponding T . Figure 1 depicts an example for making vegetable lasagna, where the input is the first row and the output is the second row. We call this a 'storyboard', since it unravels the most important steps of a procedure associated with corresponding natural language text. The sequential context differentiates this task from image captioning in isolation. The narration of procedural content draws slight differentiation of this task from visual story telling. The dataset is similar to that presented by ViST Huang et al. (2016) with an apparent difference between stories and instructional in-domain text which is the clear transition in phases of the narrative. This task supplements the task of ViST with richer context of goal oriented procedure (how-to). This paper attempts at capturing this high level structure present in procedural text and imposing this structure while generating sequential text from corresponding sequences of images.Numerous online blogs and videos depict various categories of how-to guides for games, do-ityourself (DIY) crafts, technology, gardening etc. This task lays initial foundations for full fledged storyboarding of a given video, by selecting the right junctions/clips to ground significant events and generate sequential textual descriptions. However, the main focus of this work is generating text from a given set of images. We are going to focus on the domain of cooking recipes in the rest of this paper, leaving the exploration of other domains to future. The two important dimensions to address in text generation are content and structure. In this paper, we discuss our approach in generating more structural/coherent cooking recipes by explicitly modeling the state transitions between different stages of cooking (phases). We address the question of generating textual interpretation of Figure 1 : Storyboard for the recipe of vegetable lasagna the procedure depicted as a sequence of pictures (snapped at different instances of time as the procedure progresses). We introduce a framework to apply traditional FSMs to enhance incorporation of structure in neural text generation. We plan to explore backpropable variants in place of FSMs in future to design structure aware generation models.The two main contributions of this paper are:1. A dataset of 16k recipes targeted for sequential multimodal procedural text generation. 2. Two models (SSiD: Structural Scaffolding in Decoder ,and SSiL: Structural Scaffolding in Loss) for incorporating high level structure learnt by an FSM into a neural text generation model to improve structure/coherence.The rest of the paper is organized as follows. Section 2 describes the related work performed along the lines of planning while generating, understanding food and visual story telling. Section 3 describes the data we gathered for this task and related statistics. In Section 4, we describe our models: a baseline model (Glocal), SSiD and SSiL in detail. Section 5 presents the results attained by each of these models both empirically and qualitatively. Section 6 concludes this work and presents some future directions. The two dimensions explored in clustering and FSM are the number of phases that are learnt in unsupervised manner (P) and the number of states attained through state splitting algorithm in FSM (S). The results of searching this space for the best configuration are presented in Table 2 . Table 2 : BLEU Scores for different number of phases (P) and states(S)The BLEU score BID25 is the highest when P is 40 and S is 100. Fixing these values, we compare the models proposed in TAB4 . The models with hard phases and hard states are not as stable as the one with soft phases since backpropagation affects the impact of the scaffolded phases. Upon manual inspection, a key observation is that for SSiD model, most of the recipes followed a similar structure. It seemed to be conditioned on a global structure learnt from all recipes rather than the current input. However, SSiL model seems to generate recipes that are conditioned on the structure of each particular example. Human Evaluation: We have also performed human evaluation by conducting user preference study to compare the baseline with our best performing SSiL model. We randomly sampled generated outputs of 20 recipes and asked 10 users to answer two preference questions: (1) preference for overall recipe based on images, (2) preference for structurally coherent recipe. For the second question, we gave examples of what structure and phases mean in a recipe. Our SSiL model was preferred 61% and 72.5% for overall and structural preferences respectively. This shows that while there is a viable space to build models that improve structure, generating an edible recipe needs to be explored to improve the overall preference. Our main focus in this paper is instilling structure learnt from FSMs in neural models for sequential procedural text generation with multimodal data. Recipes are being presented in the form of graphic novels reflecting the cultural change in expectations of presenting instructions. With this change, a storyboard is a comprehensive representation of the important events. In this direction, we gather a dataset of 16k recipes where each step has text and associated images. The main difference between the dataset of ViST and our dataset is that our dataset is targeted at procedural how-to kind of text (specifically presenting cooking recipes in this work). We setup a baseline inspired from the best performing model in ViST in the category of human evaluation. We learn a high level structure of the recipe as a sequence of phases and a sequence of hard and soft representations of states learnt from a finite state machine. We propose two techniques for incorporating structure learnt from this as a scaffold. The first model imposes structure on the decoder (SSiD) and the second model imposes structure on the loss function (SSiL) by modeling it as a hierarchical multi-task learning problem. We show that our proposed approach (SSiL) improves upon the baseline and achieves a METEOR score of 0.31, which is an improvement of 0.6 over the baseline model. We plan on exploring backpropable variants as a scaffold for structure in future. We also plan to extend these models to other domains present in these sources of data. There is no standard way to explicitly evaluate the high level strcuture learnt in this task and we would like to explore evaluation strategies for the same.
The paper presents two techniques to incorporate high level structure in generating procedural text from a sequence of images.
{"src": "scitldr-AIC_train", "paper_id": "rJeQE8LYdV"}