In this work we combine two research threads from Vision Graphics and Natural Language Processing to formulate an image generation task conditioned on attributes in a multi-turn setting. By multiturn, we mean the image is generated in a series of steps of user-specified conditioning information. Our proposed approach is practically useful and offers insights into neural interpretability. We introduce a framework that includes a novel training algorithm as well as model improvements built for the multi-turn setting. We demonstrate that this framework generates a sequence of images that match the given conditioning information and that this task is useful for more detailed benchmarking and analysis of conditional image generation methods.
Beyond the brief overview of concrete problems and long-term goals in NLP that help to define our task, we would like to draw the reader's attention to the significant body of work in the Human-Computer Interaction community over the past several decades on the space of control, feedback, and multi-step interaction with computers and technology. Much of this work phrases these interactions as a form of collaborative conversation @cite_11 , while also building upon empirical and theoretical results showing that people interact with computers in such a conversational fashion @cite_26 @cite_0 . The emphasis of @cite_29 on incremental and rapid feedback to changes taken by the user suggest that our task has implications for building better experiences using vision and NLP, beyond the usefulness of phrasing problems as multi-turn for performance.
@cite_0 @cite_29 @cite_26 @cite_11
2962809918 2411480514 1974302391 2295887377
Enabling a computer to understand a document so that it can answer comprehension questions is a central, yet unsolved goal of NLP. A key factor impeding its solution by machine learned systems is the limited availability of human-annotated data. (2015) seek to solve this problem by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and show that a neural network can then be trained to give good performance on this task. In this paper, we conduct a thorough examination of this new reading comprehension task. Our primary aim is to understand what depth of language understanding is required to do well on this task. We approach this from one side by doing a careful hand-analysis of a small subset of the problems and from the other by showing that simple, carefully designed systems can obtain accuracies of 72.4 and 75.8 on these two datasets, exceeding current state-of-the-art results by over 5 and approaching what we believe is the ceiling for performance on this task.1 Enabling a computer to understand a document so that it can answer comprehension questions is a central, yet unsolved goal of NLP. A key factor impeding its solution by machine learned systems is the limited availability of human-annotated data. (2015) seek to solve this problem by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and show that a neural network can then be trained to give good performance on this task. In this paper, we conduct a thorough examination of this new reading comprehension task. Our primary aim is to understand what depth of language understanding is required to do well on this task. We approach this from one side by doing a careful hand-analysis of a small subset of the problems and from the other by showing that simple, carefully designed systems can obtain accuracies of 73.6 and 76.6 on these two datasets, exceeding current state-of-the-art results by 7-10 and approaching what we believe is the ceiling for performance on this task. Abstract : Persistent controversies about human multiple task performance suggest that research on it will benefit from increased use of precise computational models. Toward this objective, the present report outlines a comprehensive theoretical framework for understanding and predicting the performance of concurrent perceptual motor and cognitive tasks. The framework involves an Executive Process Interactive Control (EPIC) architecture, which has component modules that process information at perceptual, cognitive, and motor levels. On the basis of EPIC, computational models that use a production system formalism may be constructed to simulate multiple task performance under a variety of conditions. These models account well for reaction time data from representative paradigms such as the psychological refractory period (PRP) procedure. With modest numbers of parameters, good fits between empirical and simulated reaction times support several key conclusions: (1) at a cognitive level, people can apply distinct sets of production rules simultaneously for executing the procedures of multiple tasks; (2) there is no immutable central response selection or decision bottleneck; (3) people's capacity to process information and take action at peripheral perceptual motor levels is limited; (4) to cope with such limits and to satisfy task priorities, flexible scheduling strategies are used; (5) these strategies are mediated by executive cognitive processes that coordinate concurrent tasks adaptively. The initial success of EPIC and models based on it suggest that they may help characterize multiple task performance across many domains, including ones that have substantial practical relevance. The goal of this work is to develop computational models of social intelligence that enable robots to work side by side with humans, solving problems and achieving task goals through dialogue and collaborative manipulation. A defining problem of collaborative behavior in an embodied setting is the manner in which multiple agents make use of shared resources. In a situated dialogue, these resources include physical bottlenecks such as objects or spatial regions, and cognitive bottlenecks such as the speaking floor. For a robot to function as an effective collaborative partner with a human, it must be able to seize and yield such resources appropriately according to social expectations. We describe a general framework that uses timed Petri nets for the modeling and execution of robot speech, gaze, gesture, and manipulation for collaboration. The system dynamically monitors resource requirements and availability to control real-time turn-taking decisions over resources that are shared with humans, reasoning about different resource types independently. We evaluate our approach with an experiment in which our robot Simon performs a collaborative assembly task with 26 different human partners, showing that the multimodal reciprocal approach results in superior task performance, fluency, and balance of control.