Story Understanding through Semantic Analysis and Automatic Alignment of Text and Video
Stories are the pinnacle of human creativity, and yet a ubiquitous phenomenon. An important element of human communication consists of telling and listening, reading, and nowadays watching stories enacted on film. Among different means of story telling, videos (TV series and movies) are a very powerful medium as they have the potential to engage multiple human senses. Artificial Intelligence (AI) has made large strides in the last decade, through which several advances have been achieved in both language and vision. However, these fields have been primarily studied separately, and only in the last few years do we see joint analysis rising to prominence. We argue that the ability to model, analyze, understand, and create new stories is a stepping stone for strong AI a machine that could perform any intellectual task that a human can. Towards this grander goal of story understanding, we seek to build upon a wave of research in the joint study of vision and language. TV series and movies are perfect candidates for such a study, as they are videos produced for the specific purpose of story telling. In this thesis, we define machine understanding of stories as the ability to perform human like tasks upon those stories, such as indexing and searching for story events in large collections, summarizing the stories, and answering questions about them. We address the problem of story understanding from three vantage points. First, we introduce the use of novel sources of natural language text that allow to better learn the content of the videos. Next, we propose a visualization technique to obtain a big picture overview of the story conveyed in a video. Finally, we provide a means to examine machine understanding of stories by using question answering as a surrogate task.