Many semantic video analysis tasks can benefit from multiple, heterogenous signals. For example, in addition to the original RGB input sequences, sequences of optical flow are usually used to boost the performance of human action recognition in videos. To learn from these heterogenous input sources, existing methods reply on two-stream architectural designs that contain independent, parallel streams of Recurrent Neural Networks (RNNs). However, two-stream RNNs do not fully exploit the reciprocal information contained in the multiple signals, let alone exploit it in a recurrent manner. To this end, we propose in this paper a novel recurrent architecture, termed Coupled Recurrent Network (CRN), to deal with multiple input sources. In CRN, the parallel streams of RNNs are coupled together. Key design of CRN is a Recurrent Interpretation Block (RIB) that supports learning of reciprocal feature representations from multiple signals in a recurrent manner. Different from RNNs which stack the training loss at each time step or the last time step, we propose an effective and efficient training strategy for CRN. Experiments show the efficacy of the proposed CRN. In particular, we achieve the new state of the art on the benchmark datasets of human action recognition and multi-person pose estimation.
In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. It directly models the probability distribution of generating a word given previous words and an image. Image captions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on four benchmark datasets (IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO). Our model outperforms the state-of-the-art methods. In addition, we apply the m-RNN model to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. The project page of this work is: www.stat.ucla.edu junhua.mao m-RNN.html . Abstract : For most people, watching a brief video and describing what happened (inwords) is an easy task. For machines, extracting the meaning from video pixelsand generating a sentence description is a very complex problem. The goal of myresearch is to develop models that can automatically generate natural language(NL) descriptions for events in videos. As a first step, this proposal presentsdeep recurrent neural network models for video to text generation. I build onrecent deep machine learning approaches to develop video description modelsusing a unified deep neural network with both convolutional and recurrentstructure. This technique treats the video domain as another language andtakes a machine translation approach using the deep network to translate videosto text. In my initial approach, I adapt a model that can learn on images andcaptions to transfer knowledge from this auxiliary task to generate descriptionsfor short video clips. Next, I present an end-to-end deep network that can jointlymodel a sequence of video frames and a sequence of words. The second part ofthe proposal outlines a set of models to significantly extend work in this area.Specifically, I propose techniques to integrate linguistic knowledge from plaintext corpora; and attention methods to focus on objects and track their interactionsto generate more diverse and accurate descriptions. To move beyondshort video clips, I also outline models to process multi-activity movie videos,learning to jointly segment and describe coherent event sequences. I proposefurther extensions to take advantage of movie scripts and subtitle informationto generate richer descriptions.