University of Amsterdam and Renmin University at TRECVID 2016: Searching Video, Detecting Events and Describing Video
In this paper we summarize our TRECVID 2016 [1] video recognition experiments. We participated in three tasks: video search, event detection and video description. Here we describe the tasks on event detection and video description. For event detection we explore semantic representations based on VideoStory and an ImageNet Shuffle for both zero shot and few example regimes. For the showcase task on video description we experiment with a deep network that predicts a visual representation from a natural language description, and use this space for the sentence matching. For generative description we enhance a neural image captioning model with Early Embedding and Late Reranking. The 2016 edition of the TRECVID benchmark has been a fruitful participation for our joint team, resulting in the best overall result for zeroand few example event detection as well as video description by matching and in generative mode. 1 Task I: Event Recognition The MediaMill approach to multimedia event detection is optimized for recognition scenarios when video examples are scarce or even completely absent. The key in such a challenging setting is a semantic video representation [13] Our experiments focus on exploring such semantic representations for video search. 1.1 Representation I: VideoStory The first representation is based on VideoStory, as detailed in [6, 7] To summarize, it learns the video representation from freely available web videos and their descriptions using an embedding between video features and term vectors. In the embedding the correlations between the words are utilized to learn a more effective representation by optimizing a joint objective balancing descriptiveness and predictability. We start from a dataset of videos, represented by video features X, and their textual descriptions, represented by binary term vectors Y indicating which words are present in each video description. Then, our VideoStory representation is learned by minimizing: LV(A,W min S Ld(A,S) Lp(S,W (1) where A is the textual projection matrix, W is the visual projection matrix, and S is the VideoStory embedding. The loss function Ld corresponds to our first objective for learning a descriptive VideoStory, and the loss function Lp corresponds to our second objective for learning a predictable VideoStory. The embedding S interconnects the two loss functions. Descriptiveness. For the Ld function, we use a variant of regularized Latent Semantic Indexing. This objective minimizes the quadratic error between the original video descriptions Y and the reconstructed translations obtained from A and S:
TRECVID