VIREO TRECVID 2016: Multimedia Event Detection, Ad hoc Video Search, Video to Text Description
The vireo group participates in 3 tasks: multimedia event detection, ad hoc video search, video to text description. In this paper, we will separately present frameworks for these tasks and discuss experimental results. Multimedia Event Detection (MED) We will present an overview and comparative analysis of our system designed for the TRECVID 2016 [1] multimedia event detection (MED) task. We submit 12 runs, of which 3 runs for the pre specified zero example (PS 0Ex) 4 runs for the pre specified ten examples (PS10Ex) 3 runs for the pre specified hundred examples (PS 100Ex) and 2 runs for the ad hoc ten examples (AH 10Ex) We do not participate in the interactive run. This year we focus on two aspects: 1) Refining concept bank by extending its vocabulary size and replacing its classifiers with the state of the art deep architecture; 2) Capturing visual scenario of video with a finer granularity object level representation. Among our 12 submitted runs, the following runs achieved the top 2 performances: VIREO MED16 MED16EvalFull PS 0Ex MED p VisualTextualRernk 4: Zero example system with manually refined semantic query, highlighted concept reranking and late fusion with OCR (Best 0Ex performance) VIREO MED16 MED16EvalFull PS 10Ex MED p Fus 4: 10 example system using Concept Bank feature and Object VLAD video representation (Ranked 2nd on EvalSub/Full set in terms of MinfAP200) VIREO MED16 MED16EvalFull AH 10Ex MED p fus 4: 10 example system using Concept Bank feature and Object VLAD video representation (Ranked 2nd on EvalFull set and 1st on EvalSub set in terms of MinfAP200) Ad hoc Video Search (AVS) For AVS, we adopt our successful zero example MED system developed in the last year. In addition, we add a new highlighted concept reranking feature to further enhance human in the loop. This technique allows a user to emphasize a group of concepts highlighting a specific characteristic. For example, "outdoors" and "at night." We submit a total of 6 runs, with variations include (1) fully automatic or manually assisted query processing, (2) with or without highlighted concept reranking, and (3) using concepts learned only from IACC data or not. The best runs in the respective two training set types are: M D VIREO.16 5: Manually assisted query processing. Concept detectors of more than 10K which are collected from several off the shelf public datasets. M A VIREO.16 3: Manually assisted query processing. 346 concept detectors are only learned from IACC.1 and IACC.2 data themselves. Video to Text Description (VTT) We only participate in the matching and ranking subtask. The task here is to return for each video URL a ranked list of the most likely text description that correspond to the video. This year we focus on the concept based retrieval of sentence and inter modality correspondence learning by salient region selection. Our four runs are summarised below: Vireo Average VGG C3D: Each video is represented by average pooling over the VGG [2] features extracted from frames and text description is represented by the TCNN [3] feature. Stacked attention network (SAN) [4] is adopted to learn the implicit alignment between visual and textual features. Furthermore, C3D [5] is also employed for the extraction of motion features. Similarity scores from the two SANs trained on VGG TCNN and C3D TCNN are averagely fused for the final ranking. Vireo Flat VGG C3D: Rather than average pooling, frame features are concatenated into one feature vector to preserve the unique information of different frames. Then, two SANs are trained on the concatenated features, whose similarity scores are averagely fused for final ranking. Vireo Fusion: To observe whether Vireo Average VGG C3D and Vireo Flat VGG C3D are complementary to each other, we further averagely fuse the scores from these two runs for ranking. Vireo Concept: Each video is represented by our Concept Bank feature and the detected concepts are matched to the keywords in text descriptions based on a flexible phrase matching method, which is combined with word similarities of WordNet [6] and weighting strategies, such as TF IDF. 1 Multimedia Event Detection (MED) Compared with our multimedia event detection system proposed in TRECVID 2015 benchmark, we modify visual systems for PS 10Ex, AH 10Ex and PS 100Ex subtasks, whereas, keep the subsystem for PS 0Ex. Specifically, the differences of visual systems are summarized as follows: 1) Hand crafted features (e.g. the improved dense trajectory [7] are no longer used in visual system; 2) By including ImageNet Shuffel 13K [8] the size of concept bank is increased to 15,762 and parts of concept classifiers are re trained with the architecture of deep residual neural networks [9] 3) A finer granularity object level video representation named Object VLAD is introduced in our visual systems. Experiments conducted on MED14 Test and MED16 EvalFull datasets verify the effectiveness of our modification. The next section explains how to generate different modalities for our system. Section 1.1 and Section 1.2 describe our systems and runs for the different amount of training examples, i.e. 0Ex, 10Ex, or 100Ex. Section 1.3 describes results and analysis. Section 1.4 concludes the MED section. 1.1 Modalities In our system we use three types of modalities: visual, texutal and speech. 1.1.1 Visual For our visual system, we first decompose each video into two granularity levels keyframe level and shot level. The keyframe sampling rate is set to be one frame per two seconds and the time duration of shot is set to be five seconds. For each video, we generate 7 kinds of high level concept feature and 1 kind of object level deep feature. All the features used in our visual system are summarized as below: ImageNet 1000 We use the architecture of deep residual networks (i.e. ResNet 50) proposed by K. He [9] to train ImageNet 1000 classifiers. The parameters of ResNet 50 are learned on ILSVRC dataset [10] which is a subset of ImageNet dataset with 1.26 million training images for 1,000 categories. The neural responses of the output layer (soft max) for each keyframe are averagely pooled to form the videolevel feature vector. SIN 346 A set of 346 concept detectors fine tuned with the ResNet 50 architecture on TRECVID SIN 2014 dataset [11] is applied on all keyframes and averagely pooled to form a video level representation. RC 497 Similar to [12] we select 497 concepts from the MED14 Research Collection dataset [13] We manually annotate at most 200 positive keyframes for each concept and fine tuned 497 concept detectors using the RestNet 50 architecture. Same as the prior methods, we concatenate the responses of the concept classifiers on each keyframe and averagely pool the resulting feature vectors to form a video feature representation. Places 205 To capture the scenery information of multimedia events, we fine tune 205 scene categories on MIT places dataset [14] with the RestNet 50 architecture. The scenery concept responses are extracted from each keyframe and further averagely pooled to generate the video level representation. FCVID 239 To capture the action/motion information of multimedia events, 239 concept detectors are trained with SVM on FCVID [15] dataset, which contains 91,223 web videos. Since concepts from this dataset are mainly action/motion concepts annotated at video level, we extract neural responses from fc7 layers of AlexNet [16] for each frame and pool them across the temporal domain to generate training features and train video level concept detectors. We concatenate the responses of 205 concept detectors of an Image. Sports 487 487 concept detectors are trained with 3D CNN structure [5] on the Sport 1M dataset [17] containing 1 million videos. Similar to FCVID, Sports 1M are mainly action/motion concepts annotated at video level. ImageNet Shuffle 12988 In [8] P. Mettes et. al proposed to reorganize 21,814 classes of full ImageNet [10] dataset by merging them into 12,988 categories for training a large scale concept bank with GoogLeNet architecture [18] We extract neural responses from output layer of the published ImageNet Shuffle 13K GoogLeNet for each frame and averagely pool them across the temporal domain to generate video level representation. To capture semantical meaning of multimedia event, we collect all the above concept features and form a large Concept Bank with 15,762 semantical concepts related to visual objects, background scenery and actions. Object VLAD Due to the employment of max pooling and softmax layer in DCNN structure, concept detectors tend to overlook activations of small size objects, emphasize those of primary objects and produce a very sparse semantical vector. To equally accumulate responses of multiple objects within a frame, we propose a finer granularity object level representation named Object VLAD for videos. Specifically, for each video, we separately encode frame level features and object level features with VLAD. For the prior, we directly adopt CNN VLAD [19] which encodes frame level features with VLAD, whereas for the latter, we utilize selective search [20] for proposing candidate objects, describe objects with deep features and encode them with VLAD. We adopt neural responses from Res5A layer of the RestNet 50 architecture (trained on ImageNet 1000 categories) [9] as feature descriptors for each frame and regional object. 1.1.2 Textual Tesseract OCR [21] is used to extract text from video frames. The engine is applied in a brute force manner with post processing on the resulting extracted texts. From a video segment, every key video frame (roughly one per second) is decoded using the FFmpeg open source library. All key frames are then fed to the Tesseract engine for OCR. The resulting texts are analyzed to check whether meaningful text is extracted from the video frame. This post processing checks the words in the extracted text per frame using the following rules: e
TRECVID