The second-best approach was proposed by Shin et al. [1]}, who proposed a temporal segmentation strategy based on action localization from the videos. To extract motion features, they used the Dense Trajectories approach [2]}.
They also used a temporal sliding window and non-maximum suppression for temporal action localization; these temporal sliding windows were fixed in specific lengths of 30, 60, 90, and 120 frames. Furthermore, Shin et al. used backward co-reference resolution to caption generation from multiples frames, which means each segment of the video generates a caption, but at the end, a unique caption must describe the entire video content. To deal with that, the authors used first a gender annotator from Stanford CoreNLP [3]}, which assigns likely gender to tokens, then the co-reference resolution was employed with POS (Part of Speech) tagger and lemmatization strategies. Another issue to deal with is the appropriate transition in captions with connective words (e.g., “then” word); to solve that, they ran a CKY parser for PCFG trained on Penn Treebank [4]}. Later, the instances, not considering the connective words, are computed as 300-dimensional vectors using the Sentence2Vec [5]} approach. Ultimately, to do the caption generation, they perform matching of vectorization using the L2 distance.
The most outstanding feature of the proposal of Shin et al. is to exploit the content of the video by segments of different sizes, to look for action, not classifying them, but localizing them. The above involves a lot of specific and cumbersome (at least we think) tasks to training specifically. One may think, if there is no action in the video, what will be the result? In that case, they only employ the middle frame to generate the video caption.
As well as the solution proposed in the first place (see [6]}), Shin et al. tested their approach with three different datasets.
