Most existing methods formulate the WS-TAL as a video action recognition problem. Specifically, each input video is first transformed into snippet features, then the snippet-wise classifications are calculated to generate a temporal class activation sequence (T-CAS). Since only the video action labels are known, they aggregate scores from snippets with high probabilities of belonging to the action to form a video classification score.
There are two main strategies for this purpose: multi-instance learning (MIL) [1]} or attention mechanism [2]}.
The former strategy obtains video-level class scores from T-CAS by applying a pooling on the top-k values for each class. The later strategy introduces snippet attention weights to highlight action snippets and eliminate irrelevant background snippets. Both strategies end up with a classification loss using video labels. Despite of the efforts, the performances of these methods rely on the quality of the snippet-level predictions (including the attention weights). However, in absence of frame-level supervision, it is difficult for the WS-TAL models to obtain accurate snippet predictions.
