After detecting the action regions in the frames, some methods [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]} use optical flow to capture motion cues. They employ linking algorithms to connect the frame-level bounding boxes into spatio-temporal action tubes. Gkioxari et al. [1]} used dynamic programming approach to link the resulting per-frame detection. The cost function of the dynamic programming is based on detection scores of the boxes and overlap between detection of consecutive frames. Weinzaepfel et al. [6]} replaced the linking algorithm by a tracking-by-detection method. Then, two-stream Faster R-CNN was introduced by [3]}, [5]}. Saha et al. [5]} fuse the scores of both streams based on overlap between the appearance and the motion. Peng et al. [3]} combine proposals extracted from the two streams and then classify and regress them with fused RGB and multi-frame optical flow features. They also use multiple regions inside each action proposal and then link the detection across a video based on spatial overlap and classification score.
