Kalogeiton et al. [1]} proposed action tubelet detector (ACT-detector) that takes as input a sequence of frames and outputs action categories and regressed tubelets, i.e., sequences of bounding boxes with associated scores. The tubelets are then linked to construct action tubes (sequence of bounding boxes of action). Gu et al. [2]} further demonstrate the importance of temporal information by using longer clips and taking advantage of I3D pre-trained on the large-scale video dataset [3]}. In order to generate action proposals, they extend 2D region proposals to 3D by replicating them over time, assuming that the spatial extent is fixed within a clip. However, this assumption would be violated for the action tubes with large spatial displacement over time, in particular when the clip is long or involves rapid movement of actors or camera. Thus, using long cuboids directly as action proposals is not optimal, since they introduce extra noise for action classification.
