RNNs are used for sequence modeling and are capable of capturing long-term dependencies in videos. Buch et al. in Sst [1]} and SS-TAD [2]} used RNNs for action detection. They partition the video into non-overlapping equal-length segments and feed each segment to a visual encoder for feature extraction. At time \(t\) , visual feature \(f_t\)  and the hidden state of the previous time step (\(h_{t-1}\) ) are fed to a Gated Recurrent Unit (GRU)-based architecture to produce hidden state \(h_t\) . This hidden state is then fed to fully connected layers to evaluate multi-scale proposals at time \(t\)  by producing actionness scores (def REF ). In an earlier work, Yuan et al. in PSDF [3]} captured the motion information over multiple resolutions and utilized RNNs to improve inter-frame consistency. Yeung et al. learn decision policies for an RNN-based agent [4]}, and later proposed an LSTM model to process multiple input frames with temporal attention mechanism [5]}. LSTMs are also used in other frameworks such as [6]}, [7]}, [8]} to evaluate temporal proposals. The advantage of using RNNs is that hidden state at time \(t\)  encodes the information from previous time steps which is useful to capture temporal dependencies. However, RNNs are not capable to encode very long videos as the hidden vector gets saturated after some time steps.
<FIGURE>