Based on these pioneering studies, three major types of deep learning methods are currently utilized to process spatio-temporal inputs: convolution neural network (CNN) based methods, RNN based methods and two-stream based methods.
The first type in general extends the 2-dimensional CNN architecture to a 3-dimensional counterpart by which the convolution filters are extended to filter 3-dimensional inputs,
such as C3D [1]}, [2]}, P3D [3]} and I3D [4]}.
By considering the input as a 2-dimensional spatial sequence, the second type aims to model the temporal structure with RNNs [5]}, [6]}, [7]}, which have been widely used to model sequential patterns.
In particular, LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) networks are proposed to address the gradient vanishing and exploding issues of the original RNN by using gate mechanism [8]}, [9]}. GRU involves less computations than LSTM while achieving comparable performance to LSTM, thus improving the efficiency of RNNs. Moreover, GRU has demonstrated better performance on smaller datasets [10]}.
The last type which is based on the pioneering two-stream approach represents video content with both appearance and motion features [11]}.
