We use an encoder block (a simple CNN) to transform the input video into a latent representation. A very common way for modeling 4-dimensional data such as videos is using 3D convolutional neural networks [1]}, [2]}, [3]}. This strategy allows for both spatial and temporal dimensions to be directly learned. However, 3D CNNs have a large number of parameters, requiring significantly more data and processing power to train. To address this drawback, an approach was proposed to factorize the 3D CNNs [4]}, breaking down the 3D convolutions into a 2D operation followed by a 1D operation. The resulting model with residual connections [5]} was called R(2+1)D [4]}. There are a couple of advantages to this model. First, the factorized operations contain a non-linear function in between the layers, resulting in more non-linearity in the network, which is a desired property.
Second, the factorization helps the optimizer during training by lowering the training loss and training time. It was also shown in [4]} that factorizing the 3D convolution into two distinct spatial and temporal operations improves the performance of learning in large scale video classification tasks. In our paper, we use this concept in the encoder block of the proposed network. This notion is depicted in Figure REF .
<TABLE><FIGURE>