Fine-grained recognition can benefit from deeper temporal modeling. Full-3D CNNs (C3D [1]}, [2]}) process the video in space-time by
expanding the kernels of a 2D ConvNet along the temporal dimension. Deep C3Ds are designed to learn powerful representations in the joint spatio-temporal feature space with more parameters (3D kernels) and computations (kernels slide over 3 densely sampled dimensions). In practice however, they may under-perform due to the lack of sufficiently large datasets for training them at scale. To cope with these issues arising from the curse of dimension one can narrow down network capacity by design. Fig. REF  shows several C3D kernel decomposition approaches proposed for spatio-temporal feature learning in video. A most intuitive approach is to factorize 3D spatio-temporal kernels into 2D spatial plus 1D temporal,
resulting in a structural decomposition that disentangles spatial from temporal interactions (P3D [3]}, R(2+1)D [4]}, S3D [5]}). An alternative design is separating channel interactions and spatio-temporal interactions via group convolution (CSN [6]}), or modeling both spatial and spatio-temporal interactions in parallel with 2D and 3D convolution on separated channel groups (GST [7]}). Temporal convolution can be constrained to hard-coded time-shifts that move some of the channels forward in time or backward (TSM [8]}). All these existing approaches learn structured kernels with a hard-wired connectivity and propagation pattern across the network. There is no data dependent decision taken at any point in the network to route features selectively through different branches, for example, group-and-shuffle patterns are fixed by design and learning how to shuffle is combinatorial complexity.
