In this paper we introduce spatial gating and channel fusion in spatial-temporal decomposition of 3D kernels. We implement this concept with gsf as shown in Fig. REF . gsf is lightweight and turns a 2D-CNN into a highly efficient spatio-temporal feature extractor. gsf first applies 2D convolution, then decomposes the output tensor using a learnable spatial gating into two tensors: a gated version of it, and its residual. The gated tensor goes through a 1D temporal convolution while its residual is skip-connected to its output. We implement spatial gating as group spatio-temporal convolution with single output plane per group. We use hard-coded time-shift of channel groups instead of learnable temporal convolution. Channel fusion is implemented by generating channel weights via a light weight 2D convolution. The two feature groups, gate-shifted and residual, are then fused by weighted averaging across the channels. With gsf plugged in, a 2D-CNN learns to adaptively route features through time and combine them, at almost no additional parameters and computational overhead. For example, when gsf is plugged into TSN [1]}, an absolute gain of \(+32\)  percentage points in accuracy is obtained on Something Something-V1 dataset with just \(0.48\%\)  additional parameters and \(0.58\%\)  additional flop.
