Multi-scale architectures. Inspired by the effectiveness of Two-stream architectures using RGB and optical flow images, [1]} [2]}, [3]} developed multi-stream architectures operating on two streams of RGB images. A multi-resolution cnn architecture is developed in [1]} where low resolution frames are processed by one stream and high resolution center crops are processed by the second stream. [3]} proposes to apply alternate frames of high resolution through a compact network in addition to a second stream processing low resolution images. This allows the network to encode features at multiple spatial scales and increase the number of processed frames while maintaining reasonable computational complexity. SlowFast [2]} encodes frames using a dual pathway network. One pathway encodes frames sampled at a slower frame rate while the second encodes frames sampled at a faster frame rate, enabling the network to capture information at multiple temporal scales. These approaches rely on multiple networks to encode information at different scales. To address this issue, TPN [7]} introduces a feature level pyramid network to encode features at multiple hierarchies thereby allowing the network to extract features at different temporal scales. Since the pyramid structure is constructed at feature level, TPN do not require multiple architectures, thereby reducing the number of parameters and computational complexity. MSTI-Net [8]} proposes to split the feature tensors across the channel dimension into four groups and apply convolution operation with different kernel size and aggregate them hierarchically. Varying the kernel size of the convolution operation enables the network to encode spatio-temporal features at multiple scales.
<FIGURE>