In order to produce frame level predictions across a long untrimmed video many temporal modelling methods have been used. Most simply, a sliding window [1]}, often operating with a recurrent network [2]} or a feature buffer [3]}, [4]} to allow for arbitrary length temporal modelling in an online manner, can be utilised to make sequence predictions. Many approaches replace the relatively short term temporal modelling capabilities of recurrent models with temporal convolutions [5]}, improving action boundaries and minimising over segmentation errors via encoder/decoder methods [6]}, [7]} typically operating on precomputed features extracted by a pretrained 3D CNN model. In contrast, we show show through the use of surround sampling while training on in domain data, a simple temporally aware label cleaning method at inference can outperform more computationally expensive encoder/decoder methods operating on kinetics [8]} pretrained features, while performing real time action segmentation.
