We adopt MViT [1]} as the backbone, with weights initialized from Kinetics-400 pretraining [2]}. The token embedding stride is set as \(s_T=2\) , \(s_H=4\)  and \(s_W=4\)  and the embedding dimension is \(D=96\) . The encoder is composed of 16 self-attention layers that are divided into 4 blocks. The number of tokens is downsampled at the transition between two blocks. We build the decoder with 4 decoder blocks corresponding to the 4 blocks in the encoder. After getting raw output from decoder, softmax is applied on each frame with a temperature \(\tau \) . This can be formally written as \(\hat{p}_{ij} = \frac{\exp (\hat{y}_{ij} / \tau )}{\sum _{i,j}\exp ({\hat{y}_{ij} / \tau )}}\) 
where \(\hat{y}_{ij}\)  is the logit at location \((i,j)\)  from the model and \(\hat{p}_{ij}\)  is probability after softmax. In experiments, \(\tau \)  is empirically set as 2. We use KL-divergence loss to capture the difference between labels and predictions. More details of the training parameters can be found in the supplementary.
