To produce the gaze distribution with the desired spatio-temporal resolution, we adopt a decoder to upsample the encoded features. We utilize a transformer decoder based on the multiscale self-attention block of MViT [1]}. Suppose each decoder layer takes visual features \({X}_d\in \mathbb {R}^{T^{\prime }H^{\prime }W^{\prime }\times D^{\prime }}\)  as inputs and the corresponding query, key and value matrices are \({Q}_d^{T^{\prime }H^{\prime }W^{\prime }\times D^{\prime }}\) , \({K}_d^{T^{\prime }H^{\prime }W^{\prime }\times D^{\prime }}\)  and \({V}_d^{T^{\prime }H^{\prime }W^{\prime }\times D^{\prime }}\) . As shown in Fig. REF (d), we replace the original pooling operation for the query matrix with an upsampling operation implemented with trilinear interpolation and keep the pooling for the key and value matrices. Following [1]}, \(\widehat{{Q}}_d\)  is obtained by applying a deconvolutional operation on \({Q}_d\) , while \(\widehat{{K}}_d\)  and \(\widehat{{V}}_d\)  are obtained by applying convolutional operations on \({K}_d\)  and \({V}_d\) . Then, the output of self-attention is calculated in the same way as Eq. REF . In addition, we keep the skip connection in the self-attention layers and replace the pooling operation in skip connections with trilinear interpolation, which produces the upsampled output with dimension \(\widehat{T}\widehat{H}\widehat{W}\times D^{\prime }\) . Our decoder is composed of 4 decoding blocks. Skip connections are used to combine intermediate features of the encoder with corresponding decoder features. Finally, another linear mapping function is used to output the final gaze prediction.
