At training time, we randomly sample 8 frames from each video with a sampling interval of 8 as input (i.e. selecting 8 frames from a 72-frame window with equal spacing). All videos are spatially downsampled to 256 in height while keeping the original aspect ratio. We further implement multiple data augmentations including random flipping, shifting, and resizing. We then randomly crop each frame to get an input with dimensions \(8\times 256\times 256\) . The output from the decoder is a downsampled heatmap with dimension \(8\times 56\times 56\) . For visualization, the output heatmap is upsampled to match the input size by trilinear interpolation. At inference time, the input clip is center-cropped. For gaze labels, we generate a gaussian kernel centered at the gaze location in each input frame with a kernel size of 19 following [1]}. We use a uniform distribution for frames where gaze is not tracked in training and only calculate metrics on frames with fixated gaze in testing as in [2]}. For the EGTEA Gaze+ [2]} dataset, we determine which frames to calculate metrics on by using the provided label of gaze fixations and saccades. On the Ego4D [4]} dataset, no label of gaze type is available. We calculate the euclidean spatial distance of gaze between adjacent frames and consider the tracked gaze to be a saccade if the distance is above a threshold, and treat it as fixation otherwise. We adopt an empirical threshold of 40.
