Recur, Attend or Convolve? On Whether Temporal Modeling Matters for Cross-Domain Robustness in Action Recognition

Most action recognition models today are highly parameterized, and evaluated on datasets with appearance-wise distinct classes. It has also been shown that 2D Convolutional Neural Networks (CNNs) tend to be biased toward texture rather than shape in still image recognition tasks, in contrast to humans. Taken together, this raises suspicion that large video models partly learn spurious spatial texture correlations rather than to track relevant shapes over time to infer generalizable semantics from their movement. A natural way to avoid parameter explosion when learning visual patterns over time is to make use of recurrence. Biological vision consists of abundant recurrent circuitry, and is superior to computer vision in terms of domain shift generalization. In this article, we empirically study whether the choice of low-level temporal modeling has consequences for texture bias and cross-domain robustness. In order to enable a light-weight and systematic assessment of the ability to capture temporal structure, not revealed from single frames, we provide the Temporal Shape (TS) dataset, as well as modified domains of Diving48 allowing for the investigation of spatial texture bias in video models. The combined results of our experiments indicate that sound physical inductive bias such as recurrence in temporal modeling may be advantageous when robustness to domain shift is important for the task.


Introduction
One of the most fundamental questions when it comes to video understanding is how to model the dependency between frames in such a way that temporal relationships relevant to the activity in the video can be learned. A robust action recognition system should be able to figure out how frames relate to each other, and which shapes and objects have changed or persisted over time. With this knowledge, it can start to infer relationships at a higher level, such as object-object or agent-object relationships.
Three principally different approaches to frame dependency are 3D convolutions, self-attention and recurrence.
These methods model the world (the visual sequence) in principally different manners: linearly, non-linearly, and non-linearly with a time-causal direction, meaning that they each use different inductive biases for the temporal modeling (Fig. A). In spite of its essentiality, the frame dependency question has almost disappeared from action recognition articles, possibly in the race to improve on the classification benchmarks. Emphasis is instead placed on other aspects of deep video models, such as advanced architectural superstructures, regularization or training schemes. A shift toward attention-based video models has recently taken place, but without a discussion of the physical interpretation of its underlying temporal model.
Humans are still significantly stronger at generalization than artificial neural networks in vision tasks [20,46]. Recurrent models are critical in the only visual system that has been 'solved' to date -biological vision [1,10,12,13,15,32,34,44,52]. Based on the observation that feedback connections are abundant in biological but not computer vision [32,56], in this article, we hypothesize that the lack of recurrence when learning spatiotemporal features may be one reason for this discrepancy. We therefore investigate the following research question empirically in extensive and systematic experiments: does the principally different mathematical natures of 3D convolutions, self-attention and recurrence affect cross-domain robustness in video modelsand in particular, does recurrence bring about an advantage?
Video models lack robustness to domain shift [8,61,62], and it has been repeatedly shown [8,25,35,60] that the datasets most frequently cited during the 2010s (UCF-101 [51], HMDB [33], Kinetics [29]) exhibit significant spatial biases. This is a plausible reason for the poor cross-domain robustness in action recognition, since overly relying on spatial rather than motion cues intuitively results in overfitting to one domain (e.g., certain backgrounds, viewpoints or similar actor appearances).
Contemporary state-of-the-art approaches to action recognition are predominantly either fully convolutional [5,17,18,21,60], combine convolutions with temporal sampling and fusion strategies [58,59,63], or, more recently, attention-based Video Transformers (VTs) [4, 23, Less texture More texture S1 S2 T Figure 1. Animated figure, displayed on click in Adobe Reader. Example clip showing our three modified domains of Diving48 enabling the investigation of texture bias in video models: S1 (segmented divers over a blurred background), S2, (cropped bounding boxes of divers over a blurred background), and T (masked boxes of divers, and the original background). 31,45,53]. The sheer size of the models, typically more than 50M trainable parameters, gives them a strong capacity to learn in-domain patterns. As models grow larger, ever more resources are spent to train them. State-of-the-art models should display competitive benchmarking numbers on large-scale datasets, such as Kinetics-400 and Kinetics-600. It is questionable whether these benchmarks are suitable for temporal modeling, or rather for how large amounts of YouTube clips efficiently can be stored as weight representations. At the same time, the reciprocal dependency between the hardware and software of standard graphics processing units (GPUs), on the one hand, and models requiring massive parallel computation for their training, on the other hand, is becoming ever more intertwined [30,41]. The question looms whether we have cornered ourselves in action recognition, in the expectancy to work on ever larger models, in industry as well as in academia. Theoretical works [2,39,50] have indicated that overparametrization helps generalization, in that local minima of the loss landscape for such models often are thought to be global. These studies are made on held-out data, but never on data with significant domain shift, to the best of our knowledge.
Although less efficient to train on GPUs, recurrent video models have a more parameter-efficient approach per timestep, which may hinder over-reliance on texture cues, and promote learning the temporally relevant motion cues. The need to be economical with the use of trainable parameters, we hypothesize, creates incitement to learn better shape representations instead of texture representations. In turn, this allows for better generalization across datasets and in the wild. For contour detection, it was found that a model with recurrent dynamics was more sample-efficient and generalized better than a feed-forward model [37,38].
The primary contributions of our paper are as follows: • We present the first empirical results from systematic experiments on how the choice of frame dependency modeling in action recognition can affect crossdomain robustness.
• We introduce a lightweight dataset allowing for investigation of both temporal shape modeling ability and domain generalization, called the Temporal Shape dataset.
• We provide the first discussion and experiments on shape vs. texture bias (following Geirhos et al. [19]) in deep video models.
• We make segmentation-based shape and texture versions of the Diving48 dataset public (as well as 303 instance-segmented frames), allowing studies on whether a video model has learned to rely more on (temporal) shape or on texture.

Related Work
Domain shift in action recognition. In [7,61], crossdomain datasets are introduced to study methods for video domain adaptation. [7] proposes to align the temporal features where the domain shift is most notable, whereas [61] proposes to improve the generalizability of so-called local features instead of global features, and use a novel augmentation scheme. Strikingly, however, all experiments in [7,61] are based on features extracted frame-by-frame, by a 2D ResNet [26], and aggregated after-the-fact, meaning that they in effect do not handle spatiotemporal features. Using frame-wise features saves large amounts of time and computation, but it avoids an essential aspect of video modeling. Different from the field of Domain adaptation, we are not proposing methods on top of base architectures to reduce domain shift, but rather study empirically which types of fundamental video models inherently seem to be more robust to it. In an important work by Yi et al. [62], benchmarks are introduced to study robustness against common video corruptions, evaluated for spatiotemporal attentionand convolution-based models. Different from our work, the domain shift is restricted to data corruptions rather than the same classification task in a new domain, and recurrent models are not evaluated.
Emphasis on temporality in action recognition. Many works emphasize the importance of temporal modeling, as the field of video understanding is growing, e.g., [14,22,40,43,47,49,60,63]. [22] and [40] compare temporal model-ing abilities between principally different architectures, but without explicitly investigating domain shift generalization. [49] examines video architectures and datasets for human activity understanding on a number of qualitative attributes such as pose variability, brevity and density of the actions. [28] investigates how much the motion contributes to the classification performance of the C3D architecture [54]. Both [6] and [55] perform large-scale studies of the features of different variants of 2D and 3D CNNs in action recognition. Last, we are connected to [47], which discusses the risk that models with strong image modeling abilities may prioritize those cues over the temporal modeling cues. Reminiscent of the findings of [19], the authors of [47] find that inflated convolutions tend to learn classes better where motion is less important, and that generalization can be helped by training on more temporally focused data (in analogy to training on shape-based data in [19]). Different from our work, however, only fully convolutional models are studied and the focus is not on comparing models with fundamentally different approaches to frame dependency modeling.

Experiment design
In this section, we describe the experiment design for the two datasets: Temporal Shape and Diving48.
Main idea. In all experiments, we begin by training on a specific domain, and validating on a held-out dataset from the same domain. We save the model checkpoint which performed the best on the validation set, and then proceed to evaluate it on other domains that are different in some respects but share the same task. Following [7], the domain we train on will be referred to as the source, and the unseen domains that we evaluate on as the target. To measure crossdomain robustness, we define the robustness ratio (rr.) as the ratio between a model's accuracy on a target domain and its best validation accuracy on the source domain. When the target task corresponds to the source task, this number should ideally be close to one (higher is better). It can be noted that the rr. is a heuristic metric, which builds on the assumption that the performance on the in-domain validation set typically is higher than on other domains. If the performance on the validation set is poor to begin with, the rr. is less informative.
Method common to all experiments. In our study, we are purposefully comparing the basic functionality of models. No pre-training, dropout, or data augmentation is applied in our experiments, except for 50% chance of horizontal flipping of the clips on Diving48. Sequences are uniformly sub-sampled into equal length (a fixed input size is required for the input to both 3D CNNs and attention-based models). There are non-uniform frame sampling methods, which can be used as augmentation, or as informed priors (e.g., the TimeSformer only samples the middle frames during inference in [4]); these are thus not used in our study, in order to study the bare bones of the models. Code related to neural networks was written in PyTorch [42] using Lightning [16]. Further implementation details and code can be found in the corresponding repositories ( Temporal Shape experiments, Diving48 experiments and diver segmentation). The datasets are available for download on Harvard Dataverse and linked to from the repositories.

Models
We will compare ConvLSTMs, 3D CNNs and VTs, since these present three principally different temporal modeling approaches with varying types and degrees of inductive bias. As VT, we will use the TimeSformer [4], because it recently achieved state-of-the-art results on a number of action recognition benchmarks.
It is a challenging task to compare neural network models which have principally different architectures. In our work, we decided on controlling for three different factors: the performance on a particular dataset, the number of trainable parameter and the layer structure (i.e., the number and expressivity of hierarchical abstractions). The experiments were designed prior to running them, to keep the process as unbiased as possible. The experiments are further completely reproducible as they were run on five fixed random seeds throughout the study.
Convolutional LSTMs. The ConvLSTM [48] layer functions like an LSTM layer [27], but with matrix multiplication replaced with 2D convolutions. This crucially means that they allow for the input to maintain its spatial structure, contrary to classical recurrent layers which require a flattened input. Frame dependency is modeled using recurrence, which introduces non-linearities between timesteps. Further, time can only flow in the causal direction. A Con-vLSTM video model, in this work, is a model fully based on these types of layers, with a classification head on top.
TimeSformer. The TimeSformer (hereon, TimeSf) [4] is a VT, relying entirely on self-attention mechanisms to model frame dependency. As in [11], each frame is first divided into patches, which are flattened. We use the TimeSformer-PyTorch library [57], mainly with the standard settings unless otherwise specified (divided space-time attention). Self-attention is applied both among the patches of one frame (spatial attention) and across patches located in the same positions across the temporal axis (temporal attention). Two variants are used in the TS experiments, with the number of heads A set to either 1 or 8 (TimeSf-1 and TimeSf-8). TimeSf-1 is closer to the ConvLSTM and 3D CNN in terms of parameter count, whereas TimeSf-8 is the standard setting. We note again that in order to study the fundamental behavior of the models, we do not use pretraining, advanced data augmentation, nor averaging over multiple predictions. This results in a lower performance on Diving48 for TimeSf than its state-of-the-art results. In order to control for layer structure or number of parameters which requires architectural modifications, it is not possible to use a pre-trained checkpoint. It is well-known that VTs, or Vision Transformers (ViTs) in general, require a lot of training data due to their minimal inductive bias. We therefore stress that we are not questioning the overall performance of these models -a pre-trained version would have performed better on the Diving48 task than in our experiments, but we are investigating the fundamental behavior of models in our experiments, prior to more advanced or large-scale training schemes.
3D CNNs. In a 3D CNN, time is treated as space, and thus the input video as a volume, across which we convolve local filter volumes. Convolution is a linear operation, meaning that the order of frames that the 3D filter traverses does not matter. Instead, all non-linearities are applied hierarchically, between layers, which is how this model still can learn the arrow of time. Its layer structure is typically similar to a 2D CNN, including batch normalization and pooling. This is also the case for the instances used in our study.

Experiments on the Temporal Shape dataset
Our proposed TS dataset is a synthetically created dataset for classification of short clips showing either a square dot or a random MNIST digit tracing shapes with their trajectories over time (Fig. 2). The dataset has five different trajectory classes (i.e., temporal shapes): circle, line, arc, spiral and rectangle. The task is to recognize which class was drawn by the moving entity across the frames of the sequence. The spatial appearance of the moving object is not correlated with the class, and can thus not be employed in the recognition. In the 2Dot, 5Dot and MNIST domains, the background is black, and in MNIST-bg, the background contains white Perlin noise. The Perlin noise can be more or less fine-grained; scale is regulated by a random parameter σ ∈ [1,10]. The dataset can be thought of as a heavily scaled-down version of an action template dataset, such as 20BN-Something-something-v2 [24], stripped of appearance cues.
The sequences consist of 20 64x64 frames, in grey scale. Each of the five classes has different amounts of possible variation in their states. The shapes can have varying starting positions, starting angles, direction, size and speeds. In the experiments, 4000 clips were used for training and 1000 for validation (model selection), and 500 clips for evaluation only. The classes are sampled so as to obtain class balance.
Since the dataset is small, we use lightweight models. We control for layer structure by letting the compared mod-els have three layers each of analogous blocks with the same number of hidden units in each. One block for the ConvL-STM and 3D CNN consists of a model-specific layer, max pooling, followed by batch normalization. These two models used the same convolutional kernel sizes in all three layers (3×3). For the TimeSformer, we used one TimeSformer layer as one block, and the latent dimension for each attention head, D h , as the number of hidden units, since these were similar in scale.
We run experiments for different numbers of hidden units per layer, h ∈ {2, 4, 6, 8, 10, 12, 16, 24, 32, 48}. For each of the ten experiments of varying model sizes, we train the models on the five-class task on the source domain for 100 epochs, with ten epochs of early stopping patience, repeated under five different random seeds set from the beginning of the study. For TimeSf-1 and TimeSf-8, the maximum number of epochs is 300 (100 for early stopping) because they demand more epochs to converge than the other two types of models, due to their minimal inductive bias. We then evaluate the best model checkpoint from the source domain on different target domains with the same classification task. Experiments were conducted in two 'directions', training on 2Dot and evaluating on the other domains, or training on MNIST-bg and evaluating on the other domains, since these represent two extremes on the continuum of less to more spatial noise.
Training on the TS data is light-weight compared to real video data, and runs fast (in the minutes-range, up to an hour, for the model sizes we evaluated) on one GPU card. We train with a batch size of 64 in all TS experiments.

Experiments on Diving48
Diving48 [35] is a well-known dataset for fine-grained and time-critical action recognition. It consists of 18k short clips with dives from 48 classes. Successfully classifying these dives requires temporal modeling, since one needs to keep track of the movements of the divers and their order. The dataset is varied appearance-wise, in terms of both backgrounds and viewpoints, which may contain unknown biases. The same competition sites can be present in both the training and test split, "to avoid biases for certain competitions", according to the authors [35]. Instead, in our view, this in fact increases the risk for bias, since the ability to recognize a dive at an unseen site is never tested. It would have been preferable to separate competition locations entirely between training and test set. Thus, even though the dataset presents a very challenging classification task from a temporal modeling perspective, it is likely not free from spatial biases (as will be demonstrated by our experiments). Modified domains of Diving48. We always train on the original dataset, but evaluate our trained models on slightly modified domains of the original test set. We modify the test set into three new domains: two based on shape and 2Dot 5Dot MNIST MNIST-bg Figure 2. The videos can be displayed on click in Adobe Reader. Example clip showing the four domains of the TS dataset, for the class circle. In 2Dot and 5Dot, the circle is drawn with a square of width 2 and 5 pixels. In MNIST and MNIST-bg, the circle is drawn with a MNIST digit, w/ and w/o a Perlin noise background.
To do this, we extend the concepts of shape and texture bias from [19] to the temporal domain in the following way. In the shape domains, we blur the background and only maintain the segmented diver(s) (S1), or the divers and their bounding boxes (S2). In the texture domain (T), we conversely mask bounding boxes where the diver(s) are in each frame, and only keep the background. The masked boxes are filled with the average Imagenet [9] pixel value, following [8]. The class evidence should lie only in the divers' movement; hence, the texture version should not contain any relevant signal, and the accuracy should ideally drop to random performance. In this way, we can study how different models drop in score when tested on the shape or texture domain, indicating both cross-domain robustness (for S1 and S2) and texture bias (for T).
Instance Segmentation of Diving48. The segmentation of divers in Diving48 is detailed in the supplemental. We release 303 manually labeled frames with instance segmentation (single or double dives), since off-the-shelf COCOtrained [36] networks fail at this task for the class Person, presumably because of the unusual shapes assumed in the air by the diver, or include people in the audience.
Training. Just as for TS, we deliberately avoid bells and whistles when training models on Diving48, to study their fundamental behavior. All three models are trained with the same SGD optimizer, cross-entropy loss, and a constant learning rate of 0.001. Each model is trained for 500 epochs maximally, with an early stopping patience of 30 epochs if the validation performance does not improve. The only data augmentation used is horizontal flipping of 50% probability for the entire clip. The models are trained using PyTorch Lightning's ddp parallelization scheme across eight A100 GPUs, with a batch size of 8 and a clip length of 32 uniformly sampled frames, at 224×224. Given that the purpose of our experiments is not to optimize classification performance, we evaluate the models at different levels of performance, ranging from 30% to 50% accuracy. Some of the advanced state-of-the-art methods today, including pre-training and heavy data augmentation, obtain up to 80% performance on Diving48, but when the dataset was introduced in 2018, and standard video methods were tested off-the-shelf on it, the best result was 27% ac-curacy [35]. Thus, the range of 30-50% is reasonably wellperforming, and well above random (which is at 2.1%).
Experiments. We conduct three different kinds of experiments on Diving48, namely control for: layer structure and performance (a-c), performance of the best performing variants (d), and number of parameters and performance (e-h). ConvLSTM has four blocks of 128 hidden ConvL-STM units each (14.3M params.) in all experiments.
a-c. Controlling for layer structure and performance. In this experiment, we let the models have four layers, with h = 128 in each. We again treat D h as the hidden unit analogy for TimeSf. We evaluate model checkpoints at different performance levels: 30%, 35%, and 38.3% accuracy. The last accuracy, was chosen because it was the limiting, highest performance by the 3D CNN in this experiment. Having the same layer structure gives rise to a varying number of parameters for each type of model. Here, the 3D CNN has 10.6M params., and TimeSf 85M.
d. Controlling for performance only. Here, we compare models at their best performance, after hyperparameter search. Since it was not possible to train TimeSf to a higher accuracy than 39.7% in all variants we tried, this experiment was only conducted with the 3D CNN and ConvLSTM. 1 The 3D CNN was an 11-layer VGG-style model (23.3M params.). The checkpoints used were both at exactly 50.07% validation accuracy. e-h. Controlling for number of parameters and performance. Here, we have chosen models with a similar amount of trainable parameters, in this case 14M. To arrive at this number of parameters for TimeSf, its depth was reduced from 12 to 11, and D h and D were halved, to 32 and 256, respectively, relative to the default model. The 3D CNN has six blocks with 128 units in each.

Results and discussion
Having presented the experimental design for both datasets, next, we discuss our empirical findings, first on TS, and then on Diving48 and its modified domains.

Temporal Shape
Condensed results: TimeSf and ConvLSTM are more cross-domain robust than the 3D CNN in the absence of spatial texture bias. Training on 2Dot. Fig. 3a shows that although the 3D CNN generally obtains higher results on the source validation set and the nearby 5Dot domain, the ConvLSTM and TimeSf drop less compared to their original results when tested on MNIST (further from the source domain). ConvL-STM in fact outperforms the 3D CNN in absolute numbers on the MNIST domain. The inductive bias of a 3D CNN is highly local in space and time, which might impede learning of these temporal shapes. Generalization to the MNIST-bg domain proves too challenging for all three models.
Robustness ratio vs. model size. In Fig. 10, we have plotted the rr. for the three target domains when training on 2Dot. For 5Dot, the rr. for ConvLSTM decreases slightly with model size, whereas the 3D CNN and TimeSf, in contrast, increase the rr. with increased model size. For MNIST, which is further from the validation domain, the upward trend for the 3D CNN is broken, and less pronounced for TimeSf. For the most challenging domain, MNIST-bg, the rr. becomes very low for all three models with increased size. The trends in Figs. 10 a-c point to how a larger model size with promising performance in a nearby domain can potentially be an obstacle in domains that are further from the source for TimeSf and the 3D CNN.
Training on MNIST-bg. In this experiment, TimeSf-8 and TimeSf-1 were the most robust (Fig. 3 b). A VT is an excellent model when it comes to learning sparse, long-term dependencies in space and time. We hypothesize that this allowed TimeSf to to fully disregard the Perlin noise (which is highly stochastic and demanding to model) and learn the true temporal shapes, and that this, in turn, allowed it to be unbiased in the other domains, since the training data was designed to exclude spatial bias. In real-world data, however, there will always be biases, and it is therefore best to construct models which inherently encode as little bias as possible, regardless of the training data.

Diving48: Sensitivity to shape and texture
Condensed results: ConvLSTM exhibits less texture bias and is more cross-domain robust than TimeSf and 3D CNN.  Table 1 shows the average results for the Diving48 experiments. We note that ConvLSTM drops the most for T, both relative to the validation (T/V) and to the S1 (T/S1) accuracies. ConvLSTM is also most robust to the S2 domain, whereas the 3D CNN is most robust to the S1 domain. Experiments a-d. In experiments a-c (Fig. 5), where we vary the validation accuracy on the source domain between 30% and 38.3%, both TimeSf and the 3D CNN perform better on T than on S1 and S2, even if only the two latter contain class evidence. This suggests that spatial bias is indeed present in Diving48, and that these models are more prone to encode it than ConvLSTM. Tables 2-4 show that T/S1 > 1 for these two models, also visible in Fig. 5 a-c. 2  Table 3. Results for experiment b: 4x128, 35% validation accuracy. * when a low T/V result is not accompanied by T/S1 < 1.
Model  Table 4. Results for experiment c: 4x128, 38.3% validation accuracy. * when a low T/V result is not accompanied by T/S1 < 1.
In contrast, ConvLSTM clearly drops for T. TimeSf is large here, at 85M params., whereas the 3D CNN is interestingly quite small at 10.6M params. This suggests that 2 In Tables 3-4, TimeSf drops the most for T relative to the validation set (T/V). This can be explained by its overall large drops, rather than being robust to texture bias, most clearly visible in Fig. 5 a-c. For T/V to be a meaningful metric, T/S1 should be < 1. Therefore, we have put asterisks on the lowest T/V results which are not accompanied by T/S1 < 1. not only the parameter count causes susceptibility for overfitting, but that there may be innate tendencies to overfitting in the choice of spatiotemporal modeling. A recurrent model necessarily takes each timestep into account as it traverses the sequence in the time-causal direction, since each timestep is non-linearly registered in the hidden state. We hypothesize that this enables it to register motion changes over time more in detail, and count these as salient, when that is the case (as it should be for Diving48).
Model  In experiment d, where we compare a ConvLSTM and a 3D CNN at 50.07% validation accuracy -the best results on Diving48, the 3D CNN does not longer improve on the texture dataset relative to S1 and S2, but the drop on T is S1 S2 T markedly larger for ConvLSTM (Table 5).
Qualitative examples and diving attributes. Table 6 shows a breakdown of the models' predictions on five randomly selected clips from a randomly chosen class (34). The models instances used here are from experiment c (38.3% acc.). Top-1 acc. for these five clips being equal for all models at 0.4, we note that ConvLSTM has 100% top-5 acc. for both S1 and S2, whereas the 3D CNN has 80% and 60% (40% and 40% for TimeSf). As for the texture (T) results, the top-5 acc. of the 3D CNN remains at 80% relative to S1 and even improves from 60% to 80% relative to S2, whereas ConvLSTM drops by 40% and TimeSf drops by 50%. Thus, so far ConvLSTM and TimeSf display sound dropping on T. Next, we study the predictions made by the models in detail to observe that there is a qualitative difference between the predictions of ConvLSTM and TimeSf. Each label of Diving48 has four attributes: takeoff, somersault, twist and flight position. Among the top-1 predictions for both S1 and S2 (Table 7), we study how many attributes are correct in the misclassifications for each model.  (32,35) obtain three correct attributes, and for TimeSf, the best misclassification has only two correct attributes. This suggests that the 3D CNN and TimeSf have modeled the classes in terms of the true attributes to a lesser extent than ConvLSTM, i.e., ConvLSTM has learned more relevant temporal patterns, at the same global validation performance. Observing the three lower sections of Table  7 for further randomly selected classes 12, 22 and 45, the ConvLSTM still achieves the largest proportion of correct attributes in the misclassifications. Just as for class 34, the  Table 7. Examples of predictions and misclassifications. Each class has four attributes, and the Correct attr. column shows how many attributes were correct among the set of misclassifications.
Bv. S1 S2 T Experiments e-h. The results for experiments e-h, where the number of trainable parameters and performance are fixed, are shown in Fig. 6 (tabulated results in the supplemental). Here, the 3D CNN is the most robust out of the three, although ConvLSTM approaches the 3D CNN and drops more steeply for T in g-h, where the performance is higher (40% and 45% acc.). In these experiments, although least robust, TimeSf does not improve on T relative to S1 and S2 any more. This suggests that TimeSf is more likely to display texture bias when it has a larger amount of parameters, as it does in experiments a-c.

Conclusions and discussion
We have studied cross-domain robustness for three models that are principally different in terms of temporal modeling, in their bare-bones settings. A 3D CNN treats frames as a linear volume, a VT lets frames have non-linear but timesymmetric relationships, and a ConvLSTM models frame dependency non-linearly in a time-causal direction. Recently, a discrepancy in terms of feedback connections between biological and computer vision has been discussed [32,56], and in our work we have hypothesized that the lack of feedback connections is one reason for the similarly lacking generalization abilities in computer vision.
Our experiments were carried out on two very different datasets, one synthetic, without bias, and one with natural data, thus with more noise and potential spatial bias. The combined results (Figs. 3-5, Tables 1 and 7) on these datasets indicated that convolutional-recurrent temporal modeling is more robust to domain shift than selfattention and 3D convolutions in terms of bare-bones behavior, presumably owing to its lesser encoding of texture bias. Our results are fully reproducible with public seeds, code and data. The fact that our observations regarding texture bias are made for a fine-grained dataset such as Div-ing48, constructed to contain as little bias as possible, suggests that the issue may be worse when it comes to more spatially biased datasets such as Kinetics, which is left for future work. It is furthermore left for future work whether ImageNet pre-trained VTs display more or less texture bias than their trained-from-scratch counterparts. Another observation from our study is that when the parameter count was kept equal (experiments e-h), these trends were less pronounced.
Moreover, qualitative random examples consistently showed that the ConvLSTM learned more relevant diving patterns than the two others, when scrutinizing the three models' misclassifications -which emphasizes the texture bias tendency of TimeSf and the 3D CNN. Sharing parameters across timesteps, as recurrent models do, narrows the parameter space, possibly incentivizing these models to prioritize which patterns to learn. Another reason to use smaller models is that they require less data to train, which is ethically desirable, both in that the data can be inspected more easily, and from a sustainability perspective [3].
Our study indicates that sound physical inductive bias such as recurrence in temporal modeling may be advantageous when robustness to domain shift is important for the task. In action recognition, benchmarking has thus far mainly been conducted for in-domain-tasks where large models perform well. We encourage the video understanding community to increasingly conduct evaluation on tasks involving domain shift. We hope that our proposed datasets and framework for evaluation can help such future domain shift robustness investigations of spatiotemporal features. Figure A highlights the conceptual differences between 3D convolution, self-attention and recurrence in terms of temporal modeling.

B. Plots for each model size on the Temporal Shape dataset
In the main article (Figure 3), the shaded area of standard error is across both model sizes and repeated runs with different seeds (meaning 10 * 5 = 50 runs per model and domain). Detailed plots for each model size with five repeated runs each are shown in Figures 8-9.  Table 9. Results for experiment (f): 14M parameters, 35% validation accuracy. Model

C. Robustness ratios for training both on 2Dot and MNIST-bg
In the main article, robustness ratios vs. model size are only plotted when training on 2Dot. In Figure 10, we include results when training on MNIST-bg as well. We show the two plots next to each other for comparison. Tables 8-11 show tabularized results corresponding to Figure 6 in the main article (experiments e-h).

E. Qualitative examples on Diving48
Here, we include the top-1 and top-5 accuracies tables corresponding to the qualitative examples of classes 12, 22 and 45 shown in Table 7 in the main article. In Tables 12,  13 and 14, the trends regarding the top-1 and top-5 accuracy on the different datasets are slightly less clear. We observe that in Tables 12 and 14, ConvLSTM and TimeSf drop the clearest in top-5 performance on T relative to S1 and S2. On the other hand, in Table 13 (Class 22), the top-5 accuracy is relatively improved on T compared to S1 and S2 for ConvL-STM and the 3D CNN, whereas TimeSf is unchanged. We inspected these clips, to verify that the segmentation had not failed, which it had not. However, the ConvLSTM is still the only one out of the three to have 20% in top-1 accuracy both for S1 and S2 on class 22, dropping to 0 in top-1 on T (Table 13). Last, for class 45, the ConvLSTM has the best results on S1 and S2 (20% top-5 accuracy) out of the three   table structure as Table 12.  In the experiments, 4000 clips were used for training and 1000 for validation. The number of samples was chosen so as to be able to sample randomly with replacement, while still keeping the risk low that an identical clip occurs in both the training and the validation set. For the 2Dotdomain, each class has more than 30k possible variations (lower bounds: 31k circle, 34k line, 51k rectangle, 150k arc), except the spiral class which has 7200 as a lower bound on the possible variations. When the training set consists of 5000 samples in total, we generate around 1000 samples per class. For the spiral class, a frequentist estimation gives that 800/7200 = 0.11 of the 200 spiral validation samples might be present in the training split (22 clips). However, this is still an over-estimation, since the spirals sometimes bounce against the sides of the frame which gives rise to extra variation. We decided to consider this as acceptable noise of the dataset. Some amount of data leakage can be considered interesting since this may occur in standard datasets as well.

F.2. Instance Segmentation of Diving48
To segment divers, it did not suffice to apply a pretrained network and use the class "Person", which we first attempted (DeeplabV3 pre-trained on MS-COCO, provided by PyTorch). First of all, the off-the-shelf model could often not recognize the divers in the air as the "Person" class -they can be up-side down, or assume strange shapes in the air. Secondly, the model would often detect pixels of the "Person" class in the audience, when there was audience visible, which we, naturally, did not want to include.
Thus, we resorted to labelling our own segmented frames from the dataset (no segmentation masks were available online). We manually labelled 303 frames from the dataset containing one or two divers, picked from 303 randomly chosen videos of the training split. When there were two Best val. Best val. MNIST 5Dot 2Dot h=48 Figure 9. Average results (% acc.) across ten trials with varying numbers of hidden units per layer, repeated five times each. Training and validation on the MNIST-bg domain. The shaded area corresponds to standard deviation across the trials. divers, we segmented each as its own instance. The segmentation masks will be made public.
We fine-tuned a MaskRCNN on our labeled dataset, using a random split of 290 frames as training set and 13 frames to validate, and monitored the bounding box IoU on the validation set. The best model achieved 93% validation bounding box IoU, which we used to segment the frames of the entire dataset (at 32 frames per clip). We used the confidence of the mask predictions as a threshold. The nonzero predictions were mostly confined to a bounding box surrounding the diver(s). When the threshold was t = 0, bounding boxes around the divers were used as crops (S2). When increased to t = 0.4, we obtained proper segments of the diver shape (S1). The frames contain a lot of motion blur which made the segmentation more challenging, and the segmentation at t = 0.4 is not perfect -sometimes parts of for example an arm or foot is missing. The performance of the segmentation at t = 0.4 was deemed sufficient after manual inspection of 100 randomly chosen videos, where all videos had enough evidence to recognize the development of the dive. The segmentation at t = 0 (bounding boxes, S2) was satisfactory in all 100 clips inspected.  Table 17 lists the different variants we tested when training on Diving48 from scratch. In all variants, the number of heads was 8 (A = 8), the patch size was 16 × 16, the learning rate was fixed at 0.001, and the weight decay was 0.00001. When SGD was used, the momentum was always 0.9. Table 16 lists the different model specifications for each of the eight experiments a-h on Diving48 in the main article. For further details on the models, this is described in the main article and in the code repository.