ActMAD: Activation Matching to Align Distributions for Test-Time-Training

Test-Time-Training (TTT) is an approach to cope with out-of-distribution (OOD) data by adapting a trained model to distribution shifts occurring at test-time. We propose to perform this adaptation via Activation Matching (ActMAD): We analyze activations of the model and align activation statistics of the OOD test data to those of the training data. In contrast to existing methods, which model the distribution of entire channels in the ultimate layer of the feature extractor, we model the distribution of each feature in multiple layers across the network. This results in a more fine-grained supervision and makes ActMAD attain state of the art performance on CIFAR-100C and Imagenet-C. ActMAD is also architecture- and task-agnostic, which lets us go beyond image classification, and score 15.4% improvement over previous approaches when evaluating a KITTI-trained object detector on KITTI-Fog. Our experiments highlight that ActMAD can be applied to online adaptation in realistic scenarios, requiring little data to attain its full performance.


Introduction
When evaluated in laboratory conditions, deep networks outperform humans in numerous visual recognition tasks [7][8][9]. But this impressive performance is predicated on the assumption that the test and training data come from the same distribution. In many practical scenarios, satisfying this requirement is either impossible or impractical. For example, a perception system of an autonomous vehicle can be exposed not only to fog, snow, and rain, but also to rare conditions including smoke, sand storms, or substances distributed on the road in consequence of a traffic accident. Unfortunately, distribution shifts between the training and Correspondence: muhammad.mirza@icg.tugraz.at  Figure 1. ActMAD is well suited for online test-time adaptation irrespective of network architecture and task. Our intended application is object detection on board of a vehicle driving in dynamically changing and unpredictable weather conditions. Here, we report the class-averaged Mean Average Precision (mAP@50) over adaptation time 1 , attained by continuously adapting a KITTI [12]-trained YOLOv3 [33] detector. Clear refers to the weather condition of the original dataset, mostly acquired in sunny weather, while Fog, Rain, and Snow are generated by perturbing the original images [15,19]. Gray vertical lines mark transitions between weather conditions. test data can incur a significant performance penalty [2,23,28,31,38,40]. To address this shortcoming, test-time-training (TTT) methods attempt to adapt a deep network to the actual distribution of the test data, at test-time.
Existing TTT techniques limit the performance drop provoked by the shift in test distribution, but the goal of online, data-efficient, source-free, and task-agnostic adaptation remains elusive: the methods that update network parameters by self-supervision on test data [10,28,36] can 1 Time (seconds) for adaptation is hardware specific. For reference, we run these experiments on an RTX-3090 NVIDIA graphics card and it takes ∼75s to adapt to 3741 images (one weather condition) in the KITTI test set. only be used if the network was first trained in a multitask setup, the approaches that minimise the entropy of test predictions [26,38], or use pseudo-labels, or pseudoprototypes [22,23], are not easily adapted to object detection or regression, and the ones that update statistics of the batch normalization layers [31,34] are limited to architectures that contain them. As a consequence of these limitations, while existing methods fare well in image classification, none of them is well suited for the scenario where the need for online adaptation is the most acute, that is, object detection on board a car driving in changing weather conditions.
Our goal is to lift the limitations of the current methods and propose a truly versatile, task-, and architectureagnostic technique, that would extend TTT beyond image classification and enable its deployment in object detection for automotive applications. To that end, we revisit feature alignment, the classical domain adaptation technique [35,43], also adopted by several TTT algorithms [23,28,31,34]. It consists in aligning the distribution of test set features to that of the training set. By contrast to previous methods, that align the distributions of entire channels in the ultimate layer of the feature extractor, ActMAD brings the feature alignment approach to another level by individually aligning the distribution of each feature in multiple feature maps across the network. On the one hand, this makes the alignment location-aware. For example, the features from the bottom of the image, most often representing road and other vehicles, are not mixed with features from the top part, more likely to describe trees, buildings, or the sky. On the other hand, it results in a more fine-grained supervision of the adapted network. Our ablation studies show that ActMAD owes most of its performance to these two contributions. Additionally, while several authors suggested aligning higherorder moments [23,43], we demonstrate that aligning means and variances should be preferred when working with small batches. Unlike methods that only update mean and variance [31,34], or affine parameters of batch normalization layers [23,31,34], ActMAD updates all network parameters. Our approach is architecture-and task-agnostic, and does not require access to the training data or training labels, which is important in privacy-sensitive applications. It does require the knowledge of activation statistics of the training set, but this requirement can be easily satisfied by collecting the data during training, or by computing them on unlabelled data without distribution shift.
Our contribution consists in a new method to use Activation Matching to Align Distributions for TTT, which we abbreviate as ActMAD. Its main technical novelty is that we model the distribution of each point in the feature map, across multiple feature maps in the network. While most previous works focus on combining different approaches to test-time adaptation, ActMAD is solely based on feature alignment. This is necessary for our method to be applicable across different architectures and tasks, but we show that discarding the bells and whistles, while aligning activation distributions in a location-aware manner results in matching or surpassing state-of-the-art performance on a number of TTT benchmarks. Figure. 1 presents the performance of ActMAD in a simulated online adaptation scenario, in which an object detector runs onboard a vehicle driven in changing weather conditions. Most existing methods cannot be used in this setup, because they cannot run online, or because they cannot be used for object detection. Note, that our method recovers, and even exceeds, the performance of the initial network once the weather cycle goes back to the conditions in which the detector was trained.

Related Work
Unsupervised domain adaptation. The idea of aligning neural feature distributions to bridge the domain gap between training and test data was first used for unsupervised domain adaptation (UDA). In CORAL [35], the alignment is performed by first whitening the target domain features and then re-coloring them with covariances computed on the source domain. Alternative methods train the network to minimize a difference of features statistics, like the higherorder central moments [43], or the higher-order cumulant tensor [3]. These UDA algorithms cannot be used for testtime-training (TTT), because they work offline and require access to the source training data and entire test data. Our ActMAD, and some existing TTT approaches e.g. [28,31], also explicitly compute and align feature statistics of the training and test data, but perform online adaptation without access to the training data.
Test-Time-Training. TTT consists in adapting a network to a distribution shift between the test and training data at test-time without access to the training data. The technique that gave the name to the whole group of methods [36] augments the training routine with a self-supervised task, which enables re-training the network on the test set even though no test labels are available. The original work employed rotation prediction [13] as the auxiliary task, but subsequent works [10,28] replaced it with the Masked Autoencoder reconstruction task [16] or contrastive learning [4]. These methods improve performance, but cannot be used if the network was not trained with the auxiliary task. Moreover, contrastive learning requires a large portion of unlabelled data to be effective. By contrast, our ActMAD can be used to adapt a network trained with an arbitrary protocol, and requires very little data to attain its full performance.
Instead of requiring that the network is trained in a multitask setup, CoTTA [39] employs a student-teacher setup only during test-time adaptation. ActMAD performs on par with CoTTA on its own benchmark, while consuming less memory and computation which CoTTA needs to run the teacher ReLU ReLU Figure 2. Schematic of ActMAD. Given a pre-trained model and statistics of the clean activations from the training data, it aligns the activation responses from the shifted test data to the clean activations at test-time. We model the activation distributions in terms of the means and variances of each activation, such that the statistics have the same shape as the feature maps. The statistics of the training activations are pre-computed on the training set, or computed on unlabelled data without distribution shift.
network. CoTTA relies on the entropy of the predicted class distribution to transfer the knowledge from the teacher to the student, so it cannot be used for detection-specific architectures that regress the bounding box coordinates. By contrast, ActMAD is seamlessly embedded in detection systems. While other methods adapt network parameters, T3A [22] casts test-time adaptation as prototype learning. It relies on pseudo-labels to create test-set-specific pseudo-prototypes, which then substitute the classifier learned on the training set. ActMAD outperforms T3A in classifying ImageNet-C and CIFAR-100C images.
Similar to the authors of the T3A, Boudiaf et al. [2] propose to forego updating the network parameters and develop an alternative prediction strategy. It generates class likelihoods by minimizing an objective function that combines a term promoting consistency of the predictions with network outputs and Laplacian regularization. Their method is effective for alleviating the shift in the prior distribution, but brings little improvement when the shift affects appearance of test images, which is the focus of our work.
Another group of TTT methods adjusts network parameters using entropy-based objectives on the test data. SHOT [26] minimizes the expected entropy of individual predictions, while maximizing the entropy of the predicted classes over the entire test set. TENT [38] adjusts the scale and shift parameters of batch normalization layers to minimize the entropy of test predictions. The entropy-based loss functions prevent direct application of these methods to object detection and regression, but ActMAD outperforms them by a fair margin also in image classification.
ActMAD is most closely related to TTT approaches that align statistics of training and test features. Along these lines, NORM [34] and DUA [31] update the batch normalization statistics computed on the training set to match the distribution of the test set. CFA [23] goes further and updates the affine layers in the network to match class-conditioned higher-order central moments of the last-layer test features to those computed for the training set. The reliance on classconditioning prevents application of CFA to object detection. By contrast, ActMAD is free from this constraint. It aligns means and variances of individual features in multiple layers across the network, and updates the entire parameter set. This makes it outperform DUA and NORM, and match the performance of CFA in image classification.

ActMAD
We are given a deep network f (x; θ), where x is an input image and θ is a vector of parameters. We are also given a parameter vector θ * , obtained by training the network on a dataset S of images and their annotations. Our goal is to use f to process a test set T of images that may differ in appearance from the ones in S, but are consistent with them in terms of their content. In many applications, the test images are delivered in a stream, and while we can compose them into batches of moderate size, we are not capable of iterating over the test set. In our flagship application, f is an object detector deployed in a car, S is a sequence of images acquired on a sunny day, and T is a video stream acquired during driving the car in changing weather conditions. To address both the detection and classification scenarios, we impose no constraints on the form of f (x; θ). To ensure maximum versatility of ActMAD, we make no assumptions about the architecture of the network, or the training process.
The requirements of task-and architecture-independence rule out the use of techniques that rely on the output of the network to be a probability distribution [22,26,38,39], or employ auxiliary tasks during training [28,36]. We thus follow the only viable design choice: we treat selected layers of the network as random variables and align the distribution of these variables in the test set to that of the training set. An overview of our method is presented in Figure. 2.
Location-aware activation alignment. We denote the activation responses of the l-th layer of the network f computed for image x by a l (x; θ). This is a feature map of shape C × W × H, denoting the number of channels and its spatial dimensions. Typically, a l does not depend on all model parameters θ, but we abuse the notation in the interest of simplicity. Existing TTT methods based on feature alignment [23,28,31,34] treat all the features in a channel as instantiations of the same random variable, irrespective of their location in the image. By contrast, we assume that different locations in the feature map may be distributed differently. This is particularly the case in photographs, because humans frame objects of interest when taking pictures, and in driving scenarios, where different objects occur in different regions of the image. We therefore separately align each of the C × W × H activations in the l-th layer. This results in a stronger supervisory signal than the classical approach of integrating over the spatial dimensions and aligning the distributions of C channels at the output of the encoder. Our experiments show that this stronger supervision leads to faster and more effective adaptation.
Aligning means and variances. To align activations, we could use one of the existing feature alignment methods, for example adversarial alignment [11], but this would require access to the training data, which we are not granted. We are therefore limited to methods that characterize the distribution in terms of statistics, which could be pre-computed on the training data, and enforce distribution alignment by minimizing the discrepancy of the test statistics with those of the training set. The classical way to do this is to minimize the difference between higher-order sample moments of the distributions [3,43]. However, unlike in domain adaptation, where moment-based methods gained popularity, we need to adapt the distributions based on small samples, since we model each location in each channel separately, and run the adaptation online, on small batches. We thus forego higher-order statistics and use feature means and variances where N D is the number of images in D, i.e. the dataset over which the estimates are computed. For each selected layer l, we pre-compute the activation statistics of the training set S, During adaptation, we compute the same statistics for each batch B of the test set, and minimize Aligning multiple layers. We found that aligning just the statistics of the final layer, following the common practice [23,28], is not enough for fast and effective distribution alignment. We show this in an ablation study in Section 4.5. We therefore define a set L of multiple layers for alignment and the corresponding alignment objective as Intuitively, the aligned layers should be distributed evenly across the network. In the case of convolutional architectures with the standard convolution-normalization-ReLU sequences, we found that it is best to align the features obtained after normalization and before the nonlinearities.
Updating the complete parameter set. Finally, while numerous existing methods only update affine parameters or statistics of the batch normalization layers [23,31,34,38], we update all the parameters of the adapted network. It might seem a priori that this increases the risk of making the parameter vector drift away from its initial value, but continuously aligning a very rich set of statistics actually allows the network to recover its initial performance after the distribution of the test data becomes consistent with that of the training set, as shown in Figure. 1. The ablation studies in Section 4.5 further confirm that there is no significant performance difference between the two strategies, and thus, we opt for the more deployment-friendly of the two alternatives and make a gradient step on for each batch B of the test data T .   Table 4. Continuous Adaptation (following CoTTA [39]) to distribution shifts at test-time. We report the mean error of 10 runs, while shuffling the order of the corruptions for each run.

Experimental Evaluation
We first summarize the datasets and baselines used for the evaluation, and then discuss our results.

Datasets
We ran experiments on the following data sets: -CIFAR-10C and CIFAR-100C [19] were created by corrupting test images of the CIFAR-10 and CIFAR-100 datasets [24]. We used them to benchmark the performance of ActMAD in image classification, and for ablation studies. Following the common protocol [28,31,34,36,38], we used 15 corruption types and 5 corruption levels.
-ImageNet-C [19] contains ImageNet [5] test images, corrupted in the same way as those of CIFAR-10/100C. We used them for large-scale classification experiments. Again, we tested for 15 corruption types and 5 severity levels.
-We used the KITTI dataset [12] to evaluate the per- formance of ActMAD on object detection for automotive applications. KITTI contains 8 classes of traffic participants, and was recorded mostly in clear weather. To simulate degrading weather conditions on the KITTI dataset, we used physics-based rendering of fog and rain [15], and followed Mirza et al. [30] to simulate snow. For testing, we used the dedicated test split of each dataset. On KITTI, however, the annotations for the test set are not publicly available, so we divided the public train set randomly into train and test split with equal proportion. Here, we report the results for the highest distribution shift severity. Results for lower severity are provided in the supplemental.

Baselines
We compared ActMAD against the following approaches: -Source is the model trained on source data and evaluated on test data without any adaptation. -TTT [36] adapts the network to test data using selfsupervision from a rotation prediction task. -TTT++ [28] employs a combination of self-supervised contrastive learning and feature alignment. -P-L is our adaptation of the pseudo-labeling approach [25]

Image Classification
To evaluate ActMAD on image classification, we adapted a Wide-ResNet-40-2 [42] (WRN-40-2) from the Robust Bench [19] and a Vision Transformer [6] (ViT-B/16) backbone on the CIFAR-C datasets. For Imagenet-C dataset, we used an ImageNet pre-trained ResNet-18 [17] from the PyTorch model zoo, following previous work [31]. Results on ImageNet-C with other backbones are provided in the supplemental. We used a batch size of 128 for all image classification experiments. ActMAD was run with the learning rate of 1e−2 for CIFAR-10C, 1e−3 for CIFAR-100C and 25e−5 for ImageNet-C, while the baselines used hyperparameters reported by their authors. Following prior literature [22,31,38], we performed the experiments online, using each test image to adapt the network only once. Since SHOT and TTT++ are better suited for the offline scenario, we additionally run them for 500 epochs on the entire test sets of the CIFAR-C data sets, with an early stopping criterion, according to the official TTT++ implementation 2 .
Results for CIFAR-10/100C for WRN-40-2 are reported in Table. 1 and 2, while the ViT results are listed in Table. 3. ActMAD outperforms other online methods in classifying CIFAR-10C and CIFAR-100C images by a fair margin, establishing a new state-of-the-art for both Transformer-based and convolutional architectures. When adapting the wide ResNet to CIFAR-10C, there is almost no gap between Act-MAD and the offline approaches, while on CIFAR-100C offline adaptation fares better. For the ViT, we also achieved a new state-of-the art on CIFAR-10C and CIFAR-100C, outperforming even CFA [23], designed specifically for robustifying vision transformers.
The results for the ImageNet-C dataset are shown in Table. 5. ActMAD comes in second after CFA [23], but the  Table 6. Mean Average Precision (mAP@50) for a KITTI pre-trained YOLOv3 tested on rain, fog and snow datasets. Higher is better. a) Results for the most severe fog level, i.e. only 30m visibility. b) Results for the most severe rain level, i.e. 200mm/hr rain intensity. c) Results for snow. Best mAP is shown in bold, while the second best is underlined.  performance gap in classification error is only 0.1 percentpoint. We suspect the advantage of CFA to stem from its alignment of class-conditioned statistics, which may become more important for the rich and diverse set of ImageNet classes than for the limited sets of classes of the CIFAR datasets. However, the use of class-conditional statistics limits CFA to the classification task.
Continuous adaptation to pertubations. Recently, Wang et al. evaluated their TTT method CoTTA [39] by continuously adapting to different corruptions, instead of adapting to only a single corruption at a time. We tested ActMAD and other baselines in such a scenario on CIFAR-10/100C. The results, listed in Table. 4, show that ActMAD performs on par with CoTTA, and outperforms other baselines, while consuming less memory and computing power than CoTTA, which relies on a student-teacher setup.

Object Detection
For object detection experiments, we adapted a KITTIpretrained YOLOv3 [33] to degrading weather conditions, such as fog, rain, and snow. We used a learning rate of 1e−4, and a batch size of 30. We compared our object detection results with DUA, NORM, and TTT. DUA and NORM could be applied to detector adaptation without any modifications, thanks to the fact that they do not rely on the network to produce normalized probabilities and are not limited to adapting models trained in a multi-task setting. By contrast, to deploy TTT, we had to re-train the detector with the auxiliary, self-supervised rotation prediction task. We describe the implementation details in the supplemental. The results, reported in Table. 6, show that ActMAD outperforms the baselines by a significant margin. Additionally, in Figure. 3 we plot the performance of the network during adaptation, showing that ActMAD stabilizes its precision as adaptation progresses, while the baselines provoke performance fluctuations. Needless to say, consistent level of performance is of utmost importance in safety-critical applications, like autonomous driving.
We also tested ActMAD in continuous online adaptation, where different weather conditions are encountered interchangeably, and show one such result in Figure. 1, while deferring two more plots to the supplemental. After adaptation to a whole weather cycle, ActMAD can quickly re-adapt to the conditions on which the network was initially trained, and even exceed its initial performance.

Ablation Studies
Effect of batch size. The number of samples used by Act-MAD to compute the location-aware statistics is equal to  the number of batch elements, and therefore much lower than the number of samples used to compute the channelwise statistics in competing methods. Statistics computed on small samples have higher variance, which could adversely affect adaptation performance. We investigate how decreasing the batch size affects the performance of ActMAD. Like in the image classification experiments, we use an ImageNet pre-trained ResNet-18 from the PyTorch model zoo. We decrease the learning rate in proportion to the decreasing batch size, following Goyal et al. [14]. As shown in Figure. 4, limiting the batch size decreases the performance gently, and even batches as small as 10 images still let us achieve 13.4 percent-point improvement over the un-adapted model.
Design alternatives. To verify how our design choices contribute to the performance of ActMAD, we removed each of its key features and evaluated the resulting algorithms on image classification and object detection. We adapted a wide ResNet from CIFAR-10 to CIFAR-10C, and YOLOv3 from KITTI to KITTI-Fog, in the same setup as in previous experiments. As can be seen in Table. 7, a large performance drop was provoked by limiting the alignment to the last feature map, as opposed to multiple feature maps throughout the network, and by aligning channel, instead of pixel, statistics.
This attests to the importance of fine-grained supervision, distributed over multiple layers and locations in the feature maps. Unsurprisingly, the use of location-aware statistics results in higher performance gain for KITTI (∼ 10 percentpoint) than for CIFAR (∼ 1 percent-point). This is expected, since road scenes of KITTI are much more structured than the CIFAR images. We also hypothesize that ActMAD's localized feature alignment benefits object detection more, because detection relies on local features to a higher degree than the location-agnostic image classification. The type of statistics used for feature alignment seems to make no difference for CIFAR-10C, but aligning means and variances gives better results than aligning higher-order central moments for the KITTI object detector. This difference might stem from the fact that the KITTI experiments use almost ten times smaller batch size, and that low batch size affects higherorder moment estimates more than estimates of means and variances. Finally, limiting the parameter update to channelwise affine transformations results in a small performance drop in object detection, while almost not affecting the image classification accuracy. However, since we observed no drift in the continuous adaptation experiment and limiting the update might require manually inserting affine layers to the network, we prefer to perform the full update.

Conclusion
We propose a new method -ActMAD -for online testtime-training based on location-aware feature alignment. In contrast to many previous TTT approaches, our ActMAD is task-and architecture-agnostic. It goes beyond image classification and can be readily applied to object detection. ActMAD outperforms existing approaches by a considerable margin, while being more stable and adapting faster to new distributions. The power of ActMAD stems from finegrained supervision which results from individually aligning each pixel in multiple feature maps across the network. Experiments show that the fine-grained supervision provided by ActMAD from location-aware feature alignment is especially helpful for tasks which require a holistic understanding of the scene, for example object detection.

Supplemental Material
In the following, we present additional continuous online adaptation scenarios in which we test ActMAD (Section A), provide all implementation details for baselines and our experiments to support reproducibility (Section B) and present additional ablation studies (Section C).

A. Continuous Online Adaptation
Our location-aware feature alignment is especially helpful for the task of object detection. The fine-grained supervision helps ActMAD to adapt to different weather scenarios in a dynamic and efficient manner. Moreover, ActMAD also shows promising results while adapting to different weather scenarios in a continuous manner. In the main manuscript (Figure 1), we show such a simulated continuous adaptation scenario (Clear→Fog→Snow→Rain→Clear). We further test ActMAD in similar scenarios where the weather conditions are encountered in a different sequence. In Figure 5a and 5b, we present results for adaptation with ActMAD in two such scenarios.
The results show that ActMAD can achieve impressive results in different scenarios and the sequence of weather conditions which are encountered does not matter. This is close to the requirements in the real-world, where different weather conditions can occur in any order during driving. We also see from the results that Act-MAD can even improve the performance of a KITTI [12] pre-trained YOLOv3 [33] on the validation split from the dataset (which represents the Clear weather condition). For example, in Figure 5b, where order of the weather conditions is Clear→Clear→Fog→Snow→Rain→Clear, Act-MAD achieves 1.0% higher Mean Average Precision while adapting from Clear→Clear at the beginning of the adaptation cycle. This is because even though the adaptation is performed from Clear→Clear, the activation statistics could be different between the training and inference, e.g., due to slight change in the environmental conditions or scenery. Our ActMAD helps to rectify this mismatch to obtain an increase in performance.

B. Implementation Details
Hardware: All our experiments are performed on a single NVIDIA ® GeForce ® RTX 3090 and a GPU server consisting of four Quadro RTX 8000.
ImageNet-C: All ImageNet-C experiments use ImageNet pre-trained models. For the main results with ResNet-18 [17] we use the pre-trained weights from the PyTorch [32] model zoo. For experiments with ResNet-50 using Group Normalization [41] (Section C.1), we use pre-trained weights from a third-party open source repository 3 . For experiments with a DeepAug [18] pre-trained ResNet-50 (Section C.2), we use the pre-trained weights from Robust Bench 4 . For experiments on ImageNet-C with ViT-B/16 model (Section C.4), we use the pre-trained weights from the open source timm library 5 .
KITTI: All KITTI [12] experiments start with a model pre-trained on the original KITTI dataset. As it is collected mostly in clear (i.e. sunny) weather conditions, we refer to it as KITTI-Clear. We adapt this model to degrading weather conditions, i.e. fog [15], rain [15] and snow [19]. For KITTI-Clear, we re-train an MS-COCO [27] pre-trained YOLOv3 in a supervised manner on the train split of the KITTI dataset. The model is trained for 100 epochs with a batch size of 30, while all the other training details are kept constant as in the PyTorch implementation of YOLOv3 6 .
Baselines: Since all the baselines used to compare our results are primarily tested for object recognition on CIFAR-10/100C [19] and ImageNet-C [19], we use their official open-source codes and use all the hyper-parameters, which they report in their paper.
For object detection experiments, only DUA [31] provides official results 7 . Therefore, we implement NORM [34] for object detection using their official repository 8 . Since NORM is a gradient-free approach, it does not rely on any hyper-parameters. We use a batch size of 30 to obtain all the results for NORM.
TTT [36] uses rotation prediction as an auxiliary task for test-time training. TTT is also primarily tested for object recognition by its authors. We implement TTT for object detection for the purpose of comparison in our paper. We use the same batch size (i.e. 30), as for ActMAD. For jointtraining with the auxiliary rotation prediction task we take the output from the feature encoder of YOLOv3 and use a single-layer linear classifier head to predict the rotation of the input image. The hyperparameters used for the KITTI pre-training (for ActMAD experiments) and joint-training are kept the same. However, we find that training a network jointly for the classification task of rotation prediction and object detection requires special design choices. For    example, we use a loss scaling factor of 0.4 for the selfsupervised task during joint-training to make the network converge. Similarly, the same loss scaling factor is used during test-time training. The evaluation protocol is kept consistent for all baselines and our method.
We also implemented TTT++ [28] on YOLOv3 in order to use it as a baseline for object detection experiments. We used the same prediction head, hyper-parameters and the augmentations (for creating two augmented views for SimCLR [4] training) used by the original TTT++ paper for their experiments on CIFAR-10/100. We experimented with several design choices for joint-training (object detection and contrastive SimCLR task), each for a total of 500 epochs on the KITTI train set, starting from the pre-trained weights on MS-COCO. However, despite the best of our efforts we could not make the joint-training converge. A potential issue could be the requirement of using larger batch sizes for training the contrastive learning task. Initially, we tested with a batch size of 100 for joint training by keeping the original aspect ratio of KITTI images (i.e. 370 × 1224). However, realizing that we might require larger batch sizes, we had to downscale the KITTI images equal to the aspect ratio of ImageNet images (i.e. 224 × 224), to be able to increase the batch size to 300, but in this case the regression loss for object detection did not converge and we get low performance on the validation set of KITTI. We suspect that it might be because of the drastic decrease in the image size, as compared to the original aspect ratio of images in the KITTI dataset, which is unsuitable for the object detection task on the KITTI dataset.

C. Additional Results
In this section we provide additional ablation studies, results on lower severity levels for ActMAD, detailed results for the vision transformer backbone [6] and the Continuous Adaptation experiment.

C.1. Different Normalization Techniques
Recent test-time training approaches which adapt the network statistics, e.g. NORM [38] and DUA [31], criti-  Table 9. Top-1 Classification Error (%) for all corruptions in CIFAR-100C (level 4 -1), highest severity is reported in the main manuscript. Lower is better. The results were obtained by adapting a WRN-40-2 backbone, trained on CIFAR-100, to CIFAR-100C. ActMAD results are averaged over 10 runs, and we report the mean error and its standard deviation.
cally rely on batch normalization [21]. However, if the network is equipped with other forms of normalization, e.g. Instance [37], Layer [1] or Group Normalization [41], these approaches are not applicable. In contrast, our ActMAD is agnostic to the architecture and can work with any type of normalization applied in the network. We already showed in the main manuscript ( Table 3) that it can be applied to a Transformer-based architecture, which uses Layer Norm. In order to test the performance of ActMAD with different normalization layers inside the convolution-based architectures, we test ActMAD with a ResNet-50 equipped with Group Normalization. Figure 6 shows the adaptation results of a ResNet-50 with both Batch Normalization (BN) and Group Normalization (GN). Using GN, our ActMAD performance decreases by only 1.8% over the same architecture using BN.

C.2. Robustifying an Already Robust Model
ActMAD can be effectively used with any off-the-shelf pre-trained model. To test this, we apply ActMAD for test-time adaptation of an already robust, AugMix [20] pretrained ResNet-50. We further decrease the batch size for adaptation on this backbone. Results for this experiment (highest severity) are provided in Figure 7. We find that Act-MAD decreases the error significantly even for an already  Table 10. Top-1 Classification Error (%) for all corruptions in ImageNet-C (level 4 -1), highest severity is reported in the main manuscript. Lower is better. The results were obtained by adapting a ResNet-18 backbone, trained on ImageNet, to ImageNet-C. ActMAD results are averaged over 10 runs, and we report the mean error and its standard deviation.  Table 11. Top-1 Classification Error (%) for all corruptions in CIFAR-10/100C and ImageNet-C (level 5). Lower is better. All results are obtained by using a ViT-B/16 backbone. ActMAD results are averaged over 10 runs, and we report the mean error and its standard deviation. robust model. For example, the Source (without adaptation) top-1 error (%) over the 15 corruptions on ImageNet-C is 61.1%, while after ActMAD adaptation the error decreases to merely 43.5%. Furthermore, we also see that while applying ActMAD to a robust model, the decrease in performance with decreasing batch size is also minimal (similar to the ablation study in the main manuscript, Figure 4, with ResNet-18 backbone). For example, while decreasing the batch size from 125 to 10, the error increases by only 4.1%.

C.3. Lower Severity Results
In the main manuscript we provide results for the highest severity (Level 5) in the Robust Bench [19]. Our ActMAD achieves impressive gains in performance even for lower severity levels. In Table 8 and Table 9 we provide the mean and standard deviation of results (over 10 random runs) ob-tained by ActMAD on lower severities (i.e. Levels 4 -1) on CIFAR-10/100C. We use the Wide-ResNet-40 model for these evaluations. Furthermore, in Table 10, we provide the mean and standard deviation of results over 10 random runs on the large scale ImageNet-C (with ResNet-18) dataset for lower severities (Levels 4 -1). The results show that ActMAD can even be applied when the distribution shift between the train and test data is not too large.

C.4. Vision Transformer
ActMAD can also be seamlessly applied to the Vision Transformer backbone. In the main manuscript we provide mean results over the 15 corruptions for CIFAR-10/100C (Level 5). For reference, in Table 11 we provide results for each individual corruption for the highest severity in the CIFAR-10/100C. Additionally, we also provide ActMAD re-  Table 12. Top-1 Classification Error (%) for all corruptions in CIFAR-10/100C (level 5), while continuously adapting to corruptions. Lower is better. These results were obtained by adapting a Wide-ResNet-40-2 backbone, trained on CIFAR-10/100, to CIFAR-10/100C. ActMAD results are averaged over 10 runs, and we report the mean error and its standard deviation. The lowest error is shown in bold, the second best is underlined.
sults for the highest severity on the ImageNet-C benchmark.

C.5. Continuous Adaptation to Perturbations
In the main manuscript (Table 4) we compare with CoTTA [39] on their continuous adaptation benchmark and provide the mean error over the 15 corruptions (highest severity) in the CIFAR-C benchmark. In Table 12 we provide detailed results, averaged over 10 random runs, for each individual corruption for documentation purposes.