How do Cross-View and Cross-Modal Alignment Affect Representations in Contrastive Learning?

Various state-of-the-art self-supervised visual representation learning approaches take advantage of data from multiple sensors by aligning the feature representations across views and/or modalities. In this work, we investigate how aligning representations affects the visual features obtained from cross-view and cross-modal contrastive learning on images and point clouds. On five real-world datasets and on five tasks, we train and evaluate 108 models based on four pretraining variations. We find that cross-modal representation alignment discards complementary visual information, such as color and texture, and instead emphasizes redundant depth cues. The depth cues obtained from pretraining improve downstream depth prediction performance. Also overall, cross-modal alignment leads to more robust encoders than pre-training by cross-view alignment, especially on depth prediction, instance segmentation, and object detection.


Introduction
Pretraining of neural networks has been an established tool in computer vision for several years [2]. It allows to successfully finetune neural networks on new tasks with fewer iterations and less annotated data [3]. Commonly, models are initialized using weights pretrained for image classification on the ImageNet dataset [4], but recently, self-supervised approaches that do not rely on manual annotations have outperformed ImageNet pretraining [5]. : Redundant information is shared across modalities, while complementary information is exclusive to a modality. In this work, we show that cross-modal visual representation learning predominantly encodes redundant information of 2D and 3D data. *Images (b) and (c) were taken from [1].
In the realm of self-supervised representation learning, contrastive learning [6] has become a popular approach for visual representation learning. In contrastive learning, the model learns to distinguish different variations of the same instance (e.g. crops of a single image) from all other instances. These variations can be created artificially or by using natural data correspondences, such as images and text [7] or from multiple viewpoints [8,1].
A common objective is to enforce similarity between features across views or sensing modalities. This is called representation alignment [9]. While severe failure cases of such representation alignment have been discussed [9,10], the effects on the representation itself were not studied, as of yet. Figure 1 shows a conceptual illustration of the information available in 2D image data and 3D point clouds in the ScanNet dataset [11]. Some information, such as about depth and surfaces, is available in both modalities. Although this information may be more detailed and more accurate in one modality than in the other, both modalities can express such properties to a certain degree. In the context of sensor fusion, this is typically called redundant information. Other information is exclusive to a modality, which is called complementary information. 3D point clouds, for example, are generally color-and textureless.
Intuitively, a representation which discards complementary information is less expressive than one that includes redundant and complementary information. On the one hand, discarding texture and color information, for example, leads to a less complete visual representation. On the other hand, ImageNet pretrained models were found to be texture-biased and by increasing their shape-bias the performance on tasks such as object detection could be improved [12]. So what is the influence of complementary and redundant information on the visual representation quality?
In this work, we study how complementary and redundant information affect visual representation learning when exploiting cross-view and cross-modal representation alignment, and how this relates to the performance of a finetuned model on a transfer learning downstream task. For this purpose, we use Pri3D [1], a method that uses 3D information of a scene to enable cross-view and crossmodal representation alignment for visual representations learning.

Related work
Self-supervised representation learning has recently surpassed supervised pretraining on ImageNet [5]. Various approaches have been explored to learn a global feature vector per image, such as contrastive learning [5,6] and methods that do not require negative samples during training [13,14]. Since many computer vision tasks require per-pixel features, several works focused on learning dense representations [15,16]. Inspired by this success for visual representations, contrastive learning has also been applied to 3D point cloud data [17,18,19].
Several approaches have explored cross-modal and cross-view visual representation learning. While some propose a single model [20] for multiple modalities, we are interested in training a vision-only model using crossmodal and cross-view input. To this end, some have matched images and natural language to improve the learned visual representation [7,21], whereas others use 3D signals [22,23,1], or match representations of different views of a scene [8,1].
While cross-modal feature spaces are a common technique [24,25,26,1,27,19], they can introduce several caveats. Especially when dealing with partially incomplete or corrupted data, representation alignment may hinder effective learning for multi-view clustering [9]. In domain adaptation, avoiding representation alignment across modalities has also proven to be beneficial to prevent discarding complementary sensor information [10]. In representation learning, cross-modal training can even lead to an emphasis on a single strong modality and potentially harm the overall performance [28].
Our goal is to understand the effects of cross-view and cross-modal representation alignment on contrastive learning in more detail. Specifically, we do not consider corrupted or incomplete sensor data, but we investigate how redundant and complementary information influence the learned representations, and whether similar effects as observed for shape-biased vs texture-biased networks [12] can be observed. We base our empirical study on Pri3D [1] as it uses both cross-view and cross-modal representation alignment for contrastive learning, as well as their combination. As our main contributions, we: 1. assess how cross-view and cross-modal representation alignment affect the complementary and redundant information encoded in the learned visual perpixel representations, 2. evaluate the transfer learning performance in different settings on various tasks and datasets to investigate the robustness of cross-view and cross-modal alignment.
2D images and 3D point clouds have low-level redundant and complementary features, as illustrated in Figure 1. Therefore, these modalities lend themselves to study cross-view and cross-modal representation alignment. Our analysis of alignment in representation learning is based on Pri3D [1]. Pri3D uses 3D information for 2D visual representation learning. It employs a selfsupervised contrastive learning approach that does not require any human annotation, but only RGB images and registered point clouds that can be obtained, for example, from an RGB-D dataset. This section summarizes the two contrastive losses of Pri3D and discusses how the feature spaces of the losses can be separated.

Pri3D: contrastive losses
Pri3D finds pairs of pixels and elements of a regular grid in 3D space, so-called voxels, for contrastive learning. Pixels and voxels are matched by their distance in 3D space to define two losses. One loss requires pairs of pixels across views, whereas the other loss is based on pairs of pixels and 3D points across modalities. Since Pri3D matches pixels and voxels based on their 3D positions, it is relatively robust with respect to incomplete sensor data.
Cross-view contrastive loss (VIEW) This contrastive loss matches pixels across different views of the same scene. It requires two RGB-D images from the same scene and the corresponding relative camera poses and calibration. Once two pixels from two different views A and B are within a 2 cm distance, they are considered a positive pair in the sense of contrastive learning [6]. Only pixels that are part of a positive pair are used in this loss, as the negative pairs are also constructed from these pixels. To form the negative pairs, each pixel a of view A of a positive pair (a, b) is paired with all pixels k of view B which are part of a positive pair but not k = b. For each image x, an encoder-decoder network f predicts a set feature vectors f (x), and f a denotes the L2normalized feature vector of pixel a. Using the set of positive pairs M , Pri3D employs a PointInfoNCE loss [18] where τ is a temperature hyperparameter. This cross-view loss is called the VIEW loss, following the convention of Pri3D. Compared to other visual representation learning approaches, VIEW does not rely on strong augmentations that may corrupt color information [6].
Cross-modal contrastive loss (GEO) Unlike the VIEW loss, the pairs for the GEO loss do not consist of two pixels from different views, but instead of a pixel and a voxel. The positive pairs are again pixels and voxels within 2 cm distance in 3D space. The negative pairs are all other non-matching pixels and voxels pairs, similar to the VIEW loss.
The voxel is associated with a feature vector obtained from a dense 3D feature encoder. This dense 3D feature encoder is jointly learned with the dense 2D feature encoder. The loss function (Eq. (1)) remains the same with f a and f b now being feature vectors from the 2D and 3D feature encoders, respectively. Thus, this crossmodal loss enforces the features to be similar across the two modalities and is called the GEO loss, following the convention of Pri3D.

Pri3D: representation space separation
In Pri3D [1], the GEO and the VIEW loss use the same visual feature vector for a single pixel. Thus, the combination VIEW+GEO enforces mutual alignment of cross-view and cross-modal representations. Interestingly, in the implementation published by the original Pri3D authors [29], the feature spaces of both losses are separated by a linear layer preceding the GEO loss (see Figure 2). We hypothesize that the original purpose of this layer was merely to match the feature dimensions of the 2D and 3D backbone (e.g. for ResNet50 2D backbone this layer projects 128 to 96 channels). Yet, this separation of feature spaces relaxes the alignment of the cross-view with the cross-modal feature representation. The feature vector of a pixel in the GEO loss is now computed by a linear function on the feature vector of the same pixel in the VIEW loss, thus the two feature vectors are not the same anymore. The cross-view and the cross-modal representations are not directly aligned. Since the effect of this linear layer was not described in the literature, we include both variants in our experiments. In the following, we will distinguish this variation by adding (lin) to models that include the linear layer that separates the feature representations. These and other differences to the original Pri3D paper are described in more detail in the supplementary material.

Experiments
In our experiments, we examine how cross-view and cross-modal representation alignment in Pri3D influence the encoded complementary and redundant information and how possible differences in the representations affect downstream transfer learning performance.

Setup
The experiments consist of two main steps: We first pretrain models based on the variations of Pri3D (see Section 3). Afterward, the pretrained models are used to initialize training on downstream tasks, where we distinguish between a frozen (Section 4.2), a half-frozen (Section 4.3), and a full finetuning (Section 4.4) setting.

Pretraining
Our implementation is mainly based on the published code of Pri3D, and thus, we follow the same pretraining protocol as presented in the paper [1] and give a coarse outline here. We always use a UNet-style ResNet50 2D backbone and a UNet-style sparse convolutional 3D backbone. As in [1], the 2D backbone is initialized using supervised ImageNet pretraining before performing selfsupervised pretraining on the ScanNet dataset [11]. The ScanNet dataset is a collection of RGB-D sequences of indoor scenes, captured with a Kinect-like setup. On this dataset, the model is trained for 5 epochs using stochastic gradient descent with a learning rate of 0.1 with polynomial decay of 0.9 and the batch size is 64, unless explicitly mentioned otherwise. This corresponds to approximately 60k iterations and 4 days of training time on 8 Nvidia V100 GPUs for pretraining a single model. The temperature parameter in the losses (Eq. (1)) is set to 0.4.

Per-pixel downstream tasks
In Sections 4.2 to 4.4 the per-pixel representations are evaluated on depth prediction, image reconstruction, and semantic segmentation while parts of the pretrained network are frozen. To decode the information from features extracted by the frozen pretrained network, we append a non-linear per-pixel MLP (see Figure 2c). This MLP is implemented as a sequence of 1x1 convolutions: (c = 3 for image reconstruction, c = 1 for depth estimation, and c = #classes for semantic segmentation). The MLP is followed by a bilinear fixed interpolation that upscales the output from 120 × 160 to 240 × 320, which is analogous to the semantic segmentation downstream experiments of Pri3D [1] with the UNet architecture. Note that we discard the last convolutional layer from the pretrained model, which, during pretraining, linearly projects the features to the space in which the contrastive loss operates. It can be regarded as the head network which aims to solve the contrastive task.
For image reconstruction and depth prediction, we choose an L2 loss directly on the RGB input pixel values and depth ground truth values, respectively. For semantic segmentation, we use the common cross-entropy loss to predict the semantic classes of each pixel. Other than that, the same training protocol as for the semantic segmentation task in [1] is applied. The mean Intersectionover-Union (mIoU) is the metric for the semantic segmentation quality. The datasets used for depth prediction and image reconstruction are: • • Cityscapes [32]: A dataset of 5000 semantically annotated frames of urban street scenes in 50 different cities.

Object-centered downstream tasks
To perform object detection and instance segmentation, the encoders of the pretrained models are used to initialize a Mask RCNN model [33] of the Detectron2 library [34] in Section 4.5. Due to the downstream architecture, the decoders are discarded. In addition to ScanNet25kframes and NYUv2, the models are evaluated on the 2017 version of the COCO dataset [35] that contains more than 118k training and 5000 validation images. The Average Precision (AP) metric is used on all datasets.

Frozen downstream tasks
With the frozen downstream tasks, where we keep all weights of the pretrained network frozen, we investigate how well the pretrained models capture color, depth, or semantic information by predicting this information from the learned per-pixel representations. Using these frozen downstream tasks, we show how the different alignment strategies in Pri3D influence the learned representations. For semantic segmentation, this frozen setup is similar to the linear evaluation protocol commonly used to evaluate the quality of global image features in visual representation learning [6]. In this linear evaluation protocol, the performance of a linear classifier on the global features is often regarded as a proxy metric for the representation quality. In our setup, we evaluate whether a non-linear MLP can provide a proxy metric for the representation quality.
The first four rows of Table 1 show the best performance on the validation data during the training process for each model variation and frozen downstream task. Compared to the cross-view model (VIEW), all other models perform worse on frozen image reconstruction and semantic segmentation. In contrast, the models that include the cross-modal GEO loss perform better than the VIEW on ScanNet depth prediction, while on NYUv2 this general trend is not visible. The VIEW+GEO (lin) model outperforms the VIEW+GEO model on all three frozen downstream tasks by more than 10%. Figures 3 and 4 show the qualitative reconstruction and depth prediction results. In both cases, the ranking of the models on ScanNet is reflected in the visual reconstruction quality. The VIEW model achieves the best visual reconstruction, followed by the VIEW+GEO (lin),  Table 1: Validation performance on the three tasks and four datasets. Rows 1-4 show the frozen downstream setting where the pretrained weights are fixed, in rows 5-8 the encoder of the UNet-style architecture is fixed, and in the last four rows, the entire models are finetuned. To have a consistent notion that a greater value means better performance, the negative L2-Loss is shown. Blue color indicates relative improvement with respect to the cross-view model (VIEW).
VIEW+GEO and GEO models. It is notable that the results differ especially on flat texture-rich surfaces, for example in the first row where the GEO model does recover the wallpaper. All model variations discard most of the color information. The most detailed and smoothest depth estimation is obtained by the VIEW+GEO (lin) model. The depth gradient along surfaces appears smooth whereas the depth estimation based on VIEW+GEO and VIEW is speckled on flat surfaces.
In conclusion, the cross-view VIEW loss preserves more texture information while the cross-modal GEO loss leads to a better depth representation. This observation is in line with the ideas of complementary and redundant information discussed in Figure 1 mation. Notably, the depth prediction result is even better than for the GEO model, suggesting that even with respect to the redundant information, this separation is beneficial. The semantic segmentation performance differs strongly between the model variations and benefits from the crossview loss and the linear layer in VIEW+GEO (lin) that lifts the direct alignment.

Half-frozen downstream tasks
By freezing only the encoder part of the UNet-style models, the following experiments provide insight to which part of architecture discards the texture information. The experimental setup, architecture, and datasets remain the same as for the frozen downstream tasks in Section 4.2, except that now also the decoder is finetuned together with the MLP and only the encoder is fixed during finetuning. The GEO model performs best across all tasks and datasets as seen in rows 5-8 of Table 1. The model excels on depth prediction, and interestingly, also outperforms the VIEW model on image reconstruction. The combined models outperform VIEW on depth prediction, but the other tasks show mixed results. The linear layer (lin) improves the VIEW+GEO performance slightly.
Overall, the results are in stark contrast to the frozen downstream tasks. Given that the GEO model outperforms the VIEW model, it seems that the decoder discards the texture information in the frozen downstream tasks, and recovering the texture is possible by finetuning. At the same time, the encoder of the GEO model extracts very robust features across tasks and datasets.

Full finetuning downstream tasks
To assess the practical value and generality of a visual representation, the models are now fully finetuned and evaluated on the downstream tasks. In contrast to the frozen downstream tasks, all weights in the model are finetuned during the downstream task training. 1 As the full model is finetuned, the per-pixel MLP is not needed anymore and is omitted. Rows 9-12 in Table 1 show the full finetuning downstream performance. Models that were trained including the GEO loss outperform the VIEW model on depth prediction. Again, GEO is the best model for image reconstruction. On semantic segmentation, VIEW performs best on Cityscapes and KITTI, but not on the other two datasets.
Also in this full finetuning setting, cross-modal alignment improves depth prediction performance. For semantic segmentation, the results are generally mixed, but it is notable that VIEW performs well on the two outdoor datasets. This may indicate that the cross-modal models are biased towards spatial layouts of indoor scenes. Adding the feature space separation (lin) to VIEW+GEO does not have a positive effect, but rather tends to decrease the performance.

Instance and object segmentation
As discussed in [12], texture-biased networks are inferior to shape-biased networks for object detection. In order to test whether this also holds for texture-biased net-works compared to depth-biased networks, the pretrained encoders are finetuned for object detection and instance segmentation. Table 2 shows that the cross-modal GEO model outperforms the VIEW model on 5 out of 6 task and dataset combinations. The combined models VIEW+GEO and VIEW+GEO (lin) models show mixed results, perform well on ScanNet, but not on COCO. On NYUv2, their difference is most pronounced where the linear feature separation (lin) leads to the worst performance among the four model variations while omitting it results in the best performance.
Combining the losses in the VIEW+GEO and VIEW+GEO (lin) models seems to be beneficial within the pretraining dataset, but not when shifting domains. Overall, although the performance differences are small, the GEO model appears to be more robust. This supports the idea that texture-biased networks are not as robust for object-centered tasks as the depth-biased GEO network.

Discussion
The comparison on the frozen downstream tasks reveals that cross-view and cross-modal representation alignment influence the complementary and redundant information encoded in the learned representations. While the VIEW model preserves the texture and the GEO model preserves the depth, the combination VIEW+GEO does not result in the different low-level features complementing each other, but instead a fully shared feature space is obtained where complementary information is lost and redundant details are less accentuated. VIEW+GEO (lin) improves the combined model on all three frozen downstream tasks and raises the question of whether this means we have found a generally better representation. The performance on the half-frozen and full finetuning downstream tasks contradict this apparent conclusion and we even find that the GEO model performs most robustly across tasks and datasets. This shows that reducing texture-bias in networks increases robustness, similar to findings in [12].
On semantic segmentation, the frozen downstream task shows a distinct ranking that is persistent across datasets. These distinct performance differences were not observed once the decoder is finetuned as well. A possible explanation could be that the texture-bias is to a certain degree useful but also easy to recover and thus not important to learn during pretraining. So although the strong differences on the frozen downstream tasks reveal that the diverse representations either incorporate or discard com-plementary information, the performance on the frozen tasks does not correlate with the finetuning downstream performance.
The comparison of VIEW+GEO with VIEW and GEO indicates different behavior on data within the same domain and across domains. The VIEW+GEO model excels especially in object detection, instance segmentation, and semantic segmentation on the ScanNet dataset. A similar effect of overfitting to the pretraining dataset is also reflected in the learning rate variations shown in the supplementary material. In a broader context, this could mean that aligning cross-modal and cross-view representations may harm generalization across datasets but could prove especially useful when exploiting unlabelled data within the same domain for better performance on a small annotated subset. Still, further experiments are needed to verify this hypothesis.
Our empirical study comes with certain limitations. Although our results are obtained on various datasets and tasks, the pretraining and finetuning results are always subject to stochastic effects in the training processes. Further, we only focused on the most obvious modalityspecific information such as color and depth. The image reconstruction and depth prediction tasks only test some characteristics of the learned representations while other characteristics may be more indicative with respect to downstream finetuning performance. Still, our results show that the different pretraining strategies yield representations that encode different information.
Aligning feature representations across views and across modalities is a common approach in self-supervised learning. Yet, the effects on the learned visual representations are often not well understood. We showed quantitatively that, in contrast to cross-view representation alignment, cross-modal representation alignment can lead to discarding complementary information of the individual modalities. In the context of cross-modal learning on images and point clouds, texture is considered complementary information of the visual data and depth is redundant information. Pretraining by cross-modal alignment was especially useful for tasks that require a notion of the spatial layout, such as depth prediction, and also instance segmentation and object detection. This effect is similar to the observations of previous work that shape-biased networks are more robust than texture-biased networks. Thus, pretraining with emphasis on the redundant information achieved the most robust performance when finetuning the visual backbone networks on various downstream tasks and datasets.
Merging complementary information is a fundamental goal of sensor fusion and discarding that information may harm the overall performance on such a task. While our results have indicated that visual representations become more robust by reducing the texture-bias through crossmodal alignment, future work should investigate whether these results still hold for self-supervised approaches that fuse different sensing modalities.

Relation to results presented in Pri3D
To the best of our knowledge, all models in Pri3D [1] were pretrained with a depth loss in addition to the VIEW and GEO loss. This proxy depth loss follows the formulation presented in [2] and details are found in the official Pri3D implementation [3]. Note that this loss is also preceded by a linear per pixel layer that reduces the backbone output channels to a single channel (see Figure 1). We did not include this proxy loss, because our goal was to focus on the effects of cross-view and cross-modal representation alignment. Table 1 show the results of the Pri3D variations once the proxy depth loss is used. The relative performance change is given with respect to the corresponding model without the depth loss, which is not listed explicitly. While the depth loss does have positive effects on semantic segmentation of ScanNet and NYUv2, it is not beneficial on the object detection and instance segmentation tasks on the same datasets. On the other datasets and tasks it leads to mixed results.
Further, from what we understand, some models in [1] were pretrained with a learning rate of 0.1 and others with 0.01. Table 2 shows the effect of the learning rate change. Within the ScanNet dataset, the higer learning rate outperforms the lower learning rate, and vice versa on the other datasets. The pretraining learning rate thus seems to be a crucial factor that should be taken into account when evaluating models within or across datasets.  Performance change in %