Transformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsics

The advent of autonomous driving and advanced driver assistance systems necessitates continuous developments in computer vision for 3D scene understanding. Self-supervised monocular depth estimation, a method for pixel-wise distance estimation of objects from a single camera without the use of ground truth labels, is an important task in 3D scene understanding. However, existing methods for this task are limited to convolutional neural network (CNN) architectures. In contrast with CNNs that use localized linear operations and lose feature resolution across the layers, vision transformers process at constant resolution with a global receptive field at every stage. While recent works have compared transformers against their CNN counterparts for tasks such as image classification, no study exists that investigates the impact of using transformers for self-supervised monocular depth estimation. Here, we first demonstrate how to adapt vision transformers for self-supervised monocular depth estimation. Thereafter, we compare the transformer and CNN-based architectures for their performance on KITTI depth prediction benchmarks, as well as their robustness to natural corruptions and adversarial attacks, including when the camera intrinsics are unknown. Our study demonstrates how transformer-based architecture, though lower in run-time efficiency, achieves comparable performance while being more robust and generalizable.


INTRODUCTION
There have been rapid improvements in scene understanding for robotics and advanced driver assistance systems (ADAS) over the past years. This success is attributed to the use of Convolutional Neural Networks (CNNs) within a mostly encoder-decoder paradigm. Convolutions provide spatial locality and translation invariance which has proved useful for image analysis tasks. The encoder, often a convolutional Residual Network (ResNet) (He et al., 2016), learns feature representations from the input and is followed by a decoder which aggregates these features and converts them into final predictions. However, the choice of architecture has a major impact on the performance and generalizability of the task.
While CNNs have been the preferred architecture in computer vision, transformers have also recently gained traction  motivated by their success in natural language processing (Vaswani et al., 2017). Notably, they have also outperformed CNNs for object detection (Cara Equal contribution ion et al., 2020) and semantic segmentation (Zheng et al., 2021). This is also reflected in methods for monocular dense depth estimation, a pertinent task for autonomous planning and navigation, where supervised transformer-based methods (Li et al., 2020;Ranftl et al., 2021) have been proposed as an alternative to supervised CNN-based methods Aich et al., 2021). However, supervised methods require extensive RGB-D ground truth collected from costly LiDARs or multi-camera rigs. Instead, self-supervised methods have increasingly utilized concepts of Structure from Motion (SfM) with known camera intrinsics to train monocular depth and ego-motion estimation networks simultaneously (Guizilini et al., 2020;Lyu et al., 2020;Chawla et al., 2021). While transformer ingredients such as attention have been utilized for self-supervised depth estimation (Johnston and Carneiro, 2020), most methods are nevertheless limited to the use of CNNs that have localized linear operations and lose feature resolution during downsampling to increase their limited receptive field (Yang et al., 2021). We readapt modules from Dense Prediction Transformer (DPT) and Monodepth2 to be trained with appearance-based losses for self-supervised monocular depth, ego-motion, and intrinsics estimation. ductive biases allow for more globally coherent predictions with different layers attending to local and global features simultaneously . However, transformers require more training data and can be more computationally demanding (Caron et al., 2021). While multiple studies have compared transformers against CNNs for tasks such as image classification (Raghu et al., 2021;Bhojanapalli et al., 2021), no study exists that evaluates the impact of transformers in self-supervised monocular depth estimation, including when the camera intrinsics may be unknown.
In this work, we conduct a comparative study between CNN-and transformer-based architectures for self-supervised monocular depth estimation. Our contributions are as follows: • We demonstrate how to adapt vision transformers for self-supervised monocular depth estimation by implementing a method called Monocular-Transformer SfMLearner (MT-SfMLearner).
• We compare MT-SfMLearner and CNNs for their performance on the KITTI monocular depth Eigen Zhou split (Eigen et al., 2014) and the online depth prediction benchmark (Geiger et al., 2013).
• We investigate the impact of architecture choices for the individual depth and ego-motion networks on performance as well as robustness to natural corruptions and adversarial attacks.
• We also introduce a modular method that simultaneously predicts camera focal lengths and principal point from the images themselves and can easily be utilized within both CNN-and transformerbased architectures.
• We study the accuracy of intrinsics estimation as well as its impact on the performance and robustness of depth estimation.
• Finally, we also compare the run-time computational and energy efficiency of the architectures for depth and intrinsics estimation.
MT-SfMLearner provides real-time depth estimates and illustrates how transformer-based architecture, though lower in run-time efficiency, can achieve comparable performance as the CNN-based architectures while being more robust under natural corruptions and adversarial attacks, even when the camera intrinsics are unknown. Thus, our work presents a way to analyze the trade-off between the performance, robustness, and efficiency of transformer-and CNN-based architectures for depth estimation.
For supervised monocular depth estimation, Dense Prediction Transformer (DPT) (Ranftl et al., 2020) uses ViT as the encoder with a convolutional decoder and shows more coherent predictions than CNNs due to the global receptive field of transformers. TransDepth (Yang et al., 2021) additionally uses a ResNet projection layer and attention gates in the decoder to induce the spatial locality of CNNs for supervised monocular depth and surface-normal estimation. Lately, some works have inculcated elements of transformers such as self-attention (Vaswani et al., 2017) in self-supervised monocular depth estimation (Johnston and Carneiro, 2020;Xiang et al., 2021). However, there has been no investigation of transformers to replace the traditional CNN-based methods (Godard et al., 2019;Lyu et al., 2020) for self-supervised monocular depth estimation.
Moreover, self-supervised monocular depth estimation still requires prior knowledge of the camera intrinsics (focal length and principal point) during training, which may be different for each data source, may change over time, or be unknown a priori (Chawla et al., 2020). While multiple approaches to supervised camera intrinsics estimation have been proposed (Lopez et al., 2019;Zhuang et al., 2019), not many self-supervised approaches exist (Gordon et al., 2019). Therefore, we investigate the impacts of transformer architectures on self-supervised monocular depth estimation for their performance, robustness, and run-time efficiency, even when intrinsics are unknown.

METHOD
Our objective is to study the effect of utilizing vision transformers for self-supervised monocular depth estimation in contrast with the contemporary methods that utilize Convolutional Neural Networks.
Given a set of n images from a video sequence, we simultaneously train depth and ego-motion prediction networks. The inputs to the networks are a sequence of temporally consecutive RGB image triplets {I −1 , I 0 , I 1 } ∈ R H×W ×3 . The depth network learns the model f D : R H×W ×3 → R H×W to output dense depth (or disparity) for each pixel coordinate p of a single image. Simultaneously, the ego-motion network learns the model f E : R 2×H×W ×3 → R 6 to output relative translation (t x ,t y ,t z ) and rotation (r x , r y , r z ) forming the affine transformation RT 0 1 ∈ SE(3) between a pair of overlapping images. The predicted depthD and ego-motionT are linked together via the perspective projection model, that warps the source images I s ∈ {I −1 , I 1 } to the target image I t ∈ {I 0 }, with the camera intrinsics described by K. This process is called view synthesis, as shown in Figure 1. We train the networks using the appearance-based photometric loss between the real and synthesized target images, as well as a smoothness loss on the depth predictions (Godard et al., 2019).

Architecture
Here we describe Monocular Transformer Structure from Motion Learner (MT-SfMLearner), our transformer-based method for self-supervised monocular depth estimation ( Figure 1).

Depth Network
For the depth network, we readapt the Dense Prediction Transformer (DPT) (Ranftl et al., 2020) for self-supervised learning, with a DeiT-Base  in the encoder. There are five components of the depth network: • An Embed module, which is a part of the encoder, takes an image I ∈ R H×W ×3 , and converts non-overlapping image patches of size p × p into where d = 768 for DeiT-Base. This is implemented as a large p × p convolution with stride s = p where p = 16. The output from this module is concatenated with a readout token of the same size as the remaining tokens.
• The Transformer block, that is also a part of the encoder, consists of 12 transformer layers which process these tokens with multi-head selfattention (MHSA) (Vaswani et al., 2017) modules. MHSA processes inputs at constant resolution and can simultaneously attend to global and local features. The subscripts of DN refer to the transformer layer from which the respective Reassemble module takes its input (see Figure  1). Input image size is H ×W , p refers to the patch size, N p = H ·W /p 2 refers to the number of patches from the image, and d refers to the feature dimension of the transformer features.
• Four Reassemble modules in the decoder, which are responsible for extracting image-like features from the 3 rd , 6 th , 9 th , and 12 th (final) transformer layers by dropping the readout token and concatenating the remaining tokens in 2D. This is followed by pointwise convolutions to change the number of channels, and transpose convolution in the first two reassemble modules to upsample the representations (corresponding to T 3 and T 6 in Figure 1). To make the transformer network comparable to its convolutional counterparts, we increase the number of channels in the pointwise convolutions of the last three Reassemble modules by a factor of 4 with respect to DPT. The exact architecture of the Reassemble modules can be found in Table 1.
• Four Fusion modules in the decoder, based on RefineNet (Lin et al., 2017). They progressively fuse information from the Reassemble modules with information passing through the decoder, and upsample the features by 2 at each stage. Unlike DPT, we enable batch normalization in the decoder as it was found to be helpful for selfsupervised depth prediction. We also reduce the number of channels in the Fusion block to 96 from 256 in DPT.
• Four Head modules at the end of each Fusion module to predict depth at 4 scales following previous self-supervised methods (Godard et al., 2019). Unlike DPT, the Head modules use 2 convolutions instead of 3 as we found no difference in performance. For further details of the Head modules, refer to Table 2.

Ego-Motion Network
For the ego-motion network, we adapt DeiT-Base  in the encoder. Since the input to the transformer for the ego-motion network consists of two images concatenated along the channel dimension, we repeat the embedding layer accordingly.
We use a Reassemble module to pass transformer tokens to the decoder. For details on the structure of this Reassemble module, refer to Table 1. We adopt the decoder for the ego-motion network from Mon-odepth2 (Godard et al., 2019). When both depth and ego-motion networks use transformers as described above, we refer to the resulting architecture as Monocular Transformer Structure from Motion Learner (MT-SfMLearner).

Appearance-based Losses
Following contemporary self-supervised monocular depth estimation methods, we adopt the appearancebased losses and an auto-masking procedure from the CNN-based Monodepth2 (Godard et al., 2019) for the above described transformer-based architecture as well. We employ a photometric reprojection loss composed of the pixel-wise 1 distance and the Structural Similarity (SSIM) between the real and synthesized target images, along with a multi-scale edgeaware smoothness loss on the depth predictions. We also use auto-masking to disregard the temporally stationary pixels in the image triplets. Furthermore, we reduce texture-copy artifacts by calculating the total loss after upscaling the depths, predicted at 4 scales, from intermediate decoder layers to the input resolution.

Intrinsics
Accurate camera intrinsics given by are essential to self-supervised depth estimation as can be seem from Equation 1. However, the intrinsics may vary within a dataset with videos collected from different camera setups, or over a long period of time. These parameters can also be unknown for crowdsourced datasets. We address this by introducing an intrinsics estimation module. We modify the ego-motion network, which takes a pair of consecutive images as input, and learns to estimate the focal length and principal point along with the translation and rotation. Specifically, we add a convolutional path in the ego-motion decoder to learn the intrinsics. The decoder features before activation from the penultimate layer are passed through a global average pooling layer, followed by two branches of pointwise convolutions to reduce the number of channels from 256 to 2. One branch uses a softplus activation to estimate focal lengths along x and y axes as the focal length is always positive. The other branch doesn't use any activation to estimate the principal point as it has no such constraint. Note that the ego-motion decoder is the same for both the convolutional as well as transformer architectures. Consequently, the intrinsics estimation method can be modularly utilized with both architectures. Figure 1 demonstrates MT-SfMLearner with learned intrinsics.

RESULTS
In this section, we perform a comparative analysis between our transformer-based method, MT-SfMLearner, and the existing CNN-based methods for self-supervised monocular depth estimation. We also perform a contrastive study on the architectures for the depth and ego-motion networks to evaluate their impact on the prediction accuracy, robustness to natural corruptions and adversarial attacks, and the run-time computational and energy efficiency. Finally, we analyze the correctness and run-time efficiency of intrinsics predictions, and also study its impact on the accuracy and robustness of depth estimation.

Dataset
We report all results on the Eigen Split (Eigen et al., 2014) of KITTI (Geiger et al., 2013) dataset after removing the static frames as per (Zhou et al., 2017), unless stated otherwise. This split contains 39, 810 training, 4424 validation, and 697 test images, respectively. All results are reported on the per-image scaled dense depth prediction without post-processing (Godard et al., 2019), unless stated otherwise.

Training Settings
The networks are implemented in PyTorch (Paszke et al., 2019) and trained on a TeslaV100 GPU for 20 epochs at a resolution of 640 × 192 with batchsize 12. MT-SfMLearner is further trained at 2 more resolutions -416 × 128 and 1024 × 320, with batchsizes of 12 and 2, respectively for experiments in Section 4.2. The depth and ego-motion encoders are initialized with ImageNet (Deng et al., 2009) pre-trained weights. We use the Adam (Kingma and Ba, 2014) optimizer for CNN-based networks (in Sections 4.3 and 4.4) and AdamW (Loshchilov and Hutter, 2017) optimizer for transformer-based networks with initial learning rates of 1e −4 and 1e −5 , respectively. The learning rate is decayed after 15 epochs by a factor of 10. Both optimizers use β 1 = 0.9 and β 2 = 0.999.

Depth Estimation Performance
First we compare MT-SfMLearner, where both depth and ego-motion networks are transformer-based, with the existing fully convolutional neural networks for their accuracy on self-supervised monocular depth estimation. Their performance is evaluated using metrics from (Eigen et al., 2014) up to a fixed range of 80 m as shown in Table 3. We compare the methods at different input image sizes clustered into categories of Low-Resolution (LR), Medium-Resolution (MR), and High-Resolution (HR). We do not compare against methods that use ground-truth semantic labels during training. All methods assume known groundtruth camera intrinsics.
We observe that MT-SfMLearner is able to achieve comparable performance at all resolutions under error as well as accuracy metrics. This includes methods that also utilize a heavy encoder such as ResNet-101 (Johnston and Carneiro, 2020) and Pack-Net (Guizilini et al., 2020).
Online Benchmark: We also measure the performance of MT-SfMLearner on the KITTI Online Benchmark for depth prediction 1 using the metrics from (Uhrig et al., 2017). We train on an image size of 1024 × 320, and add the G2S loss (Chawla et al., 2021) for obtaining predictions at metric scale. Results ordered by their rank are shown in Table 4. The performance of MT-SfMLearner is on par with stateof-the-art self-supervised methods, and outperforms

Contrastive Study
We saw in the previous section that MT-SfMLearner performs competently on independent and identically distributed (i.i.d.) test set with respect to the stateof-the-art. However, networks that perform well on an i.i.d. test set may still learn shortcut fea-  (Geirhos et al., 2020). Since self-supervised monocular depth estimation networks concurrently train an ego-motion network (see Equation 1), we investigate the impact of each network's architecture on depth estimation. We consider both Convolutional (C) and Transformer (T) networks for depth and ego-motion estimation. The resulting four combinations of (Depth Network, Ego-Motion Network) architectures are (C, C), (C, T), (T, C), and (T, T), ordered on the basis of their increasing influence of transformers on depth estimation. To compare our transformer-based method fairly with convolutional networks, we utilize Monodepth2 (Godard et al., 2019) with a ResNet-101 (He et al., 2016) encoder. All four combinations are trained thrice using the settings described in Section 4.1 for an image size of 640 × 192. All combinations assume known ground-truth camera intrinsics.

Performance
For the four combinations, we report the best performance on i.i.d. in Table 5, and visualize the depth predictions for the same in Figure 2. The i.i.d. test set used for comparison is same as in Section 4.2.
We observe from Table 5 that the combination of transformer-based depth and ego-motion networks i.e MT-SfMLearner performs best under two of the error metrics as well as two of the accuracy metrics. The remaining combinations perform comparably on all the metrics.
From Figure 2, we observe more uniform estimates for larger objects like vehicles, vegetation, and  buildings when depth is learned using transformers. Transformers are also less affected by reflection from windows of vehicles and buildings. This is likely because of the larger receptive fields of the self-attention layers, which lead to more globally coherent predictions. On the other hand, convolutional networks produce sharper boundaries, and perform better on thinner objects such as traffic signs and poles. This is likely because of the inherent inductive bias for spatial locality present in convolutional layers.

Robustness
While the different combinations perform comparably on the i.i.d. dataset, they may differ in robustness and generalizability. Therefore, we study the impact of natural corruptions and adversarial attacks on the depth performance using the following: • Natural corruptions. Following (Hendrycks and Dietterich, 2019) and ( (Madry et al., 2018) at attack strength ε ∈ {0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0}. The gradients are calculated with respect to the training loss. Following (Kurakin et al., 2016), the perturbation is accumulated over min(ε + 4, 1.25 · ε ) iterations with a step-size of 1. When the test image is from the beginning or end of a KITTI sequence, the appearance-based loss is only calculated for the feasible pair of images.
Inspired by (Wong et al., 2020) Figure 4 further shows a general improvement in adversarial robustness when learning either depth or ego-motion with transformers instead of convolutional networks. Finally, Figure 5 shows an improvement in robustness to symmetrically flipped adversarial attacks when depth is learned using transformers instead of convolutional networks. Furthermore, depth estimation is most robust when both depth and ego-motion are learned using transformers.

RMSE
Therefore, MT-SfMLearner, where both depth and ego-motion and learned with transformers, provides the highest robustness and generalizability, in line with the studies on image classification (Paul and Chen, 2021;Bhojanapalli et al., 2021). This can be attributed to their global receptive field, which allows for better adjustment to the localized deviations by accounting for the global context of the scene.

Intrinsics
Here, we evaluate the accuracy of our proposed method for self-supervised learning of camera intrinsics and its impact on the depth estimation performance. As shown in Table 6, the percentage error for intrinsics estimation is low for both convolutional and transformer-based methods trained on an image size of 640 × 192. Moreover, the depth error as well accuracy metrics are both similar to when the ground truth intrinsics are known a priori. This is also observed in Figure 6 where the learning of intrinsics causes no artifacts in depth estimation.
We also evaluate the models trained with learned intrinsics on all 15 natural corruptions as well as on PGD and symmetrically flipped adversarial examples. We report the mean RMSE (µRMSE) across all corruptions in Table 7. The RMSE for depth estimation on adversarial examples generated by PGD method for all strengths is shown in Figure 7. The mean RMSE (µRMSE) across all attack strengths for horizontally flipped and vertically flipped adversarial ex-    amples is shown in Table 8. We observe that the robustness to natural corruptions and adversarial attacks is maintained by both architectures when the intrinsics are learned simultaneously. Furthermore, similar to the scenario with known ground truth intrinsics, MT-SfMLearner with learned intrinsics has higher robustness than its convolutional counterpart.

Efficiency
We further compare the networks on their computational and energy efficiency to examine their suitabil-  ity for real-time applications.
In Table 9, we report the mean inference speed in frames per second (fps) and the mean inference energy consumption in Joules per frame for depth and intrinsics estimation for both architectures. These metrics are computed over 10, 000 forward passes at a resolution of 640 × 192 on an NVidia GeForce RTX 2080 Ti.
Both architectures run depth and intrinsics estimation in real-time with an inference speed > 30 fps. However, the transformer-based method consumes higher energy and is computationally more de- manding than its convolutional counterpart.

CONCLUSION
This work is the first to investigate the impact of transformer architecture on the SfM inspired self-supervised monocular depth estimation. Our transformer-based method MT-SfMLearner performs comparably against contemporary convolutional methods on the KITTI depth prediction benchmark. Our contrastive study additionally demonstrates that while CNNs provide local spatial bias, especially for thinner objects and around boundaries, transformers predict uniform and coherent depths, especially for larger objects due to their global receptive field. We observe that transformers in the depth network result in higher robustness to natural corruptions, and transformers in both depth, as well as the ego-motion network, result in higher robustness to adversarial attacks. With our proposed approach to self-supervised camera intrinsics estimation, we also demonstrate how the above conclusions hold even when the focal length and principal point are learned along with depth and ego-motion. However, transformers are computationally more demanding and have lower energy efficiency than their convolutional counterparts. Thus, we contend that this work assists in evaluating the trade-off between performance, robustness, and efficiency of self-supervised monocular depth estimation for selecting the suitable architecture.