CroMo: Cross-Modal Learning for Monocular Depth Estimation

Learning-based depth estimation has witnessed recent progress in multiple directions; from self-supervision using monocular video to supervised methods offering highest accuracy. Complementary to supervision, further boosts to performance and robustness are gained by combining information from multiple signals. In this paper we systematically investigate key trade-offs associated with sensor and modality design choices as well as related model training strategies. Our study leads us to a new method, capable of connecting modality-specific advantages from polarisation, Time-of-Flight and structured-light inputs. We propose a novel pipeline capable of estimating depth from monocular polarisation for which we evaluate various training signals. The inversion of differentiable analytic models thereby connects scene geometry with polarisation and ToF signals and enables self-supervised and cross-modal learning. In the absence of existing multimodal datasets, we examine our approach with a custom-made multi-modal camera rig and collect CroMo; the first dataset to consist of synchronized stereo polarisation, indirect ToF and structured-light depth, captured at video rates. Extensive experiments on challenging video scenes confirm both qualitative and quantitative pipeline advantages where we are able to outperform competitive monocular depth estimation method.


Introduction
Modern vision sensors are able to leverage a variety of light properties for optical sensing. Common RGB sensors, for instance, use colour filter arrays (CFA) over a pixel sensor grid to separate incoming radiation into specified wavebands. This allows a photosensor to detect wavelengthseparated light intensity and enables the acquisition of familiar visible spectrum images. Wavelength is however only one property of light capable of providing information.
Light polarisation defines another property and describes the oscillation direction of an electromagnetic wave. While the majority of natural light sources (e.g. the sun) emit unpolarised light, consisting of a mixture of oriented oscillations, surface reflection from non-metallic objects can  input. Uni-modal model training of p2d [11] and the monodepth2 architecture [22], compared with cross-modal training (Ours).
linearly polarise the light. Such polarised light then contains surface structure information, retrievable using analytic physical models [8]. This information can be used to harness the depth cues offered by this light property. Polarimetric imagery is a passive example for depth estimation. Passive sensors have acceptable resolution and dense depth however there exist well understood capture situations that prove challenging (e.g. textureless surface regions).
However further known properties of light (i.e. speed) provide yet more information. Indirect Time-of-Flight (i-ToF) cameras are active light sensors and use a pulsed, near infrared light source to measure object and surface distance. Further active sensors use structured-light and these emit known infrared patterns and use stereoscopic imaging to measure the distance to the surface. While i-ToF and structured-light cameras have clear advantages, such as the ability to function in low-light scenarios and good short range precision, they are susceptible to specular reflections, ambient light and range remains limited.
We argue that novel combination of active and passive light sensors offers new possibilities. We can exploit such a combination to take advantage of the discussed, modalityspecific strengths and weaknesses. We observe that (1) differing visual modalities offer information cues about com-plimentary aspects of the world and (2) there exist clear trade-offs between the complexity of capture sensor setups and the resulting data diversity and quality, accessible for supervision signals. This motivates us to systematically investigate these considerations and provide insight into training data capture design decisions and the related pay-offs. Our study results in the proposal of a framework capable of exploiting available supervision signals and is tailored to benefit from the particular strengths of unique modalities.
We instantiate our ideas by bringing together the physical understanding of Polarisation and i-ToF in a data driven fashion. In practice this affords an inference pipeline that estimates depth from a single polarisation image. We train a convolutional neural network (CNN), with cross-modal fusion using differentiable physical models. We establish a dataset comprising Ground Truth depth obtained via Multi-View Stereo (MVS) reconstruction [52] that enjoys access to information rich, full video sequences. We carry out extensive experimental work to establish the efficacy of our proposed monocular depth estimation strategies. Our contributions can be summarised as: 1. Novel multi-modal method. We propose a multi-modal training approach that allows for monocular depth estimation from polarisation images. We propose (i) differentiable analytic formulae that define modal-specific loss terms, (ii) cross-modal consistency joint-training towards improved real-world depth estimation from a single polarisation image, (iii) architectural components that increase predicted depth sharpness (see Fig. 1). 2. CroMo dataset and training modalities study. We provide a systematic analysis of the benefits afforded when multiple image modalities are available at training time, for monocular depth estimation. Investigation and exposure of improvements are enabled by the unique Cross-Modality video dataset 1 . Our multiple hardwaresynchronized cameras capture, for the first time, stereo polarisation (Pol), indirect Time-of-Flight (i-ToF) and structured-light images from active sensing. The remaining sections of the paper are thus organised: Sec. 2 provides brief review of depth estimation with respect to relevant modalities and previous work considering multiple information signals. Sec. 3 presents our model capable of monocular depth estimation from polarisation imagery and our cross-modal training procedure. In Sec. 4 we introduce CroMo, our novel multi-modal dataset, Sec. 5 reports experimental work validating our contributions and Sec. 6 provides discussion and future research issues.

Related Work
To the best of our knowledge this is the first work to study end-to-end monocular depth inference, utilising cross-modal information from Time-of-Flight (i-ToF), ac-1 Dataset is available at: https://cromo-data.github.io/ tive stereo and polarisation modalities during training. We briefly review the literature most closely related to the main components of our investigation and proposed framework.

Monocular depth estimation
Estimating depth from a single image constitutes a hard, ill-posed problem. Pioneering work on supervised monocular depth estimation [42] used synthetic samples during training. Synthetic data was also previously used in conjunction with stereo network distillation [25] for this task. To improve accuracy and convergence speed, [17] introduce a spatially-increasing discretisation. However, acquiring ground truth depth data remains a difficult task [20].
To overcome the difficulty of collecting accurate ground truth signal, multiple works [19,61] investigate a consistency loss by leveraging stereo imagery during training, towards self-supervision. While being undoubtedly path-breaking, the initial methods suffered from a nondifferentiable sampling step. Godard et al. [21] formulated a fully-differentiable pipeline with left-right consistency checks during training and have also explored the temporal components [22], even in challenging setups such as night scenes [55]. These methods predict depth with RGB input, while we utilise polarisation images. Monocular Polarisation Previous work use monocular polarisation imagery to recover depth. One route to overcome Shape from Polarisation (SfP) ambiguities is to use orthographic camera models to express polarisation intensity in terms of depth [64]. Atkinson et al. [8] compute depth without knowing the light direction through a non-linear optimization framework and yet assume fully diffuse surfaces. Linear systems have also been constructed for the task [53] by adding shape from shading equations. While theoretically interesting, the orthographic assumption has restricted their application to synthetic lab environments. Learning based Polarisation Due to lack of reasonablysized datasets, only a limited number of works focus on learning with polarisation. Ba et al. [9] provide polarisation images together with a set of plausible inputs from a physical model to estimate surface normals. The work of [34] apply polarisation for instance segmentation of transparent objects and [37] learn de-glaring of images with semitransparent objects. Recently, Blanchon et al. [11] extended the work of [22] with complementary polarimetric cues. In contrast to them, we invert a physical model to enable self-supervision through consistency cycles and additionally study the benefit of co-modal i-ToF information. Learning based i-ToF i-ToF sensors acquire distance information by estimating the time required for an emitted light pulse to be reflected [65]. Sensors measure either the time (direct) or the phase (indirect) difference between emitted and received light. The modality enjoys high precision for short range distances [27], yet suffers from limited spatial resolution and noise [15], which constitute challenging fac-tors for any learning-based approach. Obtaining reliable signals from specular surfaces is difficult and inherent Multi Path Interference (MPI), often manifests as noisy measurements and artifacts. Synthetic training is also explored for raw i-ToF input data in end-to-end fashion [5,24,57]. However, the ability to account for real world domain shifts is limited. In [4] a GAN is employed towards addressing such domain adaption issues on a limited dataset. i-ToF depth improvement MPI can be considered a critical issue and error mitigation has been the focus of a body of work [60]. Two-path approximations [23] have been used within optimization schemes [14,35] and multiple frequencies are used to constrain the problem [16]. Kadambi et al. [33] propose a hardware solution to address scenes with translucent objects and a number of scholars incorporate light transport information to correct for MPI [2,26,44,46].

Depth with multiple sensors
Depth completion has been carried out via combining multiple input modalities, for example, a sparse but accurate LiDAR signal in combination with RGB [58]. It is difficult to address sparse signals with CNNs [41] and LiDAR sensors can produce problematic artifacts resulting in unreliable Ground Truth depth estimates [39]. One strategy towards removing dependence on this form of supervision are self-supervision cues however these fall behind supervised pipelines in terms of accuracy [40]. i-ToF and x Confidence-based combination of i-ToF depth and classical RGB stereo is explored with the network architecture of [3] and a semi-supervised approach for this combination is explored by [47] in a generic framework. While these approaches improve upon the individual depth estimates, they rely on a late fusion paradigm. Son et al. [54] use a robotic arm to collect 540 real training images of short range scenes with structured light ground truth.
By inserting micro linear polarizers [45] in front of a photo-receiver chip, Yoshida et al. [62] build an i-ToF sensor capable of acquiring both i-ToF depth and polarisation scene cues. Combination of both the absolute depth (i-ToF) and relative shape (polarisation cues) allowed reconstruction of depth for specular surfaces. While this pipeline requires i-ToF and polarisation input to solve an optimization problem, we alternatively explore cross-modal selfsupervised training and single image inference. Depth from multi-view Polarisation Another route to predict depth is the use of more than one polarisation image [7] which enables methods based on physical models. An RGB+Polarisation pair can provide sharp depth maps with stereo vision [66]. Other methods [12] use more than two polarisation images. Despite the sharpness of the results, the difficulty to acquire multi-view polarisation images is still a major hurdle. Atkinson et al. [6] combine polarisation methods with photometric stereo. Two images of a scene, from an identical view point yet with dif-ferent light exposures, are leveraged. An extension dealing with mixed reflectivity is established via a combined photometric-polarisation linear system in [38] and Garcia et al. [18] solve for polarisation normals using circularly polarised light. Traditional multi-view methods also benefit from polarisation. Miyazaki et al. [43] recover surfaces of black objects using polarisation physics and space carving. Depth refinement with Polarisation Consumer depth estimation tools progress significantly in recent years however their predictions are noisy and lack details. Using polarisation cues, [32] enhance sharper depth maps from RGBD cameras by differentiating their depth maps to resolve polarisation ambiguities and perform mutual optimization.
Despite clear improvements in monocular depth estimation methods, their performance remains bounded by the chosen modality hence calling for multi-modal depth estimation. Our method alleviates this problem with a learning based approach. During training we leverage complementary modalities such that our model can compensate the drawbacks of the single modality used at inference time.

Method
Our multi-modal monocular depth investigation leads to a new model architecture that accounts explicitly for prediction blur and introduces two novel analytic losses. We discuss these components in the following sections.

Architecture
Our architecture employs multiple encoder-decoder networks illustrated in Fig. 3a. We observe that monocular depth estimation methods often incur blurry image predictions and we address this problem by introducing architectural components that account for prediction blur. Firstly convolutions in our encoders are coupled with gated convolution. Our network then composes a traditional U-Net [51] with skip connections and the gated convolutions [63]. The encoder utilises a ResNet [28] style block, while the decoder is a cascade of convolutions with layer resizing.
Secondly, drawing on the fact that Displacement Fields (DF) can be utilised to aid sharpness [49], we estimate a DF using a self-supervised sharpening decoder. Depth pixels with strong local disparity have values redefined to mirror a nearest neighbour that does not exhibit strong local disparity. Groundtruth (GT) displacement fields can thus be defined for each predicted depth during training ("on-thefly"), guiding our dedicated displacement field prediction. We inspect predicted depth with and without our DF strategy and observe significant improved sharpness, most evident when employing 3D visualisations (Fig. 4).

Loss Formulation
Our study considers multiple modalities and various sensor configurations at training time. We explore several loss terms to exploit our unique setting (see Tab   in our training procedure are enabled through both coordinate frame projections (P1, P2) and analytic transforms (A1, A2) of individual network (N1, N2) outputs (see Figs. 3a, 3b). We firstly process input modalities individually using distinct networks. These ingest i-ToF correlation and left polarisation images respectively and output initial depth maps. We propose two analytic losses, derived from properties of i-ToF and polarisation, to train and link the networks. We train the i-ToF module without ground truth and also leverage the available multi-modal information through image recovery via related analytical formulae (A1, A2). Strategically similar to previous work [13], at inference time we require only a single modality (in our case polarisation), and can discard network N1 completely.
Terms L corr→pol and L corr evaluate discrepancies between each input image and respective recovered images, obtained using auxiliary and final depth maps (Fig. 3b). Our individual branches share information through the loss term L corr→pol . Explicitly, we recover a polarisation image from an auxiliary depth map and then project this, using projection P1, to the polarisation sensor frame of reference via the final depth map D pol . Finally our third loss term L stereo is used to train the polarisation network (N2) by comparing the right polarisation image, projected using the predicted depth D pol , with the left polarisation image. We next provide details of our analytical formulae for image recovery and the loss terms that enable our training procedure.
Depth to polarisation (A2) Polarisation cameras capture polarised intensity along directions ϕ pol . The measured intensity is given by [66] where ϕ pol is the polariser angle, i un is the intensity of unpolarised light, ρ is the degree of linear polarisation and φ is the Angle of Polarisation (AoP). The polarisation parameters ρ ∈ {ρ s , ρ d } and φ ∈ {φ s , φ d } depend on local reflection type, either diffuse (d) or specular (s) as follows: with θ ∈ [0, π/2] the viewing angle and η the object refractive index, typically 1.5, and if the pixel is diffuse if the pixel is specular The π-ambiguity is denoted as [π] in (3), and α denotes the azimuth angle of the surface normal n. Azimuth angle α and viewing angle θ are obtained as with v the viewing vector defined as the vector pointing toward the camera center from the 3D point P (x, y) corresponding to pixel (x, y) with depth d(x, y) and n the outward pointing normal vector, defined as the cross product of the partial derivatives with respect to x and y [66]: with f x , f y , c x , c y the camera intrinsics.
Hence, from a given depth map d, one can compute the azimuth angle α and the viewing angle θ using (4) and (5), followed by the polarisation parameters ρ and φ with (2) and (3). The polarisation images for diffuse and specular surfaces I diffuse pol and I specular pol are finally recovered using the calculated polarisation parameters with (1).
Depth to correlation (A1) Indirect ToF measures the correlation between a known emitted signal and the received signal. The emitted signal at frequency f M is a sinusoid: and the signal, reflected by the scene, is of the form [30] f where the τ is the time delay between the emitted signal g(t) and the reflected signal f (t). The i-ToF measurement c(x) is the correlation between the two signals: where we only consider the direct reflection signal and ignore the multipath interference (MPI) and sensor imperfections. We are interested in the phase ϕ, proportional to the depth d between the objects in the scene and the sensor: where c is the speed of light and [2π] represents the 2π-ambiguity. Using the four bucket strategy [36] to sample can be obtained to recover the phase ϕ, the amplitude α and the intensity β.
Hence, from a given depth d, one can compute the phase ϕ using (9) and then reformulate the recovered i-ToF correlation using (10), (11) and (12) in turn to form I corr .
Stereo loss L stereo This loss requires that left and right image pairs are accessible during training. While only the left image I l is fed to the network, the right image I r can guide the model towards generating valid depth, and vice versa. More formally, let K l and K r be camera matrices with intrinsic parameters for left and right images respectively, and D a depth map on the left reference frame. Let T left→right denote the transformation that moves 3D points from the left coordinate system to the right. An image coordinate transformed from left coordinate p l to the right image is A backward differentiable warping [31] is used to reproject an image onto the left view as I right→left .
We form a stereo loss L stereo , and related mask loss L mask similarly to [22], which aid network training and deal with occluded pixels respectively as (15) where the photometric error is similar to [22]: Analytical losses L corr and L corr−→pol Depth D corr is firstly inferred directly from i-ToF correlation input, and then two recovered images I corr and I pol are formed. Recovered images represent the 'ideal' input for each modality, i-ToF and polarisation respectively, conditioned on the inferred depth. Since I pol is generated from D corr , we reproject it using D pol to form a recovered final polarisation image I i corr −→ D pol pol , i ∈ {diffuse, specular}. In each case, discrepancies between the recovered image and the true input image provide a strong indication of the quality of the generated depth.
We use this signal to guide the network. Formally where I l is the left polarisation image, I corr −→ D pol pol the recovered polarisation image aligned to I l , I corr the i-ToF correlation input, and I corr the recovered correlation image. We use a min operator for L corr→pol to lift the problem of classifying a pixel as diffuse or specular by computing both possibilities and letting the network select the best solution.
Finally, following [59], we use an additional loss L struct in the objective function, derived from structured-light information (see appendix for further detail).
In summary, depending on the input modalities available at training time, we can add or remove the introduced losses L corr , L corr→pol , L stereo and L struct as appropriate. We explicitly note that hyper parameter tuning for balancing of these loss terms is not required, due to our formulation.
Our total loss L can then be formulated as: where L DF is the L 2 norm between predicted and GT DF.

Data
We next provide details on our custom camera rig (Sec. 4.1) and CroMo dataset (Sec. 4.2), comprising synchronised image sequences capturing multiple modalities, at video-rate across real-world indoor and outdoor scenes.

Camera capture rig
Our prototype custom-camera hardware rig is shown in Fig. 5. Our rig is constructed in order to capture synchronised data across multiple modalities including stereo polarisation, i-ToF correlation, structured-light depth and IMU. We rigidly mount two polarisation cameras (Lucid PHX050S-QC) providing a left-right stereo pair, an i-ToF camera (Lucid HLS003S-001) operating at 25Mhz and a camera (RealSense D435i) for active IR stereo capture. All devices are connected with a hardware synchronisation wire resulting in time-aligned video capture at a frame rate of 10fps. The left polarisation camera is the lead camera which generates the genlock signal and defines the world reference frame. Accurate synchronisation was validated using a flash-light torch and was further confirmed by the respectable quality observed from stereo Block Matching results [29]. The focus of all sensors is set to infinity, the aperture to maximum, and the exposure is manually fixed at the beginning of each capture sequence. The calibration on all four cameras' extrinsics, intrinsics, and distortion coefficients is done with a graph bundle-adjustment for improved multi-view calibration (see appendix for further details).

CroMo dataset
We collect a unique dataset comprising multi-modal captures such that each time point pertains to measurement of (1) Polarisation: raw stereo polarisation cameras produce 2448×2048 px stereo images. (2) i-ToF: 4 channel 640×480 px correlation images. (3) Depth: a structuredlight capture of the scene resulting in a 848×480 px depth image. In addition to the three main sensors, IMU information is recorded to further enable future research directions. Our dataset consists of more than 20k frames, totalling >80k images of indoor and outdoor sequences in challenging conditions, with no constraints on minimum or maximum scene range. We group these sequences into four different scenes which we name: Kitchen, Station, Park and Facades. Despite the multitude of senors, operating ranges are not unlimited and our data collection also does not cover all possible scenarios; we further discuss limitations in our appendix. We report statistics per captured scene in Tab. 1 (lower). These statistics characterise our scene captures and provide useful information, e.g., that the median scene depth differs greatly between indoor (Kitchen) and outdoor (Station, Park and Facades) scenes. This is a strong indicator for whether the i-ToF sensor will perform well. Tab. 1 (upper) provides a comparison with other depth datasets showing that CroMo is the first publicly available, modality rich dataset containing a large quantity of image data.

Experiments
Our experimental design evaluates (1) the effect of multiple modalities, accessible at training time, for monocular depth estimation and (2) the effect that changing network architecture has on depth quality, under consistent input signal. Our capture setup allows us to employ a standard MVS approach [52] on full temporal sequences of polarisation intensity frames (left-camera), to serve as ground-truth depth for our experimental work. This expensive offline optimisation leverages accordances amongst all frames per sequence, affording high quality depth to evaluate our ideas.
Multi-modal training We firstly evaluate combinations of training input signal by changing the number of sensors available to the model. We fix network encoder-decoder backbone components (i.e. ResNet50, analogous to [22]) and train models that leverage cues from a maximum of four sensors; left and right polarisation, i-ToF correlation and structured-light. We show predicted depth improvements, attainable by systematic addition of sensors, and quantify where best gains can be made. The training signal components used for our monocular depth estimation experiments are as follow: Temporal (M) extracts information from video sequences (3 frames), Stereo (S) uses stereo images, i-ToF (T) leverages i-ToF correlations via our two interconnected depth branches (see Sec. 3.2). Finally, Structured-light (L) incorporates an additional mask into the objective function, derived from information provided by our structured-light sensor. The structured-light signal is utilised only when the mask improves the projection loss. We explore alternative strategies to exploit the structured-light signal and discuss details on practical benefits (e.g. convergence speed) in the appendix.
Introduced signal components define our set of training experiments. For example, Stereo and Structure Light (SL) train the model using self-supervised stereo (S) and structured-light (L) information. Experiments therefore use differing subsets of the introduced loss terms (see Tab. 3). Qualitative results are shown in Fig. 6. Unsurprisingly, self-supervised stereo (S) is relatively blurry and struggles to capture fine details, such as the thin, metallic arch on the   Figure 6. Same ResNet50 architecture as in [22] with different modalities: each new modality closes the gap with GT.
of-parameters at inference (MP) and the giga-multiplyaccumulates per second (GMACs) in order to evaluate size and compute-cost per architecture. Architectures consist of the ResNet18 U-Net used in [22] and their supplementary material ResNet50 variant, the p2d architecture [11] using ResNet50 with a different data representation (Stokes), the MiDaS [50] architecture and Ours (see Sec. 3.1, Fig. 3a).
Qualitative results are shown in Fig. 7. It may be observed that the ResNet18 architecture with smallest (MP) fails to obtain good background detail of the swing frame structure (Station sample) or of the tree (Park sample). The ResNet50 variant slightly improves detail, especially with raw measurements instead of Stokes (p2d [11]). Even when increasing network capacity c. three-fold with MiDaS, results are unsatisfying. Our proposed architecture (Ours) requires smaller capacity and computation for a sharper reconstruction of the swing and the tree. We disentangle the benefits of additional sensor modalities from our model contributions, highlighting the advantage of gated convolutions and our DF-based approach towards reducing blur.
Quantitative results are reported in Tab. 2. The smallest architecture ResNet18 [22] performs worst. The larger U-Net ResNet50 performs better, and has been generally adopted [11,21]. Note p2d [11] uses a different data representation (Stokes) for polarisation cf . ResNet50; performance decreases. We believe the Stokes representation, us-

Station sample Park sample
GT Label

Ours
MiDaS [50] p2d [11] ResNet50 [22] ResNet18 [22] Pol. Intensity ing angle directly, is more sensitive to noise and not appropriate for an SSIM loss with the self-supervised stereo (S) training strategy. MiDaS [50] provides second best performance and yet necessitates roughly ×2 GMACs. Our architecture provides best performance while remaining relatively compact which we largely attribute to gated convolutions and displacement field estimation (see Sec. 3.1).

Conclusion
We systematically investigate the effect of using additional information from co-modal sensors at training time, for the task of monocular depth estimation from polarisation imagery. Our exploration is enabled through a unique multimodal video dataset which constitutes synchronized images from binocular polarisation, raw i-ToF and structured-light depth. We quantify the beneficial influence of both passive and active sensors, leveraging self-supervised and crossmodal learning strategies that lead to the proposal of a new method providing sharper and more accurate depth estimation. This is made possible through two physical models that describe the relationships between polarisation and surface normals on one side and correlation measures and scene depth on the other. We believe that our fundamental investigation of modality combination and the CroMo dataset can accelerate research of both spatial and temporal fusion, towards advancing cross-modal computer vision.

CroMo: Supplementary Material
We provide additional materials to supplement our main paper. In Sec. 7.1 we provide observations on the properties of light polarisation. Sec. 7.2 states the specifics for our surface normal estimation process. In Sec. 7.3, we provide additional details for our multi-view camera calibration procedure, Sec. 7.4 provides some further modelling details and finally Sec. 7.5 gives supplementary information on our network architectures and learning parameters. Most natural light sources emit unpolarized light that only becomes polarized if reflected. Hence the type of reflection, illustrated in Fig. S1, either diffuse (d) or specular (s), influences the characteristics of the reflected polarized light. More specifically, the reflective surface influences the relation between the normals' parameters (θ, α) and the polarisation parameters (ρ, φ), defined in Eq. 2 and 3 of the main paper.

Surface normals
In the main manuscript we estimate polarisation intensity using the varying coordinates of surface normals. Hence, the computation of these normals, derived from network depth prediction, plays an important role in the training process. To increase the robustness of estimated normals, we compute the cross products using four distinct pairs of orthogonal directions as in [S4]: The weighted average of these normals is calculated using weights w i where: The final surface normal (unnormalized) is then estimated by their linear combination: Weights w i result in neighbouring pixels of i un that contain strong color disparity, to be down-weighted in the normal computation. This follows from the assumption that such pixels are more likely to represent different objects. Conversely, if neighbouring pixels possess similar color, they are more likely to correspond to the same object and their associated partial derivatives are more likely to provide normals that accurately describe the observed object shape.

Graph-based bundle adjustment
As discussed in Sec. 4.1 of our main paper the calibration of extrinsics, intrinsics and distortion coefficients, for all four capture-rig cameras, is achieved using a graph-based bundle-adjustment [S2] that improves multi-view calibration. We provide here further details of our multi-view calibration procedure.
We start with well established calibration methods [S1] to obtain the intrinsics K k and distortion coefficients d k for each camera C k , where k ∈ {0, 1, 2, 3}. We use a standard pinhole camera model and define C 0 as the left polarisation camera, C 1 the right polarisation camera, C 2 the i-ToF camera, and C 3 the structure light camera. We use five parameters for the distortion coefficients and collect n images of a calibration checkerboard, from all cameras synchronously. In practice we move the checkerboard in front of the cameras while keeping the camera rig stationary. We attempt to cover as wide a field-of-view as possible for all four cameras. We find it is more important to thoroughly cover and account for the extremities of the individual images as opposed to attempting to be visible to all cameras simultaneously. Further, we estimate the rigid transformation for each camera pair composed of C 0 (our world reference), and camera C k in turn, where k ∈ {1, 2, 3}. This provides the extrinsics T k→0 = [R k→0 t k→0 ] for camera C k (with T 0→0 = [I 0]).
These initial intrinsic, extrinsic parameter values and the distortion coefficients are however sub-optimal as they are obtained by solving successive sub-optimisation problems. Towards improving the multi-camera calibration, we define the reprojection error of points X j on the image I i for the camera C k as Where T i 0 is the position of camera C 0 for image i, and x i j is the distorted 2D point from the projection function π(·) which projects a 3D Point X j visible by the camera C k at position T i 0 · T k→0 with distortion coefficients d k and intrinsic parameters K k on image I i . The function dist(·) defines the robustified distance between 2D points, i.e. a Huber m-estimator, and x i j is the 2D point detected on the checkerboard with a corner detector corresponding to the 3D point X j in image I i . The indicator function 1 x i j ∈Ii defines whether the 2D Point x i j is visible in image I i .
Finally, we used a graph-based bundle-adjustment [S2] to model the global problem, for all cameras C k , jointly as: with T 0 0 fixed to [I 0] in order to properly constrain the gauge freedom. All camera calibration parameters are initialised using the values obtained from the original individual calibrations.
This formalism, borrowed from the SLAM community [S3], allows us to optimize all parameters, i.e. the intrinsic, the extrinsic and the distortion parameters for all cameras, jointly. We find the global optimisation process is able to improve our calibration RMSE by ∼5-10%.

Additional modelling details 7.4.1 Reflection ambiguities
A diffuse-specular ambiguity initially exists in our formulation; pertaining to diffuse or specular reflection (see Eq.3, main paper). This ambiguity is addressed during training via the min operator found in Eq.18. We propose to resolve reflection ambiguities (per-pixel) by minimization of the SSIM loss between respective {specular, diffuse} images and the input image, towards consistently providing a valid training signal. Secondly, the azimuthal π-ambiguity is directly accounted for by the formulation of Eq.1; the inherent cos(·) modulation nullifies ambiguity found in its input (2φ component of the argument) and thus supervision is not adversely affected due to φ being modulo π.

Wrappings ambiguities
An analytical solution exists for the correlation to depth transform and a wrapping ambiguity remains. However we highlight that a reconstructed depth, although "phase wrapped", is still able to provide reliable surface normals that can be used to produce (1) the degree of linear polarisation and (2) the Angle of Polarisation, for both diffuse and specular surfaces. Once these are projected to the N 2 referential, this information is used in conjunction with the brightness of the left polarisation image to render valid "Recovered polarisation" images, (see Fig 3b of our main paper).

Polarization intensity recovery
To render the intensity, we require the brightness of each pixel (i un in Eq.1). We obtain the brightness of the polarisation image by channel-wise summing of the left polarisation input pixel values. Two images are rendered following Eq.1; for both the cases of diffuse and specular images. We use a binary mask to select values, pixel-wise, from either the specular or diffuse image. The mask selects pixels such that the minimum SSIM loss between the {specular, dif-fuse} image and the input image are retained.
The two images formed therefore constitute only an intermediary step towards producing a final image. We use a binary mask to then select values, pixel-wise, from either the specular or diffuse image to form a new image containing the pixels that retain the minimum SSIM loss between the {specular, diffuse} image and the input image (i.e. the min in Eq.18 is per pixel). We thus form a final image that contains both specular and diffuse components.

Correlation image rendering from depth
Analogous to the Polarization image strategy, we use the input correlation image (obtaining α, β estimates), in addition to depth information, to estimate both the ambient β and reflectance α, for correlation reconstruction. We provide additional description for the network architecture that we propose in order to process the considered input modalities. Instances of this architecture are depicted as edges 'N 1' and 'N 2' in the system design; see Fig 3a of our main paper.
We employ a standard U-Net architecture, similar to our baseline [22], including skip connections. The encoder is based on a 'Resnet' [28] style block, with the original convolutional layer replaced by gated convolution [63]. The size of the input images are 512×544×12 for polarisation and 640×480×4 for i-ToF, respectively.

Training parameters
To aid reproducibility, we report training parameters and hyperparameters. We use identical training parameters and align with our baseline [22] where possible. We use the Ranger optimiser [S5] and batch sizes of 8, a learning rate of 1e−4 with an exponential learning rate decay. We train all considered methods for 50 epochs.

Comparison with RGB input
A direct comparison with RGB input forms a relevant and interesting line of enquiry. Our custom capture rig does not currently accommodate this modality directly. However, towards investigating this experimentally, we did transform the polarisation input frame to an RGB frame by considering the polarisation intensity of each RGB channel, individually. We note that this is not directly equivalent to an RGB sensor since the bayer pattern differs. We actively decided not to include this experimental work in the main paper to avoid misinterpretation and confusion. Preliminary work evaluating our Polarisation input cf . the noted "Polarisation-converted-to-RGB" showed improvements using Polarisation (RMSE 1.4) over "Polarisationconverted-to-RGB" (RMSE 1.53).

Controlling for capture environment
We note that the i-ToF modality excels in indoor environments, however these represent a relatively smaller portion of our dataset. To corroborate this, we report an experiment that considers our various training strategies, tested on only an indoor environment (Kitchen). The addition of i-ToF (from (S) to (ST)), at training time, significantly improves the predicted depth in this restricted setting (see Tab. S1).  Table S1. Test on Kitchen scene (780 frames): additional sensors can be observed to improve performance. The largest improvement comes from the addition of the i-ToF, (from (S) to (ST)), in an exclusively indoor test setting.

Further analysis of where additional sensors help
We include preliminary further analysis with respect to investigation of scenarios where additional sensors help. We include an example that highlights two points (see Fig. S2).

Additional structured light experiments
The structured light sensor present in our camera rig offers low-noise signal which we find can also be leveraged in a supervised fashion, directly. For completeness, we compare the resulting depth estimation when supervising directly with structure-light (D) and our approach, using unsupervised signals (STLM). The structure-light signal, obtained from our Realsense sensor, is claimed reliable up to a 10 meters range according to the constructor [1]. We thus further investigate by evaluating performance over distinct 0 − 10m and 0 − 20m ranges. Results in Tab. S2 show that the fully supervised method (D) can offer similar performance to our approach (STLM) in the range 0 − 10m yet performance degrades by significant margins when considering the more challenging 0 − 20m range. This highlights the benefits of our unsupervised multi-modal strategy (STLM); leveraging information from multiple sensor sources and an ability to learn to adapt when a particular sensor results in low quality measurement, due to unsuitable physical conditions (e.g. structured light in the 10 − 20m range).

Additional details on the L struct loss
We select to use a structured light loss similar to the loss proposed in [59]. We find that such indirect supervision of  Table S2. Comparison of training strategies for two depth prediction ranges. Our training strategy (STLM) works well in spite of the operational limits of particular sensors.
the structure light signal allows to automatically select the best source of information, particularly in situations where the structure light signal fails or becomes unreliable (as discussed in Sec. 7.5.6). Formally, given a depth from the structured light D struct , the loss reads: We make use of an additional L 1 loss between predicted depth and the D struct depth, when L struct is minimal (see Eq.19 in the main manuscript).

Limitations and Societal Impact
Limitations We note distinct limitations that relate to our sensor setup. Active sensors have limited range and areas of operativity e.g. i-ToF often offers weaker performance outdoors, structure light sensors are of limited range, and polarisation sensors sacrifice spacial sampling resolution for spectral sampling resolution. Our multi-modal ideas attempt to combat these limitations indirectly however we remain bound by the physical laws of light.
Additionally, our current hardware setup is operable by a single person, and yet data capture is currently more cumbersome than e.g. use of a modern smartphone. Training data collection, that involves the acquisition of multiple modalities, currently induces a somewhat larger investment of effort over monomodal capture. With the argument being that the cost may then be recouped when assessing monocular inference time performance. Our hardware rig constitutes a research prototype and form factor likely improves as camera evolution results in further reductions in sensor size, weight and cost.
Finally we would note that our current dataset does not yet capture all possible scenarios and represents but a subset of urban scenes where depth estimation can prove valuable. Future capture sessions will look to enrich and widen the recorded capture scenarios, towards increasing the value of the data resource that we provide to the community.
Societal Impact We note that our proposed CroMo dataset was collected by only two human operators in urban environments. While care was taken towards objective scene capture, such collected data may yet reflect the biases of human operators; influencing specific content, scenarios or capture setups. Efforts towards the reduction of bias, introduced by manual human operators, might suggest mounting of the system on automatic vehicles in future. Additional ideas, toward mitigation of the axis of bias relating to manual data capture, can be considered an interesting future research direction.