Learning perceptually-meaningful Depth-based features.  
Part of the success of our method stems from
using perceptually-meaningful Depth-based features.
These features are used in two main ways within our method:
(i) They form the basis for our Depth-based Perceptual Similarity metric (explained next);
(ii) They form the main backbone of our Encoder
(a detailed description of our networks architecture is found in the Supplementary-Material).
These perceptually-meaningful depth-based features are learned as follows:
We customized the VGG architecture to accommodate a four-channel RGBD input, and another version that respects a single Depth-only input. We then train these networks from scratch on ImageNet for the task of object recognition (Fig REF b), similar to the way the original VGG network was trained.
To obtain the “ground-truth” depth data required for this training, we estimated the depth maps of all \(\sim \) 1.3M images in ImageNet classification challenge (ILSVRC) [1]} using MiDaS.
We denote the trained two depth-based object recognition networks as \(\varphi _{RGBD}\)  and \(\varphi _{D}\) .
Notably, the resulting object recognition accuracy of \(\varphi _{RGBD}\) 
was comparable with its RGB-only baseline (\(\sim \) 70% accuracy),
whereas the depth-only recognition network \(\varphi _{D}\)  achieved \(\sim \) 42% accuracy.
The goal, however, was not to improve object recognition, but rather to obtain a coarse-to-fine hierarchy of semantically-meaningful depth-related features.
