2. Learning semantically-meaningful depth features for depth-based perceptual similarity (Fig REF b).  Our trained Decoder (fMRI \(\mapsto \)  Depth, or fMRI \(\mapsto \)  RGBD) requires a similarity score/loss on the reconstructed depth maps. Perceptual Similarity [1]} has been shown to be a powerful metric for many image reconstruction tasks. However, that metric was developed for images only. We propose here a new perceptually/semantically meaningful similarity score for depth maps. This is obtained by training from scratch two VGG-like networks, trained on ImageNet for the task of object recognition, but from depth-based inputs: (i) An RGBD network that receives a 4-channel RGBD input (an image concatenated channel-wise with its estimated depth map), and (ii) A Depth-only network that is trained similarly and with the same architecture, but with a single-channel input containing only the depth map of an image. The goal of such training is not to improve object recognition, but rather to obtain a coarse-to-fine hierarchy of semantically-meaningful depth-related features (depth-only features, or RGBD features). Once trained, these features form the basis to our RGBD and Depth-only Perceptual Similarity metrics used for training and evaluating our reconstructions.
