3. Handling the insufficient fMRI training data (Fig REF cd).  To train our Decoder (fMRI \(\mapsto \)  Depth, or fMRI \(\mapsto \)  RGBD), we used a moderate-size fMRI dataset – `fMRI on ImageNet' [1]}. However, the train-set of this dataset contains only 1200 pairs of images with their corresponding fMRI recordings (this small number is typical of fMRI datasets). Such a small number of examples cannot span the huge space of natural images (nor their corresponding depth maps), resulting in a poor generalization of the Decoder to fMRIs of never-before-seen images.
To overcome this problem, we generalize the self-supervised approach of [2]} to accommodate for depth information.
This allows us to train our decoder on many (50,000) additional natural images with their estimated depth maps – images for which there are no fMRI recordings.
Specifically, our semi-supervised approach employs a two-phase training: The first phase is supervised, and focuses on training of an Encoder, \(Enc\)  – to map images and their depth maps to their corresponding fMRI recordings. This is done using the 1200 “paired” examples from the fMRI dataset, along with their MiDaS-estimated depth maps (Fig REF c). In the second phase, we train the Decoder, \(Dec\) , using two objectives: (i) supervised training, to map the limited “paired” fMRI recordings to their corresponding images and depth maps (Fig REF d1), and (ii) self-supervised cycle-consistent training on a very large number of “unpaired” natural images with their corresponding depth maps. This cycle-consistency is facilitated using the auxiliary pretrained Encoder from the first phase (Fig REF d2).
The Decoder loss uses our above-mentioned depth-based Perceptual Similarity between the reconstructed and the “ground-truth” depth maps. Using this metric ensures that the reconstructed images and their reconstructed depth maps are perceptually and semantically meaningful (well beyond their pixel-level similarity).
