Predicting marginal heatmaps.
We design our approach upon marginal heatmaps [1]}, [2]} and first predict three 2D heatmaps from our monocular input. Figure REF  summarizes our model. We first process the event-frame input with a backbone to extract intermediate representations. In particular, we adopt a ResNet-34 [3]} which is cut after the second residual block. The feature extractor initialization is a critical design choice of our approach and we experimentally ablate possible alternatives in Section REF , where we compare different initialization and pre-training strategies and provide evidence of the benefits of RGB-to-events transfer learning.
The main model consists in three branches, one for each marginal projection (\(xy\) , \(zy\) , and \(xz\) ). Each branch is further made of three stages (Figure REF (a)), each consisting in a hourglass-like CNN, as detailed in Figure REF (b). For each stage we compute an intermediate loss. The result of each stage is also aggregated (summation) with the previous output to feed the next stage, in a residual-like fashion. According to [4]}, intermediate losses help alleviating the problem of vanishing gradients.
<FIGURE>