We solve the depth prediction as learning to optimize the dense stereo matching problem [1]}, [2]}, [3]}, [4]}. Given the generated cost volume \(C_0\)  as in Sec. REF , the depth estimation of the reference image, is formulated as finding the best solution \(D^* = \operatorname{argmin} _{D} \,E(D, C_0)\) , which minimizes an energy function \(E(D, C_0)\)  (including a data term and a smoothness term). Unfortunately, such a global minimization is NP-complete due to many discontinuity preserving energies [5]}, and approximate solutions are proposed by loosing the energy function, e.g., the binocular stereo matching solved by Semi-Global Matching (SGM)[2]}. In SGM, the matching cost \(C_0\)  is iteratively aggregated by summing the costs (of all 1D minimum cost paths that end in
pixel \(\mathbf {p}\)  at disparity \(d\)With an abuse of notation, \(d\)  denotes both a depth value in MVS and a disparity value in binocular stereo.) when traversing from pixel \(\mathbf {p} - \mathbf {r}\)  to pixel \(\mathbf {p}\)  in a direction \(\mathbf {r}\)  (out of sixteen directions) and the best disparity at each pixel \(\mathbf {p}\)  is given by \({ d^{*}(\mathbf {p})=\operatorname{argmin} _{d} (C^\prime (\mathbf {p},d))}\) , with \(C^\prime \)  being the aggregated cost volume. Similar to SGM, we do not directly optimize the energy function \(E\) , but learn to process the cost volume \(C_0\) .
