Depth Refinement. Instead of using three-step MVS depth estimation as described for the finest resolution level (stage 3), we directly up-sample the depth estimated in previous stage (from resolution \(\frac{W}{2} \times \frac{H}{2}\)  to \(W \times H\) ) and refine the upsampled depth with the RGB image. Following MSG-Net [1]} and PatchMatchNet [2]}, we implement a depth residual network to perform refinement. The network uses a 2D CNN to output a residual from the upsampled depth. This residual depth is then added to the up-sampled depth, to get the refined depth map \(D^{ref}\) .
