Implementation Details Our model was trained in several stages. For all the experiments in this paper, we were using the ground truth instance segmentation as input, which could also be obtained by Mask R-CNN [1]}.
The image patches were generated from object bounding boxes and re-scaled to a fixed shape of \(256\times 256\)  pixels. For TransCG, we used AdamW optimizer [2]} for training with \(\lambda _{smooth} = 0.001\)  and the overall learning rate is \(0.001\)  to train the model till converge.
For U-Net, we used the Adam optimizer [3]} with a learning rate of \(1e^{-4}\)  to train the model until convergence.
For both surface normal estimation and depth completion, the batch size was set to 24 images.
The surface normal estimation and depth completion model were frozen during the training of the second stage.
