RiCS: A 2D Self-Occlusion Map for Harmonizing Volumetric Objects
There have been remarkable successes in computer vision with deep learning.
While such breakthroughs show robust performance, there have still been many
challenges in learning in-depth knowledge, like occlusion or predicting
physical interactions. Although some recent works show the potential of 3D data
in serving such context, it is unclear how we efficiently provide 3D input to
the 2D models due to the misalignment in dimensionality between 2D and 3D. To
leverage the successes of 2D models in predicting self-occlusions, we design
Ray-marching in Camera Space (RiCS), a new method to represent the
self-occlusions of foreground objects in 3D into a 2D self-occlusion map. We
test the effectiveness of our representation on the human image harmonization
task by predicting shading that is coherent with a given background image. Our
experiments demonstrate that our representation map not only allows us to
enhance the image quality but also to model temporally coherent complex shadow
effects compared with the simulation-to-real and harmonization methods, both
quantitatively and qualitatively. We further show that we can significantly
improve the performance of human parts segmentation networks trained on
existing synthetic datasets by enhancing the harmonization quality with our
method.