Given the single-view image and the associated mask, we would need to project the reconstructed 3D object to the observation space for computing \(\mathcal {L}_{inv}\) . However, such 3D-to-2D degradation is non-trivial.
Unlike existing image-based GAN inversion tasks where we can always assume pixel-wise image correspondence in the observation space, rendering 3D objects in the canonical frame onto the image space is explicitly controlled via camera poses. For real-world applications, unfortunately, perfect camera poses are not always available to guarantee such pixel-wise image correspondence.
While concurrently optimizing the latent code and the camera pose from scratch seems a plausible approach, this often suffers from camera-shape ambiguity [1]} and leads to erroneous reconstruction. To this end, we initialize the camera with a camera pose estimator (CMR [2]}), which can be potentially inaccurate, and jointly optimize it in the course of GAN inversion, for which we have:
\(\mathbf {z}^*,\mathbf {\pi }^* = _{\mathbf {z},\mathbf {\pi }} \mathcal {L}_{inv}(R(G(\mathbf {z}),\mathbf {\pi }),\mathbf {I}_{in}).\) 
