EpiGRAF: Rethinking training of 3D GANs
A very recent trend in generative modeling is building 3D-aware generators
from 2D image collections. To induce the 3D bias, such models typically rely on
volumetric rendering, which is expensive to employ at high resolutions. During
the past months, there appeared more than 10 works that address this scaling
issue by training a separate 2D decoder to upsample a low-resolution image (or
a feature tensor) produced from a pure 3D generator. But this solution comes at
a cost: not only does it break multi-view consistency (i.e. shape and texture
change when the camera moves), but it also learns the geometry in a low
fidelity. In this work, we show that it is possible to obtain a high-resolution
3D generator with SotA image quality by following a completely different route
of simply training the model patch-wise. We revisit and improve this
optimization scheme in two ways. First, we design a location- and scale-aware
discriminator to work on patches of different proportions and spatial
positions. Second, we modify the patch sampling strategy based on an annealed
beta distribution to stabilize training and accelerate the convergence. The
resulted model, named EpiGRAF, is an efficient, high-resolution, pure 3D
generator, and we test it on four datasets (two introduced in this work) at
$256^2$ and $512^2$ resolutions. It obtains state-of-the-art image quality,
high-fidelity geometry and trains ${\approx} 2.5 \times$ faster than the
upsampler-based counterparts. Project website:
https://universome.github.io/epigraf.