Neural Neural Textures Make Sim2Real Consistent
Unpaired image translation algorithms can be used for sim2real tasks, but
many fail to generate temporally consistent results. We present a new approach
that combines differentiable rendering with image translation to achieve
temporal consistency over indefinite timescales, using surface consistency
losses and \emph{neural neural textures}. We call this algorithm TRITON
(Texture Recovering Image Translation Network): an unsupervised, end-to-end,
stateless sim2real algorithm that leverages the underlying 3D geometry of input
scenes by generating realistic-looking learnable neural textures. By settling
on a particular texture for the objects in a scene, we ensure consistency
between frames statelessly. Unlike previous algorithms, TRITON is not limited
to camera movements -- it can handle the movement of objects as well, making it
useful for downstream tasks such as robotic manipulation.