Neural Network Assisted Depth Map Packing for Compression Using Standard Hardware Video Codecs
Depth maps are needed by various graphics rendering and processing
operations. Depth map streaming is often necessary when such operations are
performed in a distributed system and it requires in most cases fast performing
compression, which is why video codecs are often used. Hardware implementations
of standard video codecs enable relatively high resolution and framerate
combinations, even on resource constrained devices, but unfortunately those
implementations do not currently support RGB+depth extensions. However, they
can be used for depth compression by first packing the depth maps into RGB or
YUV frames. We investigate depth map compression using a combination of depth
map packing followed by encoding with a standard video codec. We show that the
precision at which depth maps are packed has a large and nontrivial impact on
the resulting error caused by the combination of the packing scheme and lossy
compression when bitrate is constrained. Consequently, we propose a variable
precision packing scheme assisted by a neural network model that predicts the
optimal precision for each depth map given a bitrate constraint. We demonstrate
that the model yields near optimal predictions and that it can be integrated
into a game engine with very low overhead using modern hardware.