Learning Distribution with VQ-VAE.
Vector quantization has been proved to be effective in lossy compression[1]}, [2]}.
Instead of designing a more sophisticated auto-encoder (AE) model, we utilize VQ-VAE[3]} to learn residual distribution. Intuitively, given a fixed storage space, for VQ-VAE, we could just store the vector index and when decoding, with the codebook we could utilize more diversity features. But for vanilla AE, we only have the fixed size features. Experiments in sub:modelablation suggest VQ-VAE's superiority over vanilla AE.
As the original image would contain much more spatial information, we choose to model the residual distribution given the original image, , \(p(r|x)\) .
