VQ is motivated by the fundamental result of Shannon’s rate-distortion theory [1]}, [2]}: better performance can always be achieved by coding vectors instead of scalars, even if the sequence of source symbols are independent random variables. K-means [3]} is the prime method for VQ. The K-means algorithm clusters data by trying to separate the samples into \(n\)  groups of equal variance, this by minimizing the intra-class inertia. Despite the performance of this algorithm, K-means based VQ has an exponential complexity in encoding (computation and memory) and decoding (memory) [4]}. Various improvements to K-means have been proposed to address complexity issues (e.g., product quantization) [5]}, but their performance is suboptimal compared to some deep neural network (DNN) based approaches [4]}, where the idea is to map input data from the original high dimensional space to the DNN latent space with lower dimensionality, and apply K-means to the latent codes. Because K-means in the latent space is sub-optimal for VQ, [4]} proposed DeepVQ, a fully-DNN architecture for vector quantization, in the context of data compression. DeepVQ is an autoencoder that overcomes the complexity issue by directly mapping to the binary index of the codeword.
Recent works [8]}, [9]} proposed novel reparameterization methods to handle the non-gradient issue for discrete random variables in VAE. VQ-VAE [10]} avoids such problem by using the identity function, namely copying gradients from the decoder input to encoder output [11]}.
