Overall Model.
Among all current algorithms, AE have the highest throughput but the lowest compression ratio.
As suggested by many previous works, AE models cannot capture local features very well, while a local AR model is capable of doing so [1]}, [2]}.
Therefore an intuitive idea would be to use a small AR to capture local features, and AE for more global ones.
Since AR model hurts the parallelism and affect the throughput heavily, it must be very small.
In this work, a Three-Way Auto-regressive (TWAR) model is designed, where each pixel is predicted with only 3 other ones, reserving the ability to decompress in parallel.
For the AE model, we replace it with VQ-VAE [3]}, which is proven to be successful in image generation [3]}, [5]}, [6]}, [7]}, while the inference time is still similar to AE.
