We propose a context-adaptive entropy model for use in end-to-end optimized image compression. Our model exploits two types of contexts, bit-consuming contexts and bit-free contexts, distinguished based upon whether additional bit
 allocation is required. Based on these contexts, we allow the model to more accurately estimate the distribution of each latent representation with a more generalized form of the approximation models, which accordingly leads to an
 enhanced compression performance. Based on the experimental results, the proposed method outperforms the traditional image codecs, such as BPG and JPEG2000, as well as other previous artificial-neural-network (ANN) based approaches, in terms of the peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) index. The test code is publicly available at https://github.com/JooyoungLeeETRI/CA_Entropy_Model. Recently, artificial neural networks (ANNs) have been applied in various areas and have achieved a number of breakthroughs resulting from their superior optimization and representation learning performance. In particular, for various problems that are sufficiently straightforward that they can be solved within a short period of time by hand, a number of ANN-based studies have been conducted and significant progress has been made. With regard to image compression, however, relatively slow progress has been made owing to its complicated target problems. A number of works, focusing on the quality enhancement of reconstructed images, were proposed. For instance, certain approaches BID4 BID17 BID24 have been proposed to reduce artifacts caused by image compression, relying on the superior image restoration capability of an ANN. Although it is indisputable that artifact reduction is one of the most promising areas exploiting the advantages of ANNs, such approaches can be viewed as a type of post-processing, rather than image compression itself.Regarding ANN-based image compression, the previous methods can be divided into two types. First, as a consequence of the recent success of generative models, some image compression approaches targeting the superior perceptual quality BID0 BID16 BID15 have been proposed. The basic idea here is that learning the distribution of natural images enables a very high compression level without severe perceptual loss by allowing the generation of image components, such as textures, which do not highly affect the structure or the perceptual quality of the reconstructed images. Although the generated images are very realistic, the acceptability of the machine-created image components eventually becomes somewhat applicationdependent. Meanwhile, a few end-to-end optimized ANN-based approaches (Toderici et al., 2017; BID1 BID18 , without generative models, have been proposed. In these approaches, unlike traditional codecs comprising separate tools, such as prediction, transform, and quantization, a comprehensive solution covering all functions has been sought after using end-to-end optimization. Toderici et al. (2017) 's approach exploits a small number of latent binary representations to contain the compressed information in every step, and each step increasingly stacks the additional latent representations to achieve a progressive improvement in quality of the reconstructed images. improved the compression performance by enhancing operation methods of the networks developed by Toderici et al. (2017) . Although Toderici et al. (2017) ; provided novel frameworks suitable to quality control using a single trained network, the increasing number of iteration steps to obtain higher image quality can be a burden to certain applications. In contrast to the approaches developed by Toderici et al. (2017) and , which extract binary representations with as high an entropy as possible, BID1 , BID18 , and regard the image compression problem as being how to retrieve discrete latent representations having as low an entropy as possible. In other words, the target problem of the former methods can be viewed as how to include as much information as possible in a fixed number of representations, whereas the latter is simply how to reduce the expected bit-rate when a sufficient number of representations are given, assuming that the low entropy corresponds to small number of bits from the entropy coder. To solve the second target problem, BID1 , BID18 , and adopt their own entropy models to approximate the actual distributions of the discrete latent representations. More specifically, BID1 and BID18 proposed novel frameworks that exploit the entropy models, and proved their performance capabilities by comparing the results with those of conventional codecs such as JPEG2000. Whereas BID1 and BID18 assume that each representation has a fixed distribution, introduced an input-adaptive entropy model that estimates the scale of the distribution for each representation. This idea is based on the characteristics of natural images in which the scales of the representations vary together in adjacent areas. They provided test results that outperform all previous ANN-based approaches, and reach very close to those of BPG BID3 , which is known as a subset of HEVC (ISO/IEC 23008-2, ITU-T H.265), used for image compression.One of the principle elements in end-to-end optimized image compression is the trainable entropy model used for the latent representations. Because the actual distributions of latent representations are unknown, the entropy models provide the means to estimate the required bits for encoding the latent representations by approximating their distributions. When an input image x is transformed into a latent representation y and then uniformly quantized intoŷ, the simple entropy model can be represented by pŷ(ŷ), as described by . When the actual marginal distribution ofŷ is denoted as m(ŷ), the rate estimation, calculated through cross entropy using the entropy model, pŷ(ŷ), can be represented as shown in equation FORMULA0 , and can be decomposed into the actual entropy ofŷ and the additional bits owing to a mismatch between the actual distributions and their approximations. Therefore, decreasing the rate term R during the training process allows the entropy model pŷ(ŷ) to approximate m(ŷ) as closely as possible, and let the other parameters transform x into y properly such that the actual entropy ofŷ becomes small. DISPLAYFORM0 In terms of KL-divergence, R is minimized when pŷ(ŷ) becomes perfectly matched with the actual distribution m(ŷ). This means that the compression performance of the methods essentially depends on the capacity of the entropy model. To enhance the capacity, we propose a new entropy model that exploits two types of contexts, bit-consuming and bit-free contexts, distinguished according to whether additional bit allocation is required. Utilizing these two contexts, we allow the model to more accurately estimate the distribution of each latent representation through the use of a more generalized form of the entropy models, and thus more effectively reduce the spatial dependencies among the adjacent latent representations. FIG0 demonstrates a comparison of the compression results of our method to those of other previous approaches. The contributions of our work are as follows:• We propose a new context-adaptive entropy model framework that incorporates the two different types of contexts.• We provide the test results that outperform the widely used conventional image codec BPG in terms of the PSNR and MS-SSIM.• We discuss the directions of improvement in the proposed methods in terms of the model capacity and the level of the contexts.Note that we follow a number of notations given by because our approach can be viewed as an extension of their work, in that we exploit the same rate-distortion (R-D) optimization framework. The rest of this paper is organized as follows. In Section 2, we introduce the key approaches of end-to-end optimized image compression and propose the context-adaptive entropy model. Section 3 demonstrates the structure of the encoder and decoder models used, and the experimental setup and results are then given in section 4. Finally , in Section 5, we discuss the current state of our work and directions for improvement. Based on previous ANN-based image compression approaches utilizing entropy models BID1 BID18 , we extended the entropy model to exploit two different types of contexts. These contexts allow the entropy models to more accurately estimate the distribution of the representations with a more generalized form having both mean and standard deviation parameters. Based on the evaluation results, we showed the superiority of the proposed method. The contexts we utilized are divided into two types. One is a sort of free context, containing the part of the latent variables known to both the encoder and the decoder, whereas the other is the context, which requires additional bit allocation. Because the former is a generally used context in a variety of codecs, and the latter was already verified to help compression using 's approach, our contributions are not the contexts themselves, but can be viewed as providing a framework of entropy models utilizing these contexts.Although the experiments showed the best results in the ANN-based image compression domain, we still have various studies to conduct to further improve the performance. One possible way is generalizing the distribution models underlying the entropy model. Although we enhanced the performance by generalizing the previous entropy models, and have achieved quite acceptable results, the Gaussian-based entropy models apparently have a limited expression power. If more elaborate models, such as the non-parametric models of or BID11 , are combined with the context-adaptivity proposed in this paper, they would provide better results by reducing the mismatch between the actual distributions and the approximation models. Another possible way is improving the level of the contexts. Currently, our methods only use low-level representations within very limited adjacent areas. However, if the sufficient capacity of the networks and higher-level contexts are given, a much more accurate estimation could be possible. For instance, if an entropy model understands the structures of human faces, in that they usually have two eyes, between which a symmetry exists, the entropy model could approximate the distributions more accurately when encoding the second eye of a human face by referencing the shape and position of the first given eye. As is widely known, various generative models BID5 BID13 BID25 learn the distribution p(x) of the images within a specific domain, such as human faces or bedrooms. In addition, various in-painting methods BID12 BID22 BID23 learn the conditional distribution p(x | context) when the viewed areas are given as context. Although these methods have not been developed for image compression, hopefully such high-level understandings can be utilized sooner or later. Furthermore, the contexts carried using side information can also be extended to some high-level information such as segmentation maps or any other information that helps with compression. Segmentation maps, for instance, may be able to help the entropy models estimate the distribution of a representation discriminatively according to the segment class the representation belongs to.Traditional codecs have a long development history, and a vast number of hand-crafted heuristics have been stacked thus far, not only for enhancing compression performance, but also for compromising computational complexities. Therefore, ANN-based image compression approaches may not provide satisfactory solutions as of yet, when taking their high complexity into account. However, considering its much shorter history, we believe that ANN-based image compression has much more potential and possibility in terms of future extension. Although we remain a long way from completion, we hope the proposed context-adaptive entropy model will provide an useful contribution to this area. The structure of the hybrid network for higher bit-rate environments. The same notations as in the figure 4 are used. The representation y is divided into two parts and quantized. One of the resulting parts,ŷ 1 , is encoded using the proposed model, whereas the other,ŷ 2 , is encoded using a simpler model in which only the standard deviations are estimated using side information. The detailed structure of the proposed model is illustrated in FIG3 . All concatenation and split operations are performed in a channel-wise manner.
Context-adaptive entropy model for use in end-to-end optimized image compression, which significantly improves compression performance
{"src": "scitldr-AIC_train", "paper_id": "HyxKIiAqYQ"}