We consider the problem of information compression from high dimensional data. Where many studies consider the problem of compression by non-invertible trans- formations, we emphasize the importance of invertible compression. We introduce new class of likelihood-based auto encoders with pseudo bijective architecture, which we call Pseudo Invertible Encoders. We provide the theoretical explanation of their principles. We evaluate Gaussian Pseudo Invertible Encoder on MNIST, where our model outperform WAE and VAE in sharpness of the generated images. We consider the problem of information compression from high dimensional data. Where many studies consider the problem of compression by non-invertible transformations, we emphasize the importance of invertible compression as there are many cases where one cannot or will not decide a priori what part of the information is important and what part is not. Compression of images for person ID in a small company requires less resolution then person ID at an airport. To loose part of the information without harm to the future purpose of viewing the picture requires knowing the purpose upfront. Therefore, the fundamental advantage of invertible information compression is that compression can be undone if a future purpose so requires.Recent advances of classification models have demonstrated that deep learning architectures of proper design do not lead to information loss while still being able to achieve state-of-the-art in classification performance. These i-RevNet models BID5 implement a small but essential modification of the popular RevNet models while achieving invertibility and a performance similar to the standard RevNet BID2 . This is of great interest as it contradicts the intuition that information loss is essential to achieve good performance in classification BID13 . Despite the requirement of the invertibility, flow-based generating models BID0 ; BID11 ; BID6 demonstrate that the combination of bijective mappings allows one to transform the raw distribution of the input data to any desired distribution and perform the manipulation of the data.On the other hand, Auto-Encoders have provided the ideal mechanism to reduce the data to the bare minimum while retaining all essential information for a specific task, the one implemented in the loss function. Variational Auto Encoders (VAE) BID7 and Wasserstein Auto Encoders (WAE) BID14 are performing best. They provide an approach for stable training of autoencoders, which demonstrate good results at reconstruction and generation. However, both of these methods involve the optimization of the objective defined on the pixel level. We would emphasise the importance of avoiding the separate decoder part and training the model without relying on the reconstuction quality directly.Combining the best of Invertible mappings and Auto-Encoders, we introduce Pseudo Invertible Encoder. Our model combines bijectives with restriction and extension of the mappings to the dependent sub-manifolds FIG0 . The main contributions of this paper are the following:• We introduce new class of likelihood-based Auto-Encoders, which we call Pseudo Invertible Encoders. We provide the theoretical explanation of their principles.• We demonstrate the properties of Gaussian Pseudo Invertible Encoder in manifold learning.• We compare our model with WAE and VAE on MNIST, and report that the sharpness of the images, generated by our models is better. 2 RELATED WORK In this paper we have proposed the new class of Auto Encoders, which we call Pseudo Invertible Encoder. We provided a theory which bridges the gap between Auto Encoders and Normalizing Flows. The experiments demonstrate that the proposed model learns the manifold structure and generates sharp images.
New Class of Autoencoders with pseudo invertible architecture
{"src": "scitldr-AIC_train", "paper_id": "SkgiX2Aqtm"}