Neural networks currently dominate the machine learning community and they do so for good reasons. Their accuracy on complex tasks such as image classification is unrivaled at the moment and with recent improvements they are reasonably easy to train. Nevertheless, neural networks are lacking robustness and interpretability. Prototype-based vector quantization methods on the other hand are known for being robust and interpretable. For this reason, we propose techniques and strategies to merge both approaches. This contribution will particularly highlight the similarities between them and outline how to construct a prototype-based classification layer for multilayer networks. Additionally, we provide an alternative, prototype-based, approach to the classical convolution operation. Numerical results are not part of this report, instead the focus lays on establishing a strong theoretical framework. By publishing our framework and the respective theoretical considerations and justifications before finalizing our numerical experiments we hope to jump-start the incorporation of prototype-based learning in neural networks and vice versa.
The definition of a network directly on the discrete cluster representation of a VQ layer was first mentioned in the VQ-Variational-Autoencoder (VQ-VAE) @cite_10 . There, the output of the VQ is a map where a pixel has the integer number @math of the closest prototype. Moreover, the output is defined as the latent space of the VQ-VAE. The method was trained via the gradient-straight-through method. The obtained results are promising. Nevertheless, the authors were unable to train the model from scratch even with soft-to-hard assignment proposed in @cite_75 . We made similar observations like in @cite_10 for our simulations performed so far. If we train using soft-to-hard assignments, the model is able to invert the continuous relaxations. We also observed that training by a simple gradient-straight-through approximation is not stable, when the model is trained from scratch. However, our proposed gradient approximation is working well for many performed settings, either to train from scratch or on a pre-trained network.
@cite_10 @cite_75
2963799213 2964164354
Learning useful representations without supervision remains a key challenge in machine learning. In this paper, we propose a simple yet powerful generative model that learns such discrete representations. Our model, the Vector Quantised-Variational AutoEncoder (VQ-VAE), differs from VAEs in two key ways: the encoder network outputs discrete, rather than continuous, codes; and the prior is learnt rather than static. In order to learn a discrete latent representation, we incorporate ideas from vector quantisation (VQ). Using the VQ method allows the model to circumvent issues of posterior collapse'' -— where the latents are ignored when they are paired with a powerful autoregressive decoder -— typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes, providing further evidence of the utility of the learnt representations. We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.