Recurrent models for sequences have been recently successful at many tasks, especially for language modeling
 and machine translation. Nevertheless, it remains challenging to extract good representations from
 these models. For instance, even though language has a clear hierarchical structure going from characters
 through words to sentences, it is not apparent in current language models.
 We propose to improve the representation in sequence models by
 augmenting current approaches with an autoencoder that is forced to compress
 the sequence through an intermediate discrete latent space. In order to propagate gradients
 though this discrete representation we introduce an improved semantic hashing technique.
 We show that this technique performs well on a newly proposed quantitative efficiency measure.
 We also analyze latent codes produced by the model showing how they correspond to
 words and phrases. Finally, we present an application of the autoencoder-augmented
 model to generating diverse translations. Autoencoders have a long history in deep learning BID3 BID10 BID16 BID7 . In most cases, autoencoders operate on continuous representations, either by simply making a bottleneck BID3 , denoising BID16 , or adding a variational component BID7 . In many cases though, a discrete latent representation is potentially a better fit.Language is inherently discrete, and autoregressive models based on sequences of discrete symbols yield impressive results. A discrete representation can be fed into a reasoning or planning system or act as a bridge towards any other part of a larger system. Even in reinforcement learning where action spaces are naturally continuous, Metz et al. (2017) show that discretizing them and using autoregressive models can yield improvements.Unluckily, using discrete latent variables is challenging in deep learning. And even with continuous autoencoders, the interactions with an autoregressive component cause difficulties. Despite some success BID1 BID17 , the task of meaningfully autoencoding text in the presence of an autoregressive decoder has remained a challenge.In this work we present an architecture that autoencodes a sequence s of N discrete symbols from any vocabulary (e.g., a tokenized sentence), into a K-fold (we test K = 8 and K = 32) compressed sequence c(s) of Since gradient signals can vanish when propagating over discrete variables, the compression function c(s) can be hard to train. To solve this problem, we draw from the old technique of semantic hashing BID11 . There, to discretize a dense vector v one computes σ(v + n) where σ is the sigmoid function and n represents annealed Gaussian noise that pushes the network to not use middle values in v. We enhance this method by using a saturating sigmoid and a straight-through pass with only bits passed forward. These techniques, described in detail below, allow to forgo the annealing of the noise and provide a stable discretization mechanism that requires neither annealing nor additional loss factors.We test our discretization technique by amending language models over s with the autoencoded sequence c(s). We compare the perplexity achieved on s with and without the c(s) component, and contrast this value with the number of bits used in c(s). We argue that this number is a proper measure for the performance of a discrete autoencoder. It is easy to compute and captures the performance of the autoencoding part of the model. This quantitative measure allows us to compare the technique we introduce with other methods, and we show that it performs better than a GumbelSoftmax BID4 BID8 in this context.Finally, we discuss the use of adding the autoencoded part c(s) to a sequence model. We present samples from a character-level language model and show that the latent symbols correspond to words and phrases when the architecture of c(s) is local. ehen, we introduce a decoding method in which c(s) is sampled and then s is decoded using beam search. This method alleviates a number of problems observed with beam search or pure sampling. We show how our decoding method can be used to obtain diverse translations of a sentence from a neural machine translation model. To summarize, the main contributions of this paper are:(1) a discretization technique that works well without any extra losses or parameters to tune, (2) a way to measure performance of autoencoders for sequence models with baselines, (3) an improved way to sample from sequence models trained with an autoencoder part. In this work, the study of text autoencoders BID1 BID17 is combined with the research on discrete autoencoders BID4 BID8 . It turns out that the semantic hashing technique BID11 can be improved and then yields good results in this context. We introduce a measure of efficiency of discrete autoencoders in sequence models and show that improved semantic hashing has over 50% efficiency. In some cases, we can decipher the latent code, showing that latent symbols correspond to words and phrases. On the practical side, sampling from the latent code and then running beam search allows to get valid but highly diverse samples, an important problem with beam search BID15 .We leave a number of questions open for future work. How does the architecture of the function c(s) affect the latent code? How can we further improve discrete sequence autoencoding efficiency? Despite remaining questions, we can already see potential applications of discrete sequence autoencoders. One is the training of multi-scale generative models end-to-end, opening a way to generating truly realistic images, audio and video. Another application is in reinforcement learning. Using latent code may allow the agents to plan in larger time scales and explore more efficiently by sampling from high-level latent actions instead of just atomic moves.
Autoencoders for text with a new method for using discrete latent space.
{"src": "scitldr-AIC_train", "paper_id": "BkQCGzZ0-"}