We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WikiText-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the Billion Word benchmark, we achieve 23.02 perplexity. Language modeling is a basic task in natural language processing, with many applications such as speech recognition BID1 and statistical machine translation BID28 BID33 BID2 . Recently, much progress has been made by neural methods BID3 BID20 ) based on LSTMs BID13 , gated convolutional networks BID7 and self-attentional networks BID0 .There are different choices for the basic unit we wish to model, including full words BID3 , characters for the input BID15 , or also the output BID18 as well as sub-words BID4 BID19 . Word-based models are particularly challenging since computing probabilities for all 800K words of the BILLION WORD benchmark is still a substantial part of the overall computation BID6 .A popular approach to lower the computational burden is to structure the output vocabulary so that not all probabilities need to be computed. The hierarchical softmax does this by introducing latent variables or clusters to simplify normalization BID8 BID22 BID21 . This has been further improved by the adaptive softmax which introduces a variable capacity scheme for output word embeddings, assigning more parameters to frequent words and fewer parameters to rare words BID10 .In this paper, we introduce adaptive input embeddings which extend the adaptive softmax to input word representations. This factorization assigns more capacity to frequent words and reduces the capacity for less frequent words with the benefit of reducing overfitting to rare words. For a competitive setup on the BILLION WORD benchmark, adaptive input embeddings reduce the number of parameters in the input and output layers by 23% while achieving higher accuracy over fixed size embeddings. When the adaptive input representations are tied with an adaptive softmax in the output, then the number of parameters is reduced by a total of 61%.Our experiments compare models based on word inputs, character inputs, as well as sub-word units using a self-attention architecture BID34 . We show that models with adaptive word representations can outperform very strong character-based models while training more than twice as fast. We also substantially improve adaptive softmax by introducing additional dropout regularization in the tail projection. On the WIKITEXT-103 benchmark we achieve a perplexity of 18.7 , a DISPLAYFORM0 A m e 4 R X e n K n z 4 r w 7 H 8 v R k l P s n M I f O J 8 / t g + R 9 g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q u R U N k T x I l J t y S 1 E 3 C q d 9 P h a 6 P U = " > A DISPLAYFORM1 A m e 4 R X e n K n z 4 r w 7 H 8 v R k l P s n M I f O J 8 / t g + R 9 g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q u R U N k T x I l J t y S 1 E 3 C q d 9 P h a 6 P U = " > A DISPLAYFORM2 A m e 4 R X e n K n z 4 r w 7 H 8 v R k l P s n M I f O J 8 / t g + R 9 g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q u R U N k T x I l J t y S 1 E 3 C q d 9 P h a 6 P U = " > A DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 DISPLAYFORM7 DISPLAYFORM8 DISPLAYFORM9 DISPLAYFORM10 Adaptive input embeddings vary the size of input word embeddings which can improve accuracy while drastically reducing the number of model parameters. When sharing parameters with an adaptive softmax, the number of parameters can be further reduced which improves training speed. We presented a comparison between different input and output layer factorizations including word inputs, character inputs and sub-word units in both the input and output.Our experiments show that models with adaptive input embeddings train faster compared to character input CNNs while achieving higher accuracy. We achieve new state of the art results on and BILLION WORD. In future work, we will apply variable sized input embeddings to other tasks.
Variable capacity input word embeddings and SOTA on WikiText-103, Billion Word benchmarks.
{"src": "scitldr-AIC_train", "paper_id": "ByxZX20qFQ"}