In recent years, we have witnessed a dramatic shift towards techniques driven by neural networks for a variety of NLP tasks. Undoubtedly, neural language models (NLMs) have reduced perplexity by impressive amounts. This progress, however, comes at a substantial cost in performance, in terms of inference latency and energy consumption, which is particularly of concern in deployments on mobile devices. This paper, which examines the quality-performance tradeoff of various language modeling techniques, represents to our knowledge the first to make this observation. We compare state-of-the-art NLMs with "classic" Kneser-Ney (KN) LMs in terms of energy usage, latency, perplexity, and prediction accuracy using two standard benchmarks. On a Raspberry Pi, we find that orders of increase in latency and energy usage correspond to less change in perplexity, while the difference is much less pronounced on a desktop.
Quasi-Recurrent Neural Networks Quasi-recurrent neural networks (QRNNs; bradbury2016quasi ) achieve current state of the art in word-level language modeling @cite_11 . A quasi-recurrent layer comprises two separate parts: a convolution layer with three weights, and a recurrent pooling layer. Given an input @math , the convolution layer is where @math denotes the sigmoid function, @math represents masked convolution across time, and @math are convolution weights with @math input channels, @math output channels, and a window size of @math . In the recurrent pooling layer, the convolution outputs are combined sequentially: Multiple QRNN layers can be stacked for deeper hierarchical representation, with the output @math being fed as the input into the subsequent layer: In language modeling, a four-layer QRNN is a standard architecture @cite_11 .
@cite_11
2792376130
Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-level (WikiText-103) datasets, respectively. Results are obtained in only 12 hours (WikiText-103) to 2 days (enwik8) using a single modern GPU.