Parameters Perplexity<cite> Chelba et al. (2013)</cite> 20m 51.3 NCE (ours) 3.4m 43.2 Recently, (Józefowicz et al., 2016) achieved stateof-the-art language modeling perplexities (30.0) on the billion word dataset with a single model, using importance sampling to approximate the normalization constant, Z(u).