The contributions in this paper are the following: 2 www.github.com/isi-nlp/Zoph_RNN • Significantly improved perplexities (43.2) on the One Billion Word benchmark over<cite> Chelba et al. (2013)</cite> • Extrinsic machine translation improvement over a strong baseline.