When combined with our two-stage fine-tuning pipeline, our method achieves improved common sense reasoning and state-of-the-art perplexity on the WritingPrompts<cite> (Fan et al., 2018)</cite> story generation dataset.