Broadly, Quiet-STaR proceeds by generating rationales after every token to explain future text (_think_), mixing the future-text predictions with and without rationales (_talk_), and then learning to generate better rationales using REINFORCE (_learn_). We apply Quiet-STaR to Mistral 7B (Jiang et al., 2023) using the web text datasets OpenWebMath (Paster et al., 2023) and Colossal Clean Crawled Corpus (C4, Raffel et al., 2020). We find that, even without dataset-specific fine-tuning, Quiet-STaR results in improvements to zero-shot direct-reasoning abilities on CommonsenseQA (36.3%\(\rightarrow\)47.2%) and GSM8K (5.9%\(\rightarrow\)10.9%), and that these improvements consistently increase with the number of tokens used in the LM's internal thoughts. Lastly, we qualitatively investigate patterns in the generated rationales.