1. **Parallel rationale generation (think, Subsection 4.2)**: In parallel across \(n\) tokens \(x_{i}\) in an input sequence \(x_{0:n}\), we generate \(r\) rationales of length \(t\): \(c_{i}=(c_{i1},\dots,c_{it})\), resulting in \(n\times r\) rationale candidates. We insert learned \(<\)\(\mid\)start\(\mid\)\(>\) and \(<\)\(\mid\)end\(\mid\)\(>\) tokens to mark each rationale's start and end. 2. **Mixing post-rationale and base predictions (talk, Subsection 4.3)**: From the hidden state output after each rationale, we train a "mixing head" - a shallow MLP producing a weight determining how much the post-rationale next-token predicted logits should be incorporated compared to the base language model predicted logits. This approach eases distribution shift early in finetuning, due to introducing rationales. 3. **Optimizing rationale generation (learn, Subsection 4.4)**: We optimize the rationale generation parameters (start/end tokens and LM weights) to increase the likelihood of rationales that make future text more probable. We use REINFORCE to provide a learning signal to rationales based on their impact on future-token prediction. To reduce variance, we apply a teacher-forcing trick to include in the loss the likelihood of predicting not only the token after the thought but also later tokens.