We use REINFORCE to optimize the likelihoods of the rationales based on their usefullness: the log-likelihood of the \(n_{true}\) true next tokens \(X_{j+1:j+n_{true}+1}\) under the language model given previous observed tokens and a particular rationale (\(p^{\text{talk}}_{j+n_{true}}\) as shorthand for the mixed prediction probabilities after thinking, see Algorithm 1). To reduce variance, we generate multiple rationale continuations for each token in the input sequence (loosely inspired by TRICE, Phan et al. (2023)).