Optimization and EvaluationFor optimization, we use the AdamW optimizer with a warmup of 20 steps, a learning rate of \(1e-6\), a weight decay of 0.001, and a batch size of 8 (along with any necessary gradient accumulation to keep this fixed across runs). Moreover, our <!startofthought|> and <!endofthought|> embedding gradient weight is \(1e2\) and our policy weight is \(1e6\). We sample with temperature \(T=1\) during training and use greedy decoding for the thoughts during evaluation. We treat our samples as importance samples by computing the REINFORCE loss at temperature \(T=3\). Because we do not prompt the model with any examples, we directly compute the probability of the correct answer, conditioned on generating an answer - for example, for multiple choice questions between \(A\cdots E\), we compute the accuracy over the logits for tokens corresponding to \(A\cdots E\). Lastly, for our training, we select a random span of 256 tokens from each sample (or pad if there are fewer than 256 tokens).