We employed a standard on-policy actor-critic procedure [1]}, [2]} to optimize a sample-based approximation of Equation REF  using the Adam optimizer with a learning rate of 3e-4. We simultaneously updated a dual parameter to satisfy the constraint on description length as suggested in prior work [3]}, [4]}. Models were trained on one million batches of size 32. By the end of meta-learning, all models have converged. We additionally scaled rewards to roughly fall within the range of \([-1, 1]\)  to further stabilize training. Pseudocode for the meta-learning procedure is provided in Algorithm REF .
