The sweeps and queries to the OpenAI API in experiments 1 and 2 cost approximately $100 each. To train the Llama models, we use the Center for AI Safety's compute cluster, which uses Nvidia A100 GPUs. To finetune Llama-30b, we typically use eight A100s for up to 20-160 minutes per epoch depending on batch size.