To ensure that our results are not specific to GPT-3 models trained with the OpenAI API, we also perform a hyperparameter sweep using Llama-7b. Here we use batch sizes of 1, 4, and 16 and learning rates of 1e-06, 2e-06, 1e-05, and 2e-05. We use Adam as our optimizer and DeepSpeed level 3 for memory efficiency. We perform full finetuning and do not use any parameter efficient finetuning techniques. The results are shown in Figure 8.