Large language models (LLMs) have revolutionized the field of natural language processing (NLP) through a training process that includes pretraining, supervised fine-tuning, and alignment tuning, with the latter ensuring the safety and usefulness of the model. Thus, reinforcement learning techniques Christiano et al. (2017); Bai et al. (2022), such as proximal policy optimization (PPO) Schulman et al. (2017), are key in this alignment phase, despite their complexity.