Language Models (LMs)  achieve substantial language capabilities when finetuned using Reinforcement Learning with Human Feedback (RLHF) . However, RLHF is an unstable and data-hungry process that continually requires new high-quality LM-generated data for finetuning. We introduce Advantage-Leftover Lunch RL (A-LoL) , a new class of offline policy gradient algorithms that enable RL training on any pre-existing data. By assuming the entire LM output sequence as a single action,A-LoLallows incorporating sequence-level classifiers or human-designed scoring functions as rewards. Subsequently, by using LM’s internal sequence-level value estimate,A-LoLfilters negative advantage (low-quality)  data points during training, making it resilient to noise. Overall,A-LoLis an easy-to-implement LM training recipe that is sample-efficient and stable. We demonstrate the effectiveness ofA-LoLand its variants with a set of four different language generation tasks. We compare against both online RL (PPO)  and recent preference-based (DPO, PRO)  and reward-based (GOLD)  offline RL baselines. On the commonly-used RLHF benchmark, Helpful and Harmless Assistant (HHA) , LMs trained withA-LoLmethods achieve the highest diversity while also being rated more safe and helpful than baselines according to humans. Additionally, in the remaining three tasks,A-LoLcould optimize multiple distinct reward functions even when using noisy or suboptimal training data.