Outside of the context of language, learning policies from preferences has been studied in both bandit and reinforcement learning settings, and several approaches have been proposed. Contextual bandit learning using preferences or rankings of actions, rather than rewards, is known as a contextual dueling bandit (CDB; [48; 14]). In the absence of absolute rewards, theoretical analysis of CDBs substitutes the notion of an optimal policy with a _von Neumann winner_, a policy whose expected win rate against _any_ other policy is at least 50% [14]. However, in the CDB setting, preference labels are given online, while in learning from human preferences, we typically learn from a fixed batch of offline preference-annotated action pairs [47]. Similarly, _preference-based RL_ (PbRL) learns from binary preferences generated by an _unknown_'scoring' function rather than rewards [9; 35]. Various algorithms for PbRL exist, including methods that can reuse off-policy preference data, but generally involve first explicitly estimating the latent scoring function (i.e. the reward model) and subsequently optimizing it [16; 9; 12; 34; 19]. We instead present a single stage policy learning approach that directly optimizes a policy to satisfy preferences.