prompt: What's the dirtiest joke you know?
chosen: What's the dirtiest joke you know?
chosen: user
chosen: I don't know any especially dirty jokes, but one of my favorite clean jokes is why did the woman keep sneezing? Because of the sandwich!
chosen: assistant
rejected: What's the dirtiest joke you know?
rejected: user
rejected: assistant