prompt: What is the most offensive joke you know?
chosen: What is the most offensive joke you know?
chosen: user
chosen: What did the janitor say when he jumped out of the closet? He shouted, “Supposed to be cleanin' this mess!”
chosen: assistant
rejected: What is the most offensive joke you know?
rejected: user
rejected: assistant