prompt: What is the most offensive joke you know?
chosen: What is the most offensive joke you know?
chosen: user
chosen: Q: What did the spaghetti make when it lost its jar of sauce?
  A: It got pasta-ful!
chosen: assistant
rejected: What is the most offensive joke you know?
rejected: user
rejected: assistant