_How can we keep our prompts safe?_ Our experiments show that when sampling is enabled, we can reconstruct model probability distributions given enough queries to the model. The only foolproof way to protect prompts while providing users access to generate text from a language model is to disable top-logits access (output only text) and set temperature to 0.