Lack of access to the underlying prompts hinders efforts to scrutinize, evaluate, and regulate language models effectively, thereby impeding advancements in the responsible and ethical development of AI technologies. Our research advances the field towards wider access to prompts while highlighting important privacy concerns for language-models-as-a-service providers.