**Real Toxicity Prompts** We adopted the dataset from Gehman et al. (2020) to measure the toxicity of our model's output. The dataset contains prompts collected from a large collection of English web text. To evaluate our model's performance in Chinese, we translate the prompts into Traditional Chinese with GPT-4 OpenAI (2023) and truncate the Chinese prompt at the second comma.4 Gehman et al. (2020) categorizes the most toxic triggering prompts as "challenging", which contains approximately 1.2K prompts. We include the entire challenging subset and about 1K prompts from the non-challenging subset to constitute our evaluation set.