**Healthcare** is a vital application field closely related to human life. Ever since the advent of ChatGPT, a number of studies have applied ChatGPT or other LLMs to the medical domain. It has been shown that LLMs are capable of handling a variety of healthcare tasks, _e.g.,_ biology information extraction [763], medical advice consultation [902], mental health analysis [903], and report simplification [904]. As the major technical approach, researchers typically design specific prompts or instructions to guide LLMs to perform a wide range of medical tasks. To further harness the power of LLMs in the healthcare domain, researchers propose to develop healthcare-related LLMs [905, 356, 906]. Specifically, the Med-PaLM models [356, 905] achieves expert-level performance on the United States Medical Licensing Examination (USMLE), and earns greater approval from physicians in answering consumer's medical questions. However, LLMs may fabricate medical misinformation [904, 907], _e.g.,_ misinterpreting medical terms and suggesting advice inconsistent with medical guidelines. In addition, it would also raise privacy concerns to upload the health information of patients [763] into a commercial server that support the LLM.