Do large language models (LLMs)  genuinely understand the semantics of the language, or just memorize the training data? The recent concern on potential data contamination of LLMs has raised awareness of the community to conduct research on LLMs evaluation. In this paper, we proposeMSTemp, an approach that creates meta semantic templates to evaluate the semantic understanding ability of LLMs. The core ofMSTempis not to perform evaluation directly on existing benchmark datasets, but togeneratenew out-of-distribution (OOD)  evaluation sets using existing datasets asseeds. Specifically, for a given sentence,MSTempleverages another language model to generate new samples while preserving its semantics. The new samples are called semantic templates to the original sentence. Then,MSTempgenerates evaluation samples via sentence parsing and random word replacement on the semantic templates.MSTempis highly flexible, dynamic, and cost-effective. Our initial experiments show thatMSTemp-generated samples can significantly reduce the performance of LLMs using existing datasets as seeds. We hope this initial work can shed light on future research of LLMs evaluation.