2020; Power et al., 2022) or simply rely on surface pattern memorization Carlini et al. (2019); Tanzer et al. (2022). Feldman (2020); Carlini et al. (2019); Zaremba et al. (2022) develop methodologies to quantify the extent of memorization in neural models. Brown et al. (2020); Power et al. (2022); Liu et al. (2022) further explored how memorization impacts the models' generalization capabilities. Recently, a seminal work by Carlini et al. (2021); Shi et al. (2023) demonstrated the ability of language models to memorize and regurgitate training data, raising significant privacy concerns Kandpal et al. (2022); Pan et al. (2020).