**Remark**.: _It's worth noting that many common LLMs Tunstall et al. (2023) include an additional step of to align the model with human preference Rafailov et al. (2023); Munos et al. (2023); von Werra et al. (2020).