Moreover, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-128K, and MiniCPM-MoE. We conduct evaluations of MiniCPM family against established benchmarks and illuminate their impressive capabilities as SLMs: (1) The foundation models surpasses Mistral-7B, LLama-13B. (2) The dpo model surpasses zephyr-7B (Tunstall et al., 2023) on MTBench (Zheng et al., 2024) (3) The 2.4B MiniCPM-128K model demonstrates performance either surpassing or matching that of models like Yarn-Mistral-7B-128K (Peng et al., 2023) and ChatGLM3-6B-128K (Du et al., 2021). (4) The MiniCPM-MoE, with 4B activated parameters, is on par with Llama2-34B (Touvron et al., 2023).