**Deep-and-thin Network.** We train MiniCPM-2.4B before training MiniCPM-1.2B. When training MiniCPM-2.4B, we adopt a deeper and thinner architecture compared to Phi-2 (Javaherippi and Bubeck, 2023) (40 layers compared to 32 layers). Recently, Liu et al.