This study introduces the Chinese Tiny LLM (CT-LLM), a pioneering endeavor to redefine the landscape of LLMs by shifting towards prioritizing the Chinese language. CT-LLM, with its 2 billion parameters, diverges from traditional approaches by being meticulously pre-trained on a comprehensive corpus comprising 1,200 billion tokens. This corpus, distinctin its composition, includes an extensive collection of 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. Our careful data processing procedures offer the Massive Appropriate Pretraining Chinese Corpus (MAP-CC), enhancing the quality of Chinese web corpora and setting a new standard for dataset preparation in the field. The strategic inclusion of a diverse and substantial amount of Chinese textual data enables CT-LLM to achieve exceptional proficiency in processing and understanding Chinese, setting a new precedent for LLM capabilities.