Our approach further refines the model's competencies through supervised fine-tuning(SFT). The SFT not only bolsters the model's adeptness in Chinese language tasks but also enhances its versatility in comprehending and generating English text, showcasing its multi-lingual prowess. We also utilize preference optimization techniques to align CT-LLM with human preferences, to enhance its harmlessness and helpfulness. Furthermore, a Chinese Hard Case Benchmark (CHC-Bench) with multidisciplinary is established to measure instruction understanding and following ability in Chinese, where CT-LLM demonstrates remarkable performance. By challenging the prevailing norms of training LLMs primarily on English corpora, CT-LLM expands the horizons of language model training, offering fresh perspectives on the potentialities of non-English-centric LLMs.