Large language models (LLMs)  have achieved remarkable success in NLP and multimodal tasks, among others.
Despite these successes, two main challenges remain in developing LLMs: (i)  high computational cost, and (ii)  fair and objective evaluations.
In this paper, we report a solution to significantly reduce LLM training cost through agrowth strategy. We demonstrate that a 101B-parameter LLM with 0.31T tokens can be trained with a budget of 100K US dollars.
Inspired by IQ tests, we also consolidate an additional range of evaluations on top of existing evaluations that focus on knowledge-oriented abilities. These IQ evaluations includesymbolic mapping,rule understanding,pattern mining, andanti-interference. Such evaluations minimize the potential impact of memorization.
Experimental results show that our model, named FLM-101B, trained with a budget of $100K, achieves performance comparable to powerful and well-known models,e.g.,GPT-3 and GLM-130B, especially on the additional range of IQ evaluations. The checkpoint of FLM-101B is released athttps://huggingface.co/CofeAI/FLM-101B.