Figure REF  shows a timeline of public disclosures of GPT-3 scale (100B - 530B) dense language models, since GPT-3.The timeline does not include sparse or mixture of experts models (e.g., GLaM [1]}), which often achieve comparable performance with similar or slightly lower compute, but are difficult to characterize in terms of a single model size. It also does not include models trained on different modalities, such as code [2]}, [3]}, or multi-modal models such as [4]}. About one year after GPT-3 was announced, a spike in similar model announcements followed. These models were developed by both large and small private organizations from around the world: Jurassic-1-Jumbo [5]}, AI21 Labs, Israel; Ernie \(3.0\)  Titan [6]}, Baidu, China; Gopher [7]}, DeepMind, USA/UK; FLAN [8]} & LaMDA [9]}, Google, USA;
Pan Gu [10]} Huawei, China; Yuan \(1.0\)  [11]}, Inspur, China; Megatron Turing NLG [12]}, Microsoft & NVIDIA, USA; and HyperClova [13]}, Naver, Korea. This suggests that the economic incentives to build such models, and the prestige incentives to announce them, are quite strong.
