Scaling up the amount of data, compute power, and model parameters of neural networks has recently led to the arrival (and real world deployment) of capable generative models such as CLIP [1]}, Ernie \(3.0\)  Titan [2]}, FLAN [3]}, Gopher [4]}, GPT-3 [5]}, HyperClova [6]}, Jurassic-1-Jumbo [7]}, Megatron Turing NLG [8]}, LaMDA [9]}, Pan Gu [10]}, Yuan \(1.0\)  [11]}, and more. For this class of modelsSome refer to this class of models as `foundation models'[12]}. the relationship between scale and model performance is often so predictable that it can be described in a lawful relationship — a scaling law. In most cases, these scaling laws predict a continued increase in certain capabilities as models get larger. At the same time, larger generative models represent an increasing proportion of the eye-catching results in machine learning. As a result, many institutions have started producing large models over the past few years, in response to the predictability afforded by scaling laws, and the fact these models can be plugged into systems that generate economic value, like search engines.We do not discuss to whom this economic value accrues, and we do not intend to imply that by default it will accrue broadly or that no one will be harmed. It has also become clear that these models present novel risks of harmful behavior, which are difficult to predict and may become more severe as the models increase in capability. Attempts to study these harms with smaller models may not accurately reflect what occurs in larger ones.
