The remainder of the paper is organized as follows. In Section , we articulate and support our central thesis about large generative models by decomposing it into four claims, each of which we support with evidence from previously published data, and in some cases, with novel experiments on large language models [1]}. In Section REF  we discuss smooth general capability scaling. More precisely, by general capability scaling we mean two things. First, the training (and test) loss improves predictably with scale on a broad data distribution. Second, this improvement in loss tends to correlate on average with increased performance on a number of downstream tasks [2]}, [3]}. We refer to the combination of these two properties throughout the paper as smooth general capability (or performance) scaling.Note that, as will be discussed later as the central thesis of the paper, smooth general capability scaling does not imply smooth scaling on any particular task. It also does not imply that the tasks typically measured are the only tasks that are important; indeed the presence of unmeasured tasks is part of our thesis.
In Section REF , we discuss abrupt specific capability scaling, in which models can also suddenly gain specific capabilities at scale. We illustrate this phenomenon with three examples from the literature [2]}, [3]}, [6]}. In Section REF , we argue that entire areas of model competency may be unknown until they are solicited from specific inputs, problem domains, or applications. In Section REF , we discuss challenges that arise from the open-endedness of model outputs and show both qualitative and quantitative examples of harmful and toxic outputs emerging with scale.
<FIGURE>