Here, we illustrate three examples of abrupt capability scaling for arithmetic [1]}, language understanding, [2]}, [3]}, and programming [4]} (Figure REF ). For arithmetic, GPT-3 displays a sharp capability transition somewhere between 6B parameters and 175B parameters, depending on the operation and the number of digits [1]}. For example, three digit addition is performed accurately less than 1% of the time on any model with less than 6B parameters, but this jumps to 8% accuracy on a 13B parameter model and 80% accuracy on a 175B parameter model – producing a “hockey stick”-style graph (Figure REF , Left) in which arithmetic ability appears suddenly after several orders of magnitude of nothing.
