A different language model, DeepMind’s Gopher [1]}, also displays an abrupt jump in performance on a different dataset, the MMLU language understanding benchmark [2]} (Figure REF , Middle, orange). For all models under 6B parameters, Gopher performs under 30% accuracy, which is a little better than chance (25% accuracy). However, the full 280B parameter Gopher model achieves 60% accuracy, a significant jump. GPT-3 displays a similar phenomenon though of smaller magnitude (Figure REF , Middle, blue).
