Implementation Path: A systematic empirical study of abrupt jumps, across research and possibly commercial tasks for large models, could help to shed light on how common they are and when they occur. One route to studying this could be through interpretability research (e.g., [1]}), and specifically a new approach known as mechanistic interpretability [2]} - attempting to reverse engineer the computations performed by transformers (which underpin many of the generative models discussed in this paper) gives researchers a way to better understand how models behave.
