The implicit bias of gradient descent has been the focus of a number of recent works [1]}, [2]}, [3]}, [4]}, [5]}, leading to a variety of noteworthy conclusions, including the convergence of GD to the maximal margin solution for logistic-type losses during late-time training [1]}. [7]}, [8]}, [9]}, [10]}, [11]}, [12]}, [13]} study the early-time dynamics and spectral biases of neural networks, leading to the conclusion that simpler functions are usually learned before more complex functions.
