In the literature, the implicit bias of first-order methods is first studied in linear settings since the analysis is more tractable, and also, there have been several theoretical and empirical evidence that certain insights from linear models translate to deep learning, e.g. [1]}, [2]}, [3]}, [4]}, [5]}, [6]}. In the linear setting, it is easier to establish implicit bias for regression tasks, where square loss is typically used and it attains its minimum at a finite value. For example, the implicit bias of gradient descent (GD) for square loss goes back to [7]}. Beyond GD, analysis of other popular algorithms such as the family of mirror descent (MD), which is an important generalization of GD, is more involved and was established only recently by [8]}, [9]}. Specifically, those works showed that mirror descent converges to the interpolating solution that is closest to the initialization in terms of a Bregman divergence. Thus, the implicit bias in linear regression is relatively well-understood by now.
