This question has been the subject of a substantial body of literature. One line of works considered the case where gradient methods converge to minimal norm solutions on kernel regression [1]}, [2]}, [3]}, [4]}, and then analyzed the generalization properties of those minimal norm solutions. However, the phenomenon of norm minimization has only been proven to occur with the quadratic loss with an appropriate initialization scheme. Another line of works focused on overparameterized models, e.g., neural networks under the Neural Tangent Kernel (NTK) regime [5]}, [6]}, [7]}, [8]}, [9]}, proving that overparameterized neural networks trained by (stochastic) gradient descent ((S)GD) have good generalization properties on certain target functions, e.g. polynomial functions. All of these works have made significant progress on the interplay of optimization and generalization. However, they focused on studying specific settings, such as the NTK and models possessing minimal norm solutions. In this paper, we propose to study general loss function conditions that induce direct connections between optimization and generalization.
