Designing machine learning models (\(\mathcal {M}\) ) that maximize data efficiency is critical to the success of solving real-world tasks. Indeed, breakthroughs in machine learning are often driven by novel architectures such as LeNet [1]}, AlexNet[2]}, Transformer [3]}, etc. While some of the inductive biases of these methods are clear (e.g., translation symmetries of CNNs), others tend to build off of prior empirical success and are less well-understood (e.g., the implicit bias of SGD). To build our understanding of these biases and how they affect learning, we conduct a theoretical analysis of them in the infinite-width setting [4]}, [5]}, [6]}, [7]}, [8]}, which preserves the most salient aspects of the architecture while enabling tractable calculations.
