From Kernel Methods to Neural Networks: A Unifying Variational Formulation
The minimization of a data-fidelity term and an additive regularization
functional gives rise to a powerful framework for supervised learning. In this
paper, we present a unifying regularization functional that depends on an
operator and on a generic Radon-domain norm. We establish the existence of a
minimizer and give the parametric form of the solution(s) under very mild
assumptions. When the norm is Hilbertian, the proposed formulation yields a
solution that involves radial-basis functions and is compatible with the
classical methods of machine learning. By contrast, for the total-variation
norm, the solution takes the form of a two-layer neural network with an
activation function that is determined by the regularization operator. In
particular, we retrieve the popular ReLU networks by letting the operator be
the Laplacian. We also characterize the solution for the intermediate
regularization norms $\|\cdot\|=\|\cdot\|_{L_p}$ with $p\in(1,2]$. Our
framework offers guarantees of universal approximation for a broad family of
regularization operators or, equivalently, for a wide variety of shallow neural
networks, including the cases (such as ReLU) where the activation function is
increasing polynomially. It also explains the favorable role of bias and skip
connections in neural architectures.