
Neural network performance has elements which cannot be replicated by kernel machines (linear optimization problems).

Neural networks can be approximated by data-dependent kernels, where the kernel function and the Reproducing Kernel Hilbert Space depend on the training data
(e.g., on unlabeled training data like “warped RKHS” [1]}).

Neural networks in practical settings can be effectively approximated by kernels, such as Neural Tangent Kernels corresponding to infinitely wide networks [2]}.

