It is thus even more fascinating that a large class of very
complex functions turn out to be linear in parameters and the corresponding complex learning algorithms are simply training kernel machines.
In my view this adds significantly to the evidence that understanding kernel learning is a key to deep learning as we argued in [1]}.
Some important caveats are in order.
While it is arguable that deep learning may be equivalent to kernel learning in some interesting and practical regimes, the jury is still out on the question of whether this point of view can provide a conclusive understanding of generalization in neural networks. Indeed a considerable amount of recent theoretical work has been aimed at trying to understand regimes (sometimes called the “rich regimes”, e.g.,  [2]}, [3]}) where the transition to linearity does not happen and the system is non-linear throughout the training process. Other work (going back to [4]}) argues that there are theoretical barriers separating function classes learnable by neural networks and kernel machines [5]}, [6]}.
Whether these analyses are relevant for explaining empirically observed behaviours of deep networks still requires further exploration.
