A remarkable recent theoretical deep learning discovery (discussed in Section REF ) is that in certain regimes very wide neural networks are equivalent to kernel machines. At this point much of the theoretical discussion centers on understanding the “rich regimes” (e.g., [1]}, [2]})), often identified with “feature learning”, i.e., learning representations from data. In these regimes, tangent kernels change during the training, hence neural networks are not approximated by kernel machines, i.e., a feature map followed by a linear method. The prevalent view among both theoreticians and practitioners, is that success of neural networks cannot be explained by kernel methods as kernel predictors. Yet kernel change during training does not logically imply useful learning and may be an extraneous side effect. Thus the the question of equivalence remains open. 
Recent, more
sophisticated, kernel machines show performance much closer to the state-of-the-art on certain tasks [3]}, [4]} but have not yet closed the gap with neural networks.
