Two main routes to clarifying our understanding of deep learning are analytic mathematical formulations of specific models and theoretical connections between models of deep learning. Gaussian processes (GPs) [1]} hold promise both due to their compact analytic representation and their connection to shallow neural networks [2]} and kernel method [3]}, [4]}, [5]}. However, these strengths do not translate directly to deep Gaussian Processes (DGPs) [6]} due to the non-Gaussian nature of composition of GPs [7]}, [8]}, [9]} and the intractable marginalization in Bayesian learning. A second compelling candidate is the Neural Tangent Kernel (NTK) [10]}, which like GPs relies on large width limits to develop analytic representations for deep learning models and learning dynamics. In this work, we build on these foundations to further develop tools for analytic characterization of, and connections between, deep learning models.
