One advantage of modeling with the function space view, such as GP, is that we can augment the model by imposing constraint on the function through inducing points [1]}, [2]}, i.e. the random function has to pass through a set of points, \(f({\bf z}_{1:M})=u_{1:M}\) , in the absence of noise. Those points can be treated as additional hyper-parameters to be optimized (empirical Bayes), or can be treated as random variable so that one has to infer their distribution in a full Bayes setting. In the context of DGP, these inducing points can serve as hidden function's support in variational inference [3]}, or they can be interpreted as the low fidelity observations in multi-fidelity regression problems [4]}, [5]}. However, it becomes less straightforward to incorporate these inducing points into the deep neural networks from a random weight space view [6]}.
