Gradient Boosting Performs Low-Rank Gaussian Process Inference
This paper shows that gradient boosting based on symmetric decision trees can
be equivalently reformulated as a kernel method that converges to the solution
of a certain Kernel Ridgeless Regression problem. Thus, for low-rank kernels,
we obtain the convergence to a Gaussian Process' posterior mean, which, in
turn, allows us to easily transform gradient boosting into a sampler from the
posterior to provide better knowledge uncertainty estimates through Monte-Carlo
estimation of the posterior variance. We show that the proposed sampler allows
for better knowledge uncertainty estimates leading to improved out-of-domain
detection.