The third main contribution of the present paper is that we provide a non-asymptotic analysis of the convergence of Bayes estimators, which is also closely related to KRR. Our non-asymptotic analysis is built on an operator-theoretic framework [1]}, [2]} and the equivalent kernel technique, covering the estimation of posterior mean and variance, derivatives of posterior mean and posterior variances of derivatives of Gaussian processes. We substantially extend the analysis in our earlier work [3]} by deriving a range of either new or sharper error bounds. The developed non-asymptotic bounds construct rate-optimal estimators of the regression function and its derivatives, and are also crucial in proving the equivalence theory and nonparametric plug-in property. We remark that the our non-asymptotic analysis may be of independent interest in broader contexts such as function estimation and information theory.
