We introduce a new and rigorously-formulated PAC-Bayes few-shot meta-learning algorithm that implicitly learns a model prior distribution of interest. Our proposed method extends the PAC-Bayes framework from a single task setting to the few-shot meta-learning setting to upper-bound generalisation errors on unseen tasks. We  also propose a generative-based approach to model the shared prior and task-specific posterior more expressively compared to the usual diagonal Gaussian assumption. We show that the models trained with our proposed meta-learning algorithm are well calibrated and accurate, with state-of-the-art calibration and classification results on mini-ImageNet benchmark, and competitive results in a multi-modal task-distribution regression. One unique ability of humans is to be able to quickly learn new tasks with only a few training examples. This is due to the fact that humans tend to exploit prior experience to facilitate the learning of new tasks. Such exploitation is markedly different from conventional machine learning approaches, where no prior knowledge (e.g. training from scratch with random initialisation) (Glorot & Bengio, 2010) , or weak prior knowledge (e.g., fine tuning from pre-trained models) (Rosenstein et al., 2005) are used when encountering an unseen task for training. This motivates the development of novel learning algorithms that can effectively encode the knowledge learnt from training tasks, and exploit that knowledge to quickly adapt to future tasks (Lake et al., 2015) . Prior knowledge can be helpful for future learning only if all tasks are assumed to be distributed according to a latent task distribution. Learning this latent distribution is, therefore, useful for solving an unseen task, even if the task contains a limited number of training samples. Many approaches have been proposed and developed to achieve this goal, namely: multi-task learning (Caruana, 1997) , domain adaptation (Bridle & Cox, 1991; Ben-David et al., 2010) and meta-learning (Schmidhuber, 1987; Thrun & Pratt, 1998) . Among these, meta-learning has flourished as one of the most effective methods due to its ability to leverage the knowledge learnt from many training tasks to quickly adapt to unseen tasks. Recent advances in meta-learning have produced state-of-the-art results in many benchmarks of few-shot learning data sets (Santoro et al., 2016; Ravi & Larochelle, 2017; Munkhdalai & Yu, 2017; Snell et al., 2017; Finn et al., 2017; Rusu et al., 2019) . Learning from a few examples is often difficult and easily leads to over-fitting, especially when no model uncertainty is taken into account. This issue has been addressed by several recent Bayesian meta-learning approaches that incorporate model uncertainty into prediction, notably LLAMA that is based on Laplace method (Grant et al., 2018) , or PLATIPUS (Finn et al., 2017) , Amortised Meta-learner (Ravi & Beatson, 2019) and VERSA (Gordon et al., 2019 ) that use variational inference (VI). However, these works have not thoroughly investigated the generalisation errors for unseen samples, resulting in limited theoretical generalisation guarantees. Moreover, most of these papers are based on variational functions that may not represent well the richness of the underlying distributions. For instance, a common choice for the variational function relies on the diagonal Gaussian distribution, which can potentially worsen the prediction accuracy given its limited representability. In this paper, we address the two problems listed above with the following technical novelties: (i) derivation of a rigorous upper-bound for the generalisation errors of few-shot meta-learning using PAC-Bayes framework, and (ii) proposal of a novel variational Bayesian learning based on implicit The few-shot meta-learning problem is modelled using a hierarchical model that learns a prior p(w i ; θ) using a few data points s ij )}. Shaded nodes denote observed variables, while white nodes denote hidden variables. generative models to facilitate the learning of unseen tasks. Our evaluation shows that the models trained with our proposed meta-learning algorithm is at the same time well calibrated and accurate, with competitive results in terms of Expected Calibration Error (ECE) and Maximum Calibration Error (MCE), while outperforming state-of-the-art methods in a few-shot classification benchmark (mini-ImageNet). We introduce and formulate a new Bayesian algorithm for few-shot meta-learning. The proposed algorithm, SImBa, is based on PAC-Bayes framework which theoretically guarantees prediction generalisation on unseen tasks. In addition, the proposed method employs a generative approach that implicitly models the shared prior p(w i ; θ) and task-specific posterior q(w i ; λ i ), resulting in more expressive variational approximation compared to the usual diagonal Gaussian methods, such as PLATIPUS or Amortised Meta-learner (Ravi & Beatson, 2019) . The uncertainty, in the form of the learnt implicit distributions, can introduce more variability into the decision made by the model, resulting in well-calibrated and highly-accurate prediction. The algorithm can be combined with different base models that are trainable with gradient-based optimisation, and is applicable in regression and classification. We demonstrate that the algorithm can make reasonable predictions about unseen data in a multi-modal 5-shot learning regression problem, and achieve state-of-the-art calibration and classification results with on few-shot 5-way tasks on mini-ImageNet data set.
Bayesian meta-learning using PAC-Bayes framework and implicit prior distributions
{"src": "scitldr-AIC_train", "paper_id": "SkgYJaEFwS"}