Gaussian Process Meta Few-shot Classifier Learning via Linear Discriminant Laplace Approximation

The meta learning few-shot classification is an emerging problem in machine learning that received enormous attention recently, where the goal is to learn a model that can quickly adapt to a new task with only a few labeled data. We consider the Bayesian Gaussian process (GP) approach, in which we meta-learn the GP prior, and the adaptation to a new task is carried out by the GP predictive model from the posterior inference. We adopt the Laplace posterior approximation, but to circumvent the iterative gradient steps for finding the MAP solution, we introduce a novel linear discriminant analysis (LDA) plugin as a surrogate for the MAP solution. In essence, the MAP solution is approximated by the LDA estimate, but to take the GP prior into account, we adopt the prior-norm adjustment to estimate LDA's shared variance parameters, which ensures that the adjusted estimate is consistent with the GP prior. This enables closed-form differentiable GP posteriors and predictive distributions, thus allowing fast meta training. We demonstrate considerable improvement over the previous approaches.


Introduction
Few-shot classification [7,36,14,15] is the task of predicting class labels of data instances that have novel unseen class semantics, potentially from a novel domain, where the learner is given only a few labeled data from the domain. It receives significant attention recently in machine learning, not only due to the practical reason that annotating a large amount of data for training deep models is prohibitively expensive, but also the constant endeavor in AI to build human-like intelligence where the human is extremely good at recognizing new categories from a few examples.
In order to build a model that can generalize well to a novel task with only a few samples, meta learning [27,3] forms a training stage that is similar to the test scenario. More specifically, during the training stage, the learner sees many tasks (or episodes) where each task consists of the support and query sets: the learner adapts the model to the current task using a few labeled data in the support set, and the performance of the adapted model is measured on the query set, which is used as a learning signal to update the learner. This is in nature a learning-to-learn paradigm [32,28,34,8], and it often leads to more promising results in certain scenarios than simple supervised feature (transfer) learning. Although recently there were strong baselines introduced for the latter with some feature transformations [35,5], in this paper we focus on the meta learning paradigm.
As meta few-shot learning essentially aims to generalize well from only a few observations about a new task domain, it is important to learn prior information that is shared across different tasks. In this sense, the Bayesian approach [29,31] is attractive in that we can express the prior belief effectively, and easily adapt our belief to a new task based on the given evidence, in a principled manner. In Bayesian meta learning [11,41,10,16,39], the adaptation to a new task corresponds to posterior predictive distribution inference, and meta learning amounts to learning a good prior distribution from many training episodes.
To enable efficient Bayesian meta learning, the posterior predictive inference needs to be fast and succinct (e.g., closed form). To this end, we consider the Gaussian process (GP) model with the linear deep kernel [38] that allows parametric treatment of GP via the weight-space view [24]. Although there was similar attempt recently [22], they resort to regression-based likelihood model for the classification problem to derive closed-form inference, and such an ad hoc strategy can potentially lead to performance degradation. Instead, we propose a novel Laplace approximation for the GP posterior with a linear discriminant plugin, which avoids iterative gradient steps to find the maximum-a-posterior (MAP) adaptation solution, and allows a closed-form predictive distribution that can be used in stochastic gradient meta training efficiently. Hence, it is computationally more attractive than gradient-based adaptation approaches [8,11,16] by construction, and more amenable to train than neural net approximations of the predictive distribution (i.e., amortization) such as [10].
We show the improved performance of our GP approach over the regression-based previous work [22] and other state-of-the-arts on several benchmark datasets in both within-and cross-domain few-shot learning problems.

Problem Setup and Background
We provide the formal training/test setup for the meta few-shot classification problem (Sec. 2.1). We then briefly review the recent GP-based few-shot learning algorithm (GPDKT) [22] in Sec. 2.2 due to its close relation to our proposed approach.

Meta few-shot learning framework
The (C-way, k-shot) episodic meta few-shot classification problem can be formally defined as follows: • Training stage (repeated for T times/episodes): 1. Sample training data (S, Q) for this episode: support set S = {(x, y)} and query set Q = {(x, y)}, where S consists of C · k samples (k samples for each of the C classes), and Q contains C · k q samples (k q samples per class). We denote by y ∈ {1, . . . , C} the class labels in (S, Q), however, the semantic meaning of the classes is different from episode to episode. 2. With (S, Q), we train a meta learner F (S) → h where the output of F is a C-way classifier, h : X → {1, . . . , C}. The training objective is typically defined on the query set, e.g., the prediction error of h on Q.
• Test stage: 1. The k/k q -shot test data (S * , Q * ) are sampled, but the query set Q * is not revealed.
For the k-shot support set S * = {(x, y)}, we apply our learned F to S * to obtain the classifier h * = F (S * ). Again, the semantic meaning of the test class labels are different from those in the training stage. The performance of h * is measured on the test query set Q * .
For instance, in the popular ProtoNet [28], the meta learner learns the parameters θ of the feature extractor φ θ (x) (e.g., convolutional networks), and the meta learner's output h = F (S) is the nearest centroid classifer where the centroids are the class-wise means in S in the feature space. Note that h(x) admits a closed form (softmax), and the meta training updates θ by stochastic gradient descent with the loss, E (x,y)∼Q [CrossEnt(y, h(x))]. Another example is the GP meta learning framework that essentially considers h = F (S) as a GP posterior predictive model, that is, where f is a GP function, h(x) is defined as a probabilistic classifier p(y|x, S), and p(f |S) ∝ p(f ) · (x,y)∈S p(y|f (x)). Meta training of F amounts to learning the GP prior distribution p(f ) (i.e., GP mean/covariance functions). The recent GPDKT [22] is one incarnation of this GP framework.

Brief review of GPDKT
GPDKT assumes the GP regression model (its usage to classification will be described shortly), where the GP covariance function k θ is defined as the deep kernel [38]: where φ θ (x) is the feature extractor (comparable to that in ProtoNet) andk(·, ·) is a conventional kernel function (e.g., Gaussian RBF, linear, or cosine similarity). We abuse the notation to denote by θ all the parameters of the deep kernel, including those from the outer kernelk. They pose the meta training as the marginal likelihood maximization on both support and query sets: Due to the regression model, the marginal likelihood admits a closed form, and one can easily optimize (5) by stochastic gradient ascent.
To extend the GP model to the classification problem, instead of adopting a softmax-type likelihood p(y|f (x)), they rather stick to the GP regression model. This is mainly for the closed-form posterior and marginal data likelihood. In the binary classification problem, they assign real-valued y = ±1.0 as target response values for positive/negative classes, respectively, during training. At the test time, they threshold the real-valued outputs to get the discrete class labels. For the multi-class C-way problem with C > 2, they turn it into C binary classification problems by one-vs-rest conversion. Then during training, they maximize the sum of the marginal log-likelihood scores over the C binary problems, while at test time the one with the largest predictive mean E[y|x, S] over the C problems is taken as the predicted class. Although this workaround allows fast adaptation and training with the closed-form solutions from GP regression, the ad hoc treatment of the discrete class labels may degrade the prediction accuracy.

Our Approach
In this section we describe our Laplace approximation GP posterior formulation for the task adaptation, where we introduce the novel linear discriminant plug-in to circumvent the iterative optimization for the MAP solution and enable the closed-form formulas. Our formalism admits the softmax classification likelihood model, more sensible than the regression-based treatment of the classification problem.
We adopt the weight-space view of the Gaussian process model [24] using the linear-type deep kernel, and consider the softmax likelihood model with We let W = [w 1 , . . . , w C ] and B = [b 1 , . . . , b C ] be the weight-space random variables for the GP functions. Note that in (7) the prior (scalar) parameters β, β b are shared over all C functions, which is reasonable considering that the semantic meaning of classes changes from episode to episode. And it is easy to see that the i.i.d. priors on (w j , b j ) makes {f j (·)} C j=1 i.i.d. GPs with a zero mean and the covariance function, This can be interpreted as adopting a linear outer kernelk(z, z ′ ) = z ⊤ z ′ in the deep kernel (4) with some scaling and biasing. Although our formulation excludes more complex nonlinear outer kernels (e.g., RBF or polynomial), it was shown that the linear or cosine-similarity outer kernel empirically performed the best among other choices [22]. Note that the latter cosine-similarity kernel is obtained by unit-norm feature transformation (φ(x) → φ(x) ||φ(x)|| ). Given the support set S = {(x, y)}, the GP posterior distribution of f j (x) at some arbitrary input , and this is determined by the posterior p(W, B|S), where (up to constant) The posterior p(W, B|S) is used to build the task(S)-adapted classifier p(y|x, S), the GP predictive distribution derived from (1). And the meta training in our model amounts to optimizing the classification (cross-entropy) loss of the adapted classifier on the query set with respect to the GP prior parameters (i.e., β, β b , and the parameters θ of the feature extractor network φ), averaged over all training episodes. That is, our meta training loss/optimization can be written as: where p(y|x, S) = p(W, B|S)p(y|x, W, B)dW dB, and the expectation is taken over (S, Q) samples from training episodes/tasks. Considering the dependency of the loss on these prior parameters as per (10), it is crucial to have a succinct (e.g., closed-form) expression for p(W, B|S), as well as the predictive distribution p(y|x, S). However, since p(W, B|S) does not admit a closed form due to the non-closed-form normalizer (i.e., the log-sum-exp of (24) over {w j , b j } j ), we adopt the Laplace approximation that essentially approximates (24) by the second-order Taylor at around the MAP estimate {w * j , b * j } j , i.e., the maximum of (24).

Laplace approximation via LDA plugin with prior-norm adjustment
Specifically we follow the diagonal covariance Laplace approximation with diagonalized Hessian of (24), which leads to the factorized posterior p(W, B|S) = C j=1 p(w j , b j |S). The approximate posterior can be derived as j , and all operations are element-wise. However, obtaining the MAP estimate {w * j , b * j } j , i.e., the maximum of (24), although the objective is concave, usually requires several steps of gradient ascent, which can hinder efficient meta training. Recall that our meta training amounts to minimizing the loss of the task-adapted classifier p(y|x, S) on a query set with respect to the feature extractor φ θ (·) and the GP prior parameters β, β b , and we prefer to have succinct (closed-form-like) expression for p(y|x, S) in terms of θ, β, β b . The iterative dependency of p(y|x, S) on φ, β, β b , resulting in a similar strategy as MAML [8], would make the meta training computationally expensive. To this end, we propose a novel linear discriminant analysis (LDA) plugin technique as a surrogate of the MAP estimate.
LDA-Plugin. We preform the LDA on the support set S, which is equivalent to fit a mixture of Gaussians with equi-covariances by maximum likelihood [4]. More specifically, we consider the Gaussian mixture model (with some abuse of notation, e.g., p(x) instead of p(φ(x))), p(x, y) = p(y)p(x|y) = π y N (φ(x); µ y , σ 2 I), where we assume the shared spherical covariance matrix across different classes. The maximum likelihood (ML) estimate on S can be derived as: where S j = {(x, y) ∈ S : y = j}, n j = |S j |, n = |S|, and d = dim(φ(x)). Then our idea is to use this ML-estimated Gaussian mixture to induce the class predictive model p(y|x) = p(x, y)/p(x), and match it with our GP likelihood p(y|F (x)) in (6) to obtain {w j , b j } j , which serves as a surrogate of the MAP estimate {w * j , b * j } j . More specifically, the class predictive from the Gaussian mixture is: . (15) We match it with the GP likelihood model p(y|F (x)) from (6), that is, which establishes the following correspondence: where α is a constant (to be estimated). We aim to plug the LDA estimates (14) in (17), to obtain the MAP surrogate. However, there are two issues in this strategy: First, the ML estimate σ 2 * can raise a numerical issue in the few-shot learning since the number of samples is too small 1 , although π * and µ * incur no such issue. Secondly, it is only the ML estimate with data S, and we have not taken into account the prior on {w j , b j } j . To address both issues simultaneously, we propose a prior-norm adjustment strategy, which also leads to a sensible estimate for σ 2 .
Prior-norm adjustment. We will find σ 2 that makes the surrogate w j in (17) become consistent with our prior p(w j ) = N (0, β 2 I). Since w j sampled from the prior can be written as w j = [βǫ j1 , . . . , βǫ jd ] ⊤ with ǫ j1 , . . . , ǫ jd iid ∼ N (0, 1), we have: where the approximation to the expectation gets more accurate as d increases due to the law of large numbers. The equation (18) implies that any w j that conforms to the prior has the norm approximately equal to β √ d. Hence we enforce this to the surrogate w j in (17) to determine σ 2 . To consider all j = 1 · · · C, we establish a simple mean-square equation, (1/C) C j=1 ||µ * j /σ 2 || 2 = β 2 d, and the solution leads to the prior-norm adjusted MAP surrogate (denoted by w * j ) as follows: Determining α. We adjust b j to take into account its prior, and from (17) this amounts to finding α properly. We directly optimize the log-posterior (24) with respect to α. Denotingb j = log π * j − ||µ * j || 2 /(2σ 2 * ) (i.e., b j =b j + α), we solve = 0, and have a MAP surrogate (denoted by b * j ) as: Algorithm 1 GPLDLA meta training and meta test.
Summary. We have derived the Laplace approximated posterior p(w j , b j |S) in (11)(12) with the MAP surrogate (w * j , b * j ) from (19) and (20). From this GP posterior, we derive the predictive distribution p(y|x, S) that is used in our meta training (10) as well as meta test. We adopt the Monte Carlo estimate with M (reparametrized) samples from the posterior: where w where ǫ

Related Work
Few-shot/meta learning [3,27] has received enormous attention recently with the surge of deep learning, and it now has a large body of literature [36,14,15]. The approaches in few-shot learning can broadly fall into two folds: feature transfer and the other meta learning. The former uses the entire training data to pretrain the feature extractor network, which is then adapted to a new task by finetuning the network or training the output heads with the few-shot test data [35,5]. On the other hand, the meta learning approaches [28,34,8] follow the learning-to-learn paradigm [32], where the meta learner is trained by the empirical risk minimization principle.
In the Bayesian meta learning [11,41,10,16,39], the prior on the underlying model parameters typically serves as the meta learner, and the adaptation to a new task corresponds to inference of the posterior predictive distribution. In this way the meta learning amounts to learning a good prior distribution from many training episodes. For the efficient meta training, the posterior predictive inference, i.e., adaptation procedure, needs to be fast and succinct (e.g., in closed forms). Some previous approaches used neural net approximation of the posterior predictive distribution (i.e., amortization) [10,39], while others are based on gradient updates [8,11,16]. The main focus of meta few-shot learning lies on how to learn the meaningful prior model that can be quickly and accurately adaptable to novel tasks with only a limited amount of evidence.
Another recent Bayesian meta learner closely related to ours is MetaQDA [42], where they consider a mixture-of-Gaussians (MoG) classifier possibly with non-equal covariances, thus representing quadratic decision boundaries. With the Normalized-Inverse-Wishart prior on the MoG parameters, the posterior admits a closed-form expression by conjugacy. One of the key differences from our approach is that the MetaQDA deals with the joint MoG modeling p(x, y), while we focus on the discriminative p(y|x). It is known that the discriminative model has lower asymptotic error and is more data efficient without requiring marginal input distribution modeling [20]. But this comes at the cost of the non-closed-form posterior, and we had to resort to Laplace approximation with the prior-norm adjusted MAP estimates. Despite superb performance, there are several shortcomings of MetaQDA: it involves a large number of Wishart prior parameters to be trained, O(Cd 2 ) for C-way classification and d-dimensional features. On the other hand, ours has only two extra scalar parameters β, β b . Moreover, MetaQDA's performance is rarely known when the backbone feature extractor network φ(x) is jointly trained. They rather fix the features and only learn the prior QDA model. Its performance is highly reliant on the underlying feature extractor used.

Experiments
In this section we test our GPLDLA on several popular benchmark tasks/datasets in meta few-shot classification. We demonstrate the performance improvement over the state-of-the-arts, especially highlighting more accurate prediction than the previous GP few-shot model, GPDKT [22].
Implementation details. For fair comparison with existing approaches, we use the same feature extractor backbone network architectures φ θ (x) (e.g., convolutional networks or ResNets [13]) as competing models such as ProtoNet [28], Baseline++ [5], SimpleShot [35], and GPDKT [22]. For all experiments we use normalized features (φ(x) → φ(x) ||φ(x)|| ), which corresponds to the cosinesimilarity outer kernel with the original feature in our deep kernel GP covariance function (8). As the GP prior parameters β, β b , the only extra parameters, are constrained to be positive, we represent them as exponential forms and perform gradient descent in the exponent space. The number of Monte Carlo samples is fixed as M = 10 for all experiments. The details of the optimization hyperparaemters are described in Appendix.

Within-domain classification
The results on the CUB, miniImageNet, and tieredImageNet datasets are summarized in Table 1, Table 2, and Table 3, respectively. To have fair comparison with existing approaches, we test our model on the four-layer convolutional network (known as Conv-4) used in [28,34] and ResNet-10 as the backbone networks for the CUB dataset. For the miniImageNet and tieredImageNet, we use the Conv-4 and ResNet-18. We compare our GPLDLA with several state-of-the-arts, including MAML [8], ProtoNet [28], MatchingNet [34], and RelationNet [30]. We also consider the simple feature transfer, as well as strong baselines such as Baseline++ [5] and SimpleShot [35]. Among others, the (hierarchical) Bayesian approaches including VERSA [10], LLAMA [11], and Meta-Mixture [16], are also compared. However, we exclude methods that use more complex backbones or more sophisticated learning schedules [40,2,21,23], and those that require a large number of extra parameters to be trained [42].
Our approach achieves the best performance on most of the setups. On the CUB dataset, GPLDLA attains the highest accuracies for three cases out of four. On the miniImageNet, GPLDLA exhibits significantly higher performance than competing methods when the simpler backbone (Conv-4) is used, while being the second best and comparable to SimpleShot with the ResNet18 backbone 2 . And our GPLDLA outperforms GPDKT with all different kernels in most of the cases. GPLDLA also performs the best on tieredImageNet.   Table 4 where we use the Conv-4 backbone for both cases. GPLDLA exhibits the best performance for most cases outperforming GPDKT, except for one case. Our GPLDLA also performs comparably well with recent approaches with the ResNet-18 backbone on the miniImageNet→CUB task as shown in Table 5.

Calibration errors
Considering the practical use of the machine learning algorithms, it is important to align the model's prediction accuracy and its prediction confidence. For instance, when model's prediction is wrong, it would be problematic if the confidence of prediction is high. In this section we evaluate this alignment measure for our approach. Specifically we employ the expected calibration error (ECE) [12] as the measure of misalignment. The ECE can be computed by the following procedure: the model's prediction confidence scores on the test cases are sorted and partitioned into H bins (e.g., H = 20), and for each bin we compute the difference between prediction accuracy (on the test examples that belong to the bin) and the confidence score of the bin. The ECE is the weighted average of these differences over the bins with the weights proportional to the numbers of bin samples. Hence the smaller the better.
Following [22], we sample 3000 tasks from the test set on the CUB dataset, and calibrate the temperature parameter by minimizing the negative log-likelihood score, and use another 3000 tasks from the test data to evaluate the ECE loss. The ECE losses averaged over five random runs are summarized in Table 6. On the 1-shot case, our GPLDLA attains the lowest calibration error, while being slightly worse than ProtoNet and GPDKT on 5-shot.

Ablation study
To verify the impact of the proposed approximation strategy of LDA plugin and prior-norm adjustment, we conduct ablation study in this section. We compare three models: i) Laplace approximation that finds the MAP solution without any approximation (neither LDA plugin nor prior-norm adjustment), ii) approximate MAP estimate by LDA plugin alone without prior-norm adjustment, and iii)  both LDA plugin and prior-norm adjustment (hence our GPLDLA). For Laplace approximation we take 5 Newton steps to find the MAP solution, where the cascaded update operations are differentiable for meta learning of the prior parameters. For the LDA plugin alone, we need to estimate the class-conditional variance parameter σ 2 , and we adopt the median distance heuristics: σ set as the median of all pairwise feature distances (in the support set), a common practice for determining feature scales in kernel machines [9]. Table 7 summarizes the test performance of the three models. Overall our GPLDLA and Laplace approximation perform equally well whereas GPLDLA slightly outperforms Laplace approximation most of the time. This may be attributed to the ProtoNet-like effect of the LDA plugin estimator, which serves as an additional regularizer for few-shot learning. We also see that LDA plugin alone significantly underperforms GPLDLA, implying that prior-norm adjustment is more effective than heuristic median-distance rule in determining the class conditional feature scale. However, the differences are less pronounced for larger backbones (ResNets) in which the importance of the sophisticated features is dominant.

Running times
Next we measure the wall clock running times for competing Bayesian meta few-shot methods. The per-episode inference time with the Conv-4 backbone on tieredImageNet is reported in Table 8. It shows that GPLDLA is the fastest thanks to the efficient closed-form inference steps. GPDKT [22] has computational overhead of solving C binarized problems separately where C is the number of Table 8: Per-episode inference time (milliseconds) with the Conv-4 backbone on tieredImageNet. We exclude the feature computation times and only measure the inference time, that is, time for computing p(y|x, S) for (x, y) ∈ Q.

Conclusion
We proposed a novel GP meta learning algorithm for few-shot classification. We adopt the Laplace posterior approximation but circumvent iterative gradient steps for finding the MAP solution by our novel LDA plugin with prior-norm adjustment. This enables closed-form differentiable GP posteriors and predictive distributions, thus allowing fast meta training. We empirically verified that our approach attained considerable improvement over previous approaches in both standard benchmark datasets and cross-domain adaptation scenarios. We would like to conclude the paper by discussing some limitations of the proposed approach. In the current study we have not carried out rigorous theoretical analysis on the impact of the cascade of approximations that we used. Our strategy looks reasonable, and works well on empirical data, but further theoretical study needs to be carried out as future work.