
Baseline (fine-tune) [1]} represents a classical way of applying transfer learning, where a neural network is pre-trained on a large dataset, then fine-tuned on a smaller domain-specific dataset. The baseline's backbone followed a single linear classification layer with a single output for each meta-training dataset class. The whole network was trained during pre-training. During meta-testing, the baseline's pre-trained linear layer was exchanged for another randomly initialized classification layer with outputs matching the task's number of classes (\(N\) -way). Fine-tuning was performed on the new randomly initialized classification layer using the support set \(\mathcal {S}\) .

Baseline (1-NN) is another classical method of applying transfer learning but using a k-nearest neighbor classifier instead of the classification layer during meta-validation. Pre-training was performed in the same was the fine-tune baselines. During the meta-testing time, instead of fine-tuning, the model matched query samples to the nearest support sample's class based on Euclidian distance.

Baseline++ [2]} augments the fine-tune baseline by using Cosine Similarity on the last layer.

Matching Network (Matching Net) [3]} uses context embeddings with an LSTM to effectively perform k-nearest neighbor in embedding space using cosine similarity to classify the query set.

Prototypical Networks (ProtoNet) [4]} maps images into a feature space and calculates class means (called prototypes). The query samples are then classified based on the closest Euclidian distance to a classes' prototype. We evaluate two models, one meta-trained like the others on 5-way episodes, and another variation trained on 20-way episodes. During 20-way meta-training, we set the class' query size to 5.

Relation Networks (Relation Net) [5]} augment the classical Prototypical Networks by introducing a relation module (another neural network) that compares the distance instead of using Euclidian. The original method uses Mean Squared Error to minimize the relation score between samples of the same type. However, we follow work by [2]}, use cross-entropy loss that expediates meta-training. The structure of the relation module is described in section REF .

DKT (formally called GPShot) proposed by [7]} is a probabilistic approach that utilizes the Gaussian Processes with a deep neural network as a kernel function. We used Batch Norm Cosine distance for the kernel type.

SimpleShot [8]} augments the 1-NN baseline model by normalizing and centering the feature vector using the dataset's mean feature vector. The query samples are assigned to the nearest prototype's class according to Euclidian distance. In contrast to the baseline models, pre-training is performed on the meta-training dataset like other meta-learning algorithms, and meta-validation is used to select the best model based on performance on tasks sampled from \(\mathcal {D}_{val}\) .

MAML [9]} is a meta-learning technique that learns a common initialization of weights that can be quickly adapted for task using fine-tuning on the support set. The task adaptation process uses a standard gradient descent algorithm minimizing Cross-Entropy loss on the support set. The original method uses second-order derivates; however, due to more efficient calculation, we use the first-order MAML, which has been shown to work just as well. We set the inner-learning rate to 0.1 with 10 iteration steps. We optimize the meta-learner model on batches of 4 meta-training tasks. These hyperparameters were selected based on our hyperparameter fine-tuning.

ProtoMAML [10]} augments traditional first-order MAML by reinitializing the last classification layer between tasks. Specifically, the weights of the layer are assigned to the prototype for each class's corresponding output. This extra step combines the fine-tuning ability of MAML and the class regularisation ability of Prototypical Networks. We set the inner-loop learning rate to 0.1 with 10 iterations. Unlike for MAML, we found that updating the meta-learner after a single meta-training task gave the best performance.

Bayesian-MAML (BMAML) [11]} augments the MAML method by replacing the inner loop's standard stochastic gradient descent with Bayesian gradient-based updates. BMAML uses Stein Variational Gradient Descent that is a non-parametric variational inference method combining strengths of Monte Carlo approximation and variational inference. The algorithm learns to approximate a posterior over the initialization parameters conditioned on the task support set. [11]} also adds a chaser loss, which utilizes the samples in the query set during meta-training to approximate the true task posterior. Minimization of the KL divergence between the true task posterior and the estimated task parameter posterior can be used to drive the meta-training process. We set the inner-loop learning rate to 0.1 with 1 inner-loop step. We found that using a higher inner-loop step number destabilized performance. We used 20 particles. The chaser loss variation used a learning rate of 0.5. Again, we found these combinations of hyperparameters to give the best results.

Bayesian-TAML (BTAML) [13]} [Left out of the main paper body.] This method augments the MAML method with four main changes: 1) task-dependent parameter initialization \(z\) , 2) task-dependent per-layer learning rate multipliers \(\gamma \) , 3) task-dependent per-class learning rate multiplier \(\omega \) , 4) meta-learned per-parameter learning rate \(\alpha \)  (similar to Meta-SGD, [14]}). However, our results for BTAML were unstable, suggesting a fault in our implementation. Unfortunately, we did not identify it in time for the submission, and we decided to move the method's results to the appendix. In our experiments, we explored several variations to this method with various parameters (\(z,\omega ,\gamma ,\alpha \) ) turned on and off, as well as different hyperparameters. We found that using a meta-learning rate of \(10^{-4}\)  performed better than \(10^{-3}\)  in contrast to the other models. We set the inner-learning rate to 0.1 with 10 iteration steps, and optimize the meta-learner model on batches of 4 meta-training tasks. Again, we found this set up worked best in our experiments.

