Meta-learning methods, most notably Model-Agnostic Meta-Learning (Finn et al, 2017) or MAML, have achieved great success in adapting to new tasks quickly, after having been trained on similar tasks.
 The mechanism behind their success, however, is poorly understood.
 We begin this work with an experimental analysis of MAML, finding that deep models are crucial for its success, even given sets of simple tasks where a linear model would suffice on any individual task.
 Furthermore, on image-recognition tasks, we find that the early layers of MAML-trained models learn task-invariant features, while later layers are used for adaptation, providing further evidence that these models require greater capacity than is strictly necessary for their individual tasks.
 Following our findings, we propose a method which enables better use of model capacity at inference time by separating the adaptation aspect of meta-learning into parameters that are only used for adaptation but are not part of the forward model.
 We find that our approach enables more effective meta-learning in smaller models, which are suitably sized for the individual tasks.
 Meta-learning or learning to learn is an appealing notion due to its potential in addressing important challenges when applying machine learning to real-world problems. In particular, learning from prior tasks but being able to to adapt quickly to new tasks improves learning efficiency, model robustness, etc. A promising set of techiques, Model-Agnostic Meta-Learning (Finn et al., 2017) or MAML, and its variants, have received a lot of interest (Nichol et al., 2018; Lee & Choi, 2018; Grant et al., 2018) . However, despite several efforts, understanding of how MAML works, either theoretically or in practice, has been lacking Fallah et al., 2019 ). For a model that meta-learns, its parameters need to encode not only the common knowledge extracted from the tasks it has seen, which form a task-general inductive bias, but also the capability to adapt to new test tasks (similar to those it has seen) with task-specific knowledge. This begs the question: how are these two sets of capabilities represented in a single model and how do they work together? In the case of deep learning models, one natural hypothesis is that while knowledge is represented distributedly in parameters, they can be localized -for instance, lower layers encode task-general inductive bias and the higher layers encode adaptable task-specific inductive bias. This hypothesis is consistent with one of deep learning's advantages in learning representations (or feature extractors) using its bottom layers. Then we must ask, in order for a deep learning model to meta-learn, does it need more depth than it needs for solving the target tasks? In other words, is having a large capacity to encode knowledge that is unnecessary post-adaptation the price one has to pay in order to be adaptable? Is there a way to have a smaller (say, less deep) meta-learnable model which still adapts well? This question is of both scientific interest and practical importance -a smaller model has a smaller (memory) footprint, faster inference and consumes less resources. In this work, through empirical studies on both synthetic datasets and benchmarks used in the literature, we investigate these questions by analyzing how well different learning models can meta-learn and adapt. We choose to focus on MAML due to its popularity. Our observations suggest depth is indeed necessary for meta-learning, despite the tasks being solvable using a shallower model. Thus, applying MAML to shallower models does not result in successful meta-learning models that can adapt well. Moreover, our studies also show that higher layers are responsible more for adapting to new tasks while the lower layers are responsible for learning task-general features. Our findings prompt us to propose a new method for meta-learning. The new approach introduces a meta-optimizer which learns to guide the (parameter) optimization process of a small model. The small model is used for solving the tasks while the optimizer bears the burden of extracting the knowledge of how to adapt. Empirical results show that despite using smaller models, the proposed algorithm with small models attains similar performance to larger models which use MAML to meta-learn and adapt. We note that a recent and concurrent work to ours addresses questions in this line of inquiry (Raghu et al., 2019) . They reach similar conclusions through different analysis and likewise, they propose a different approach for improving MAML. We believe our work is complementary to theirs. We introduce our approach by analyzing the success and failure modes of optimization-based metalearning methods. Namely, we find that, when successful, these methods tend to learn task-general features in early layers and adaptable parameters/update functions in the later layers. Moreover, we find that this learning fails when model size is reduced, indicating that optimization-based metalearning methods rely on the ability to encode task-general features and/or adaptable parameters, even when the model itself is adequate for learning on the individual tasks. As such, we introduce our method for decomposing modelling from adaptation using factored meta-optimizers. These meta-optimizers enable the forward model to use more capacity on learning task-specific features, while the expressiveness of their updates allows the forward model to adapt quickly to different tasks. We find that our approach is able to enable successful meta-learning in models that do not work with traditional optimization-based meta-learning methods.
We find that deep models are crucial for MAML to work and propose a method which enables effective meta-learning in smaller models.
{"src": "scitldr-AIC_train", "paper_id": "BkljIlHtvS"}