Humans show a remarkable capability to accurately solve a wide range of problems efficiently – utilizing a limited amount of computation and experience.
Deep learning models, by stark contrast, can be trained to be highly accurate on a narrow task while being highly inefficient in terms of the amount of compute and data required to reach that accuracy.
Within natural language processing (NLP), recent breakthroughs in unsupervised pretraining have enabled reusable models that can be applied to many NLP tasks, however, learning of new tasks is still inefficient [1]}, [2]}, [3]}.
Meta-learning [4]}, [5]}, [6]} treats the learning process itself as a learning problem from data, with the goal of learning systems that can generalize to new tasks efficiently.
This has the potential to produce few-shot learners that can accurately solve a wide range of new tasks.
However, meta-learning requires a distribution over tasks with relevant labeled data that can be difficult to obtain, severely limiting the practical utility of meta-learning methods.
