@inproceedings{
y{\"u}ksel2023firstorder,
title={First-order {ANIL} provably learns representations despite overparametrisation},
author={O{\u{g}}uz Y{\"u}ksel and Etienne Boursier and Nicolas Flammarion},
booktitle={NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning},
year={2023},
url={https://openreview.net/forum?id=89fKHVtsMR}
}
Meta-learning methods leverage data from previous tasks to learn a new task in a sample-efficient manner. In particular, model-agnostic methods look for initialisation points from which gradient descent quickly adapts to any new task.
Although it has been empirically suggested that such methods learns shared representations during pretraining, there is limited theoretical evidence of such behavior. In this direction, this work shows, in the limit of infinite tasks, first-order ANIL with a linear two-layer network successfully learns linear shared representations. This result even holds under _overparametrisation_; having a width larger than the dimension of the shared representations results in an asymptotically low-rank solution.
First-order ANIL provably learns representations despite overparametrisation
Oğuz Yüksel Etienne Boursier Nicolas Flammarion