Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks
Few-shot learning for neural networks (NNs) is an important problem that aims
to train NNs with a few data. The main challenge is how to avoid overfitting
since over-parameterized NNs can easily overfit to such small dataset. Previous
work (e.g. MAML by Finn et al. 2017) tackles this challenge by meta-learning,
which learns how to learn from a few data by using various tasks. On the other
hand, one conventional approach to avoid overfitting is restricting hypothesis
spaces by endowing sparse NN structures like convolution layers in computer
vision. However, although such manually-designed sparse structures are
sample-efficient for sufficiently large datasets, they are still insufficient
for few-shot learning. Then the following questions naturally arise: (1) Can we
find sparse structures effective for few-shot learning by meta-learning? (2)
What benefits will it bring in terms of meta-generalization? In this work, we
propose a novel meta-learning approach, called Meta-ticket, to find optimal
sparse subnetworks for few-shot learning within randomly initialized NNs. We
empirically validated that Meta-ticket successfully discover sparse subnetworks
that can learn specialized features for each given task. Due to this task-wise
adaptation ability, Meta-ticket achieves superior meta-generalization compared
to MAML-based methods especially with large NNs.