@inproceedings{
banerjee2023mermaide,
title={{MERMAIDE}: Learning to Align Learners using Model-Based Meta-Learning},
author={Arundhati Banerjee and Soham Phade and Stefano Ermon and Stephan Zheng},
booktitle={NeurIPS 2023 Workshop on Generalization in Planning},
year={2023},
url={https://openreview.net/forum?id=JNIv4ghbGy}
}
We study how a principal can efficiently and effectively intervene on the rewards of a previously unseen *learning* agent in order to induce desirable outcomes. This is relevant to many real-world settings like auctions or taxation, where the principal may not know the learning behavior nor the rewards of real people. Moreover, the principal should be few-shot adaptable and minimize the number of interventions, because interventions are often costly. We introduce MERMAIDE, a model-based meta-learning framework to train a principal that can quickly adapt to out-of-distribution agents with different learning strategies and reward functions. We validate this approach step-by-step. First, in a Stackelberg setting with a best-response agent, we show that meta-learning enables quick convergence to the theoretically known Stackelberg equilibrium at test time, although noisy observations severely increase the sample complexity. We then show that our model-based meta-learning approach is cost-effective in intervening on bandit agents with unseen explore-exploit strategies. Finally, we outperform baselines that use either meta-learning or agent behavior modeling, in both $0$-shot and $1$-shot settings with partial agent information.
MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning
Arundhati Banerjee Soham Phade Stefano Ermon Stephan Zheng