@inproceedings{
mccabe2023multiple,
title={Multiple Physics Pretraining for Physical Surrogate Models},
author={Michael McCabe and Bruno R{\'e}galdo-Saint Blancard and Liam Parker and Ruben Ohana and Miles Cranmer and Alberto Bietti and Michael Eickenberg and Siavash Golkar and Geraud Krawezik and Francois Lanusse and Mariel Pettee and Tiberiu Tesileanu and Kyunghyun Cho and Shirley Ho},
booktitle={NeurIPS 2023 AI for Science Workshop},
year={2023},
url={https://openreview.net/forum?id=M12lmQKuxa}
}
We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling. MPP involves training large surrogate models to predict the dynamics of multiple heterogeneous physical systems simultaneously by learning features that are broadly useful across diverse physical tasks. In order to learn effectively in this setting, we introduce a shared embedding and normalization strategy that projects the fields of multiple systems into a single shared embedding space. We validate the efficacy of our approach on both pretraining and downstream tasks. In pretraining, we show that a single MPP-pretrained model is able to match or outperform task-specific baselines on all training sub-tasks without the need for finetuning. For downstream tasks, we explore how the benefits of MPP scale with available finetuning data and demonstrate pretraining gains even across large physics gaps. We open-source our code and model weights trained at multiple scales for reproducibility and community experimentation.
Multiple Physics Pretraining for Physical Surrogate Models
Michael McCabe Bruno Régaldo-Saint Blancard Liam Holden Parker Ruben Ohana Miles Cranmer Alberto Bietti Michael Eickenberg Siavash Golkar Geraud Krawezik Francois Lanusse Mariel Pettee Tiberiu Tesileanu Kyunghyun Cho Shirley Ho