@inproceedings{
lanusse2023astroclip,
title={Astro{CLIP}: Cross-Modal Pre-Training for Astronomical Foundation Models},
author={Francois Lanusse and Liam Parker and Siavash Golkar and Alberto Bietti and Miles Cranmer and Michael Eickenberg and Geraud Krawezik and Michael McCabe and Ruben Ohana and Mariel Pettee and Bruno R{\'e}galdo-Saint Blancard and Tiberiu Tesileanu and Kyunghyun Cho and Shirley Ho},
booktitle={NeurIPS 2023 AI for Science Workshop},
year={2023},
url={https://openreview.net/forum?id=Zt35qMfDTp}
}
We present AstroCLiP, a strategy to facilitate the construction of astronomical foundation models that bridge the gap between diverse astronomical observational modalities. We demonstrate that a cross-modal contrastive learning approach between images and spectra of galaxies yields highly informative embeddings of both modalities. In particular, we apply our method on multi-band images and spectrograms from the Dark Energy Spectroscopic Instrument (DESI), and show that: (1) these embeddings are well-aligned between modalities and can be used for accurate cross-modal searches, and (2) these embeddings encode valuable physical information about the galaxies - in particular redshift and stellar mass - that can be used to achieve competitive zero- and few- shot predictions without further finetuning. Additionally, in the process of developing our approach, we also construct a novel, transformer-based model and pretraining approach for galaxy spectra.
AstroCLIP: Cross-Modal Pre-Training for Astronomical Foundation Models
Francois Lanusse Liam Holden Parker Siavash Golkar Alberto Bietti Miles Cranmer Michael Eickenberg Geraud Krawezik Michael McCabe Ruben Ohana Mariel Pettee Bruno Régaldo-Saint Blancard Tiberiu Tesileanu Kyunghyun Cho Shirley Ho