We acknowledge support from NSERC Discovery Grant RGPIN- 2021-04104 [E.B.], the Canada CIFAR AI Chair Program [I.R.], and the Canada Excellence Research Chairs Program [I.R.]. We would also like to acknowledge funding from the FRQNT Doctoral (B2X) scholarship [B.T.], the scholarship for Artificial Intelligence of Universite de Montreal's Etudes Superieures et Postdoctorales [A.I.], and a fellowship of the IFI program of the German Academic Exchange Service (DAAD)[M.R.]. This research was made possible thanks to the computing resources on the Summit supercomputer, provided as a part of the INCITE 2023 program award "Scalable Foundation Models for Transferable Generalist AI". These resources were provided by the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. In particular, we thank Jens Glaser for his help with the Summit supercomputer.