Submitted for Meta internal review on March 14, 2024. We would like to thank Lin Xiao and Yuchen Zhang for many helpful conversations. We would like to extend special thanks to Ian Clark, Gourab De, Anmol Mann, and Max Pfeifer from W&B, as well as Lucca Bertoncini, Liao Hu, Caleb Ho, Apostolos Kokolis, and Shubho Sengupta from Meta FAIR NextSys; Henry Estela, Wil Johnson, Rizwan Hashmi, and Lucas Noah from Meta Cloud Foundation; without their invaluable support, the extensive experiments in this paper would not have been possible.