Recent LM releases have varied in their degree of openness. For example, Mistral 8x7B provided model weights and a brief report (Jiang et al., 2024), while LLaMA came with in-depth adaptation training instructions (Touvron et al., 2023), and Mosaic Pretrained Transformer came with many details, including the dataset distribution, though not the data itself (MosaicML NLP Team, 2023). Falcon's pretraining data was partially released (Almazrouei et al., 2023), and the most open models--the Pythia suite (Biderman et al., 2023) and BLOOM (BigScience et al., 2022)--released training code, model checkpoints, training data and more.