Pretrained models are not always used as-is, but rather further fine-tuned to improve their performance, safety, and usability. Often models are first trained to follow instructions (Mishra et al., 2022; Wei et al., 2022; Sanh et al., 2022), and then further trained on human preferences (Ouyang et al., 2022) to improve the quality of their generations. We showcase the efficacy of using OLMo as a base model for further fine-tuning by training OLMo to be a general chat assistant following our Open Instruct (Tulu) data and training setup (Ivison et al., 2023). Our approach involves first performing instruction fine-tuning with a mixture of distilled and human-written instruction data and then further aligning the model with distilled preference data using Direct Preference Optimization (DPO) (Rafailov et al., 2023). We experimented with mixing the Tulu instruction data at the end of pretraining, as done in recent models such as DeepSeek-AI et al. (2024), but did not have conclusive findings.