We train a BPE tokenizer in SentencePiece (Kudo and Richardson, 2018) on data that is randomly sampled from the final 8T token dataset. To have better coverage of low-resource languages in the tokenizer, we upsample non-English data relative to the final training dataset distribution. Our tokenizer preserves whitespaces (including leading and trailing ones), splits numbers into their individual digits (Chowdhery et al., 2022), and relies on byte-level backoff to handle unknown character sequences. The final vocabulary size is 256,000 tokens.