1. Over \(1\%\) of tokens emitted unprompted from a model trained on standard datasets (e.g., C4) are part of a memorized sequence (See SS6.2)--even though the 1.5 billion parameter model is much smaller than the 350GB dataset it was trained on. By deduplicating the training dataset we reduce the rate of emitting memorized training data by a factor of \(10\times\). 2. Train-test overlap is common in nondeduplicated datasets. For example, we find _a 61-word sequence_1 in C4 (Raffel et al., 2020) that is repeated \(61{,}036\) times verbatim in the training dataset and \(61\) times in the validation set (\(0.02\%\) of the samples in each dataset).