Existing models also suffer from the problem of generating text from their train sets. We find that \(1.38\%\) of the tokens in the official release of 25k GROVER-Mega outputs 3 are part of verbatim matches in RealNews of at least length \(50\). Likewise, more than 5% of the tokens in \(\mathtt{\sim}\)200k sequences outputted by GPT-Neo 1.3B (Black et al., 2021) are part of a \(50\) token matches of its training data, the Pile (Gao et al., 2020).