We first train a series of GPT2 models on the \(\mathsf{bioS}(N)\) datasets (see Section 2.2) using mixed-precision fp16. The training protocol ensures that each piece of knowledge is presented 1000 times, a process we refer to as "1000 exposures." It's important to clarify that this differs from making 1000 passes over the data. For example, a single pass through Wiki data might expose the knowledge (US, capital, Washington D.C.) 1000 times, whereas a pass through the Common Crawl might do so a million times.