Figure 17: Scaling laws for the GPT2 MoE models with 32 experts on the \(\mathsf{bioS}(N)\) data for \(\mathsf{100}\) exposures. This figure complements Figure 16 by comparing the effects of varying _topk_ and _cap_factor_ in the 100-exposure insufficiently-trained regime. **Conclusion:** minimal differences are observed across these settings, though deeper models (e.g., GPT2-16-4 with 32 experts) seem easier to train with _topk_ = _cap_factor_ = 2.