Effect of Token Select RatioWe investigate the impact of token selecting ratios of the SLM. Generally, the selecting ratio is defined by heuristic rules, similar to the approach previously employed in the training of Masked Language Models (MLMs) (Devlin et al., 2019; Liu et al., 2019). As shown in Figure 9, the selected tokens is suitable for accounting for about 60% of the original tokens.