Is training a reference model necessary?To score tokens, we need a high-quality reference model. This could be a base model trained with a small amount of high-quality data, or a performant open-source model. In fact, since we only need input logprobs or perplexity from reference model, we could even utilize more powerful proprietary model APIs. We can input tokens and use the log probabilities of the input returned by the API as reference scores. We leave this for future works.