No token for "the"?

by FremyCompany - opened Feb 28, 2023

Feb 28, 2023

I think there might be something strange going on with the tokenizer, because it does not have a token for "Ġthe" despite having a token for "ĠThe": 409.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment