How to count tokens with tiktoken
5. Comparing encodings
Different encodings vary in how they split words, group spaces, and handle non-English characters. Using the methods above, we can compare different encodings on a few example strings.