How to count tokens with tiktoken
Encodings
Encodings specify how text is converted into tokens. Different models use different encodings.

tiktoken supports three encodings used by OpenAI models:

Encoding name	OpenAI models
cl100k_base	gpt-4, gpt-3.5-turbo, text-embedding-ada-002
p50k_base	Codex models, text-davinci-002, text-davinci-003
r50k_base (or gpt2)	GPT-3 models like davinci
You can retrieve the encoding for a model using tiktoken.encoding_for_model() as follows: