<story><title>Tiktoken: OpenAI’s Tokenizer</title><url>https://github.com/openai/tiktoken</url></story><parent_chain></parent_chain><comment><author>joelburget</author><text>A few interesting findings:&lt;p&gt;* the cl100k_base tokenizer has ~100k tokens -- previous tokenizers had ~50k. (enc.n_vocab gives 100277 but some numbers in that range don&amp;#x27;t work, starting at 100256)&lt;p&gt;* it has exactly 1110 tokens which are just digits. 10 1 digit tokens, 100 2 digit tokens and 1000 3 digit tokens! (none have preceding spaces). this is a huge improvement from GPT2&amp;#x27;s tokenizer, which was a huge mess.&lt;p&gt;* there are &amp;lt;|fim_prefix|&amp;gt;, &amp;lt;|fim_middle|&amp;gt;, and &amp;lt;|fim_suffix|&amp;gt; tokens (see &lt;i&gt;Efficient Training of Language Models to Fill in the Middle&lt;/i&gt;)&lt;p&gt;The biggest news to me is the improved handling of numbers. This could explain some improved performance on arithmetic. One disappointment is that it tokenizes from the &lt;i&gt;front&lt;/i&gt;, e.g. &amp;quot;1000000&amp;quot; -&amp;gt; 100|000|0. This is one of those &amp;quot;so close!&amp;quot; moments -- I would work for free to fix this.</text></comment>
<story><title>Tiktoken: OpenAI’s Tokenizer</title><url>https://github.com/openai/tiktoken</url></story><parent_chain></parent_chain><comment><author>est</author><text>Requires az:&amp;#x2F;&amp;#x2F; blob download.&lt;p&gt;I hope pypi libraries can provide complete standalone offline versions instead of requests+urllib3+some_object_storage shenanigans.&lt;p&gt;If these blobs are too large to host it on pypi, maybe give us an alternative way to download it altogether so we can deploy the full lib to a server without network access?</text></comment>