README.md

---
title: Tokenizer Arena
emoji: ⚡
colorFrom: red
colorTo: gray
sdk: gradio
sdk_version: 4.28.3
app_file: app.py
pinned: false
---



## 压缩率 Compress Rate


在 [cc-100](https://huggingface.co/datasets/cc100) 数据集，每个语言取1万条数据，测试不同tokenizer的压缩率。

> 压缩率示例：
llama3扩充了词典，具有更高的压缩比。同样1T字节的简体中文语料，llama分词后是 0.56万亿个token，llama3只需要0.31万亿个token。

| tokenizer                    |   vocab_size |    t_bytes/t_tokens |   t_tokens/t_bytes |   n_chars/n_tokens |
|:-----------------------------|-------------:|-------------------:|-------------------:|-------------------:|
| llama                        |        32000 |               1.8  |               0.56 |               0.7  |
| llama3                       |       128000 |               3.2  |               0.31 |               1.24 |

可通过以下脚本进行复现 
```sh
python utils/compress_rate_util.py 
```




<details> <summary>英文压缩率</summary>
在英文数据集 cc100-en 计算压缩率 

| tokenizer                   |   vocab_size |   g_bytes/b_tokens |   b_tokens/g_bytes |   t_bytes/t_tokens |   t_tokens/t_bytes |   n_chars/n_tokens |
|:----------------------------|-------------:|-------------------:|-------------------:|-------------------:|-------------------:|-------------------:|
| amber                       |        32000 |               3.56 |               0.28 |               3.47 |               0.29 |               3.81 |
| aya_101                     |       250100 |               3.3  |               0.3  |               3.22 |               0.31 |               3.53 |
| baichuan                    |        64000 |               3.74 |               0.27 |               3.65 |               0.27 |               4    |
| baichuan2                   |       125696 |               3.89 |               0.26 |               3.8  |               0.26 |               4.17 |
| bert_base_cased             |        28996 |               3.64 |               0.27 |               3.55 |               0.28 |               3.89 |
| bert_base_chinese           |        21128 |               2.78 |               0.36 |               2.71 |               0.37 |               2.97 |
| bert_base_uncased           |        30522 |               3.73 |               0.27 |               3.65 |               0.27 |               4    |
| bloom                       |       250680 |               4.07 |               0.25 |               3.97 |               0.25 |               4.36 |
| byt5_small                  |          256 |               0.92 |               1.08 |               0.9  |               1.11 |               0.99 |
| character_glm_6b            |        64794 |               3.62 |               0.28 |               3.54 |               0.28 |               3.88 |
| chatglm2_6b                 |        64794 |               3.62 |               0.28 |               3.54 |               0.28 |               3.88 |
| chatglm3_6b                 |        64798 |               3.62 |               0.28 |               3.54 |               0.28 |               3.88 |
| chatglm_6b                  |       150344 |               3.68 |               0.27 |               3.59 |               0.28 |               3.94 |
| chatyuan_large_v2           |        32128 |               1.95 |               0.51 |               1.91 |               0.52 |               2.09 |
| chinese_llama               |        49953 |               3.59 |               0.28 |               3.51 |               0.28 |               3.85 |
| chinese_llama2              |        55296 |               3.56 |               0.28 |               3.47 |               0.29 |               3.81 |
| code_davinci_002            |        50281 |               4.05 |               0.25 |               3.96 |               0.25 |               4.34 |
| crystal_coder               |        32000 |               3.68 |               0.27 |               3.59 |               0.28 |               3.94 |
| dbrx_instruct               |       100277 |               4.11 |               0.24 |               4.01 |               0.25 |               4.4  |
| deepseek_coder_33b_instruct |        32000 |               3.64 |               0.27 |               3.56 |               0.28 |               3.9  |
| deepseek_llm_7b_base        |       100000 |               3.85 |               0.26 |               3.76 |               0.27 |               4.12 |
| falcon_180b                 |        65024 |               3.99 |               0.25 |               3.9  |               0.26 |               4.27 |
| falcon_7b                   |        65024 |               3.99 |               0.25 |               3.9  |               0.26 |               4.27 |
| fastchat_t5_3b              |        32000 |               2.16 |               0.46 |               2.11 |               0.47 |               2.31 |
| flan_t5_base                |        32100 |               3.61 |               0.28 |               3.53 |               0.28 |               3.87 |
| gemma_7b                    |       256000 |               3.91 |               0.26 |               3.82 |               0.26 |               4.18 |
| gpt2                        |        50257 |               4.05 |               0.25 |               3.96 |               0.25 |               4.34 |
| gpt2_chinese                |        21128 |               2.67 |               0.37 |               2.61 |               0.38 |               2.86 |
| gpt_35_turbo                |       100277 |               4.11 |               0.24 |               4.01 |               0.25 |               4.4  |
| gpt_4                       |       100277 |               4.11 |               0.24 |               4.01 |               0.25 |               4.4  |
| gpt_nexo_20b                |        50254 |               4.04 |               0.25 |               3.94 |               0.25 |               4.32 |
| grok_1                      |       131072 |               4.06 |               0.25 |               3.96 |               0.25 |               4.35 |
| internlm2_chat_7b           |        92544 |               3.86 |               0.26 |               3.77 |               0.27 |               4.13 |
| internlm2_math_7b           |        92544 |               3.86 |               0.26 |               3.77 |               0.27 |               4.13 |
| internlm_chat_7b            |       103168 |               3.86 |               0.26 |               3.77 |               0.27 |               4.13 |
| internlm_xcomposer_7b       |       103168 |               3.86 |               0.26 |               3.77 |               0.27 |               4.13 |
| jamba_v0_1                  |        65536 |               3.82 |               0.26 |               3.73 |               0.27 |               4.09 |
| kplug                       |        10261 |               2.66 |               0.38 |               2.6  |               0.38 |               2.85 |
| llama                       |        32000 |               3.56 |               0.28 |               3.47 |               0.29 |               3.81 |
| llama2                      |        32000 |               3.56 |               0.28 |               3.47 |               0.29 |               3.81 |
| llama3                      |       128000 |               4.11 |               0.24 |               4.01 |               0.25 |               4.4  |
| mistral_7b                  |        32000 |               3.67 |               0.27 |               3.58 |               0.28 |               3.92 |
| mixtral_8_7b                |        32000 |               3.67 |               0.27 |               3.58 |               0.28 |               3.92 |
| mobilebert_uncased          |        30522 |               3.73 |               0.27 |               3.65 |               0.27 |               4    |
| moss                        |       106029 |               4.08 |               0.25 |               3.98 |               0.25 |               4.36 |
| mt5_large                   |       250100 |               3.3  |               0.3  |               3.22 |               0.31 |               3.53 |
| olmo_7b                     |        50280 |               4.04 |               0.25 |               3.94 |               0.25 |               4.32 |
| orion_14b_chat              |        84608 |               3.94 |               0.25 |               3.85 |               0.26 |               4.22 |
| phi_1                       |        50257 |               4.05 |               0.25 |               3.96 |               0.25 |               4.34 |
| phi_2                       |        50257 |               4.05 |               0.25 |               3.96 |               0.25 |               4.34 |
| pko_t5_large                |        50258 |               1.59 |               0.63 |               1.55 |               0.64 |               1.7  |
| prompt_clue                 |        32128 |               1.95 |               0.51 |               1.91 |               0.52 |               2.09 |
| qwen1_5_14b_chat            |       151643 |               4.06 |               0.25 |               3.97 |               0.25 |               4.35 |
| qwen_1_8b_chat              |       151851 |               4.06 |               0.25 |               3.97 |               0.25 |               4.35 |
| qwen_72b_chat               |       151851 |               4.06 |               0.25 |               3.97 |               0.25 |               4.35 |
| qwen_7b_chat                |       151851 |               4.06 |               0.25 |               3.97 |               0.25 |               4.35 |
| roberta_chinese_clue        |         8021 |               1.8  |               0.56 |               1.75 |               0.57 |               1.92 |
| skywork_13b_base            |        65519 |               3.56 |               0.28 |               3.47 |               0.29 |               3.81 |
| skywork_13b_math            |        65519 |               3.56 |               0.28 |               3.47 |               0.29 |               3.81 |
| solar_10_7b                 |        32000 |               3.67 |               0.27 |               3.58 |               0.28 |               3.92 |
| starchat_alpha              |        49152 |               3.63 |               0.28 |               3.54 |               0.28 |               3.88 |
| switch_c_2048               |        32100 |               3.61 |               0.28 |               3.53 |               0.28 |               3.87 |
| t5_base                     |        32100 |               3.61 |               0.28 |               3.53 |               0.28 |               3.87 |
| t5_large                    |        32100 |               3.61 |               0.28 |               3.53 |               0.28 |               3.87 |
| t5_small                    |        32100 |               3.61 |               0.28 |               3.53 |               0.28 |               3.87 |
| text_davinci_003            |        50281 |               4.05 |               0.25 |               3.96 |               0.25 |               4.34 |
| tigerbot_13b_chat_v2        |        60512 |               3.67 |               0.27 |               3.58 |               0.28 |               3.93 |
| tigerbot_70b_chat_v4_4k     |        65107 |               3.65 |               0.27 |               3.57 |               0.28 |               3.91 |
| wizardcoder_15b_v1          |        49152 |               3.63 |               0.28 |               3.54 |               0.28 |               3.88 |
| wizardcoder_python_7b_v1    |        32000 |               3.56 |               0.28 |               3.47 |               0.29 |               3.81 |
| wizardlm_7b_v1              |        32000 |               3.56 |               0.28 |               3.47 |               0.29 |               3.81 |
| wizardmath_70b_v1           |        32000 |               3.56 |               0.28 |               3.47 |               0.29 |               3.81 |
| xlm_roberta                 |       250002 |               3.49 |               0.29 |               3.41 |               0.29 |               3.74 |
| yi_34b                      |        64000 |               3.87 |               0.26 |               3.78 |               0.26 |               4.15 |
| yi_6b                       |        64000 |               3.87 |               0.26 |               3.78 |               0.26 |               4.15 |
| yi_vl34b                    |        64000 |               3.88 |               0.26 |               3.79 |               0.26 |               4.16 |
| zephyr_7b_beta              |        32000 |               3.67 |               0.27 |               3.58 |               0.28 |               3.92 |

</details>


<details> <summary>简体中文压缩率</summary>
在简体中文数据集 cc100-zh-Hans 计算压缩率 

| tokenizer                   |   vocab_size |   g_bytes/b_tokens |   b_tokens/g_bytes |   t_bytes/t_tokens |   t_tokens/t_bytes |   n_chars/n_tokens |
|:----------------------------|-------------:|-------------------:|-------------------:|-------------------:|-------------------:|-------------------:|
| amber                       |        32000 |               1.84 |               0.54 |               1.8  |               0.56 |               0.7  |
| aya_101                     |       250100 |               3.89 |               0.26 |               3.79 |               0.26 |               1.47 |
| baichuan                    |        64000 |               3.92 |               0.26 |               3.82 |               0.26 |               1.48 |
| baichuan2                   |       125696 |               4.53 |               0.22 |               4.42 |               0.23 |               1.71 |
| bert_base_cased             |        28996 |               2.73 |               0.37 |               2.66 |               0.38 |               1.03 |
| bert_base_chinese           |        21128 |               2.74 |               0.37 |               2.67 |               0.37 |               1.03 |
| bert_base_uncased           |        30522 |               2.73 |               0.37 |               2.67 |               0.38 |               1.03 |
| bloom                       |       250680 |               4.28 |               0.23 |               4.18 |               0.24 |               1.62 |
| byt5_small                  |          256 |               0.93 |               1.08 |               0.91 |               1.1  |               0.35 |
| character_glm_6b            |        64794 |               4.2  |               0.24 |               4.1  |               0.24 |               1.59 |
| chatglm2_6b                 |        64794 |               4.2  |               0.24 |               4.1  |               0.24 |               1.59 |
| chatglm3_6b                 |        64798 |               4.2  |               0.24 |               4.1  |               0.24 |               1.59 |
| chatglm_6b                  |       150344 |               4.65 |               0.22 |               4.54 |               0.22 |               1.76 |
| chatyuan_large_v2           |        32128 |               4.34 |               0.23 |               4.24 |               0.24 |               1.64 |
| chinese_llama               |        49953 |               3.93 |               0.25 |               3.84 |               0.26 |               1.49 |
| chinese_llama2              |        55296 |               3.92 |               0.26 |               3.83 |               0.26 |               1.48 |
| code_davinci_002            |        50281 |               1.31 |               0.77 |               1.28 |               0.78 |               0.49 |
| crystal_coder               |        32000 |               1.86 |               0.54 |               1.81 |               0.55 |               0.7  |
| dbrx_instruct               |       100277 |               2.26 |               0.44 |               2.21 |               0.45 |               0.85 |
| deepseek_coder_33b_instruct |        32000 |               3.4  |               0.29 |               3.32 |               0.3  |               1.29 |
| deepseek_llm_7b_base        |       100000 |               4.05 |               0.25 |               3.96 |               0.25 |               1.53 |
| falcon_180b                 |        65024 |               2.18 |               0.46 |               2.13 |               0.47 |               0.82 |
| falcon_7b                   |        65024 |               2.18 |               0.46 |               2.13 |               0.47 |               0.82 |
| fastchat_t5_3b              |        32000 |              13.7  |               0.07 |              13.38 |               0.07 |               5.18 |
| flan_t5_base                |        32100 |              14.13 |               0.07 |              13.8  |               0.07 |               5.34 |
| gemma_7b                    |       256000 |               3.82 |               0.26 |               3.73 |               0.27 |               1.44 |
| gpt2                        |        50257 |               1.31 |               0.77 |               1.28 |               0.78 |               0.49 |
| gpt2_chinese                |        21128 |               2.73 |               0.37 |               2.66 |               0.38 |               1.03 |
| gpt_35_turbo                |       100277 |               2.26 |               0.44 |               2.21 |               0.45 |               0.85 |
| gpt_4                       |       100277 |               2.26 |               0.44 |               2.21 |               0.45 |               0.85 |
| gpt_nexo_20b                |        50254 |               2.01 |               0.5  |               1.96 |               0.51 |               0.76 |
| grok_1                      |       131072 |               1.73 |               0.58 |               1.69 |               0.59 |               0.66 |
| internlm2_chat_7b           |        92544 |               4.23 |               0.24 |               4.13 |               0.24 |               1.6  |
| internlm2_math_7b           |        92544 |               4.23 |               0.24 |               4.13 |               0.24 |               1.6  |
| internlm_chat_7b            |       103168 |               4.23 |               0.24 |               4.14 |               0.24 |               1.6  |
| internlm_xcomposer_7b       |       103168 |               4.23 |               0.24 |               4.14 |               0.24 |               1.6  |
| jamba_v0_1                  |        65536 |               2.3  |               0.44 |               2.24 |               0.45 |               0.87 |
| kplug                       |        10261 |               2.72 |               0.37 |               2.65 |               0.38 |               1.03 |
| llama                       |        32000 |               1.84 |               0.54 |               1.8  |               0.56 |               0.7  |
| llama2                      |        32000 |               1.84 |               0.54 |               1.8  |               0.56 |               0.7  |
| llama3                      |       128000 |               3.28 |               0.3  |               3.2  |               0.31 |               1.24 |
| mistral_7b                  |        32000 |               2.36 |               0.42 |               2.3  |               0.43 |               0.89 |
| mixtral_8_7b                |        32000 |               2.36 |               0.42 |               2.3  |               0.43 |               0.89 |
| mobilebert_uncased          |        30522 |               2.73 |               0.37 |               2.67 |               0.38 |               1.03 |
| moss                        |       106029 |               4.4  |               0.23 |               4.3  |               0.23 |               1.66 |
| mt5_large                   |       250100 |               3.89 |               0.26 |               3.79 |               0.26 |               1.47 |
| olmo_7b                     |        50280 |               2.01 |               0.5  |               1.96 |               0.51 |               0.76 |
| orion_14b_chat              |        84608 |               4.63 |               0.22 |               4.52 |               0.22 |               1.75 |
| phi_1                       |        50257 |               1.31 |               0.77 |               1.28 |               0.78 |               0.49 |
| phi_2                       |        50257 |               1.31 |               0.77 |               1.28 |               0.78 |               0.49 |
| pko_t5_large                |        50258 |               0.97 |               1.03 |               0.95 |               1.06 |               0.37 |
| prompt_clue                 |        32128 |               4.34 |               0.23 |               4.24 |               0.24 |               1.64 |
| qwen1_5_14b_chat            |       151643 |               4.16 |               0.24 |               4.06 |               0.25 |               1.57 |
| qwen_1_8b_chat              |       151851 |               4.16 |               0.24 |               4.06 |               0.25 |               1.57 |
| qwen_72b_chat               |       151851 |               4.16 |               0.24 |               4.06 |               0.25 |               1.57 |
| qwen_7b_chat                |       151851 |               4.16 |               0.24 |               4.06 |               0.25 |               1.57 |
| roberta_chinese_clue        |         8021 |               2.7  |               0.37 |               2.64 |               0.38 |               1.02 |
| skywork_13b_base            |        65519 |               3.69 |               0.27 |               3.61 |               0.28 |               1.4  |
| skywork_13b_math            |        65519 |               3.69 |               0.27 |               3.61 |               0.28 |               1.4  |
| solar_10_7b                 |        32000 |               2.36 |               0.42 |               2.3  |               0.43 |               0.89 |
| starchat_alpha              |        49152 |               2.78 |               0.36 |               2.72 |               0.37 |               1.05 |
| switch_c_2048               |        32100 |              14.13 |               0.07 |              13.8  |               0.07 |               5.34 |
| t5_base                     |        32100 |              14.13 |               0.07 |              13.8  |               0.07 |               5.34 |
| t5_large                    |        32100 |              14.13 |               0.07 |              13.8  |               0.07 |               5.34 |
| t5_small                    |        32100 |              14.13 |               0.07 |              13.8  |               0.07 |               5.34 |
| text_davinci_003            |        50281 |               1.31 |               0.77 |               1.28 |               0.78 |               0.49 |
| tigerbot_13b_chat_v2        |        60512 |               4.25 |               0.24 |               4.15 |               0.24 |               1.61 |
| tigerbot_70b_chat_v4_4k     |        65107 |               4.25 |               0.24 |               4.15 |               0.24 |               1.61 |
| wizardcoder_15b_v1          |        49152 |               2.78 |               0.36 |               2.72 |               0.37 |               1.05 |
| wizardcoder_python_7b_v1    |        32000 |               1.84 |               0.54 |               1.8  |               0.56 |               0.7  |
| wizardlm_7b_v1              |        32000 |               1.84 |               0.54 |               1.8  |               0.56 |               0.7  |
| wizardmath_70b_v1           |        32000 |               1.84 |               0.54 |               1.8  |               0.56 |               0.7  |
| xlm_roberta                 |       250002 |               3.96 |               0.25 |               3.86 |               0.26 |               1.5  |
| yi_34b                      |        64000 |               4.17 |               0.24 |               4.07 |               0.25 |               1.58 |
| yi_6b                       |        64000 |               4.17 |               0.24 |               4.07 |               0.25 |               1.58 |
| yi_vl34b                    |        64000 |               4.11 |               0.24 |               4.02 |               0.25 |               1.56 |
| zephyr_7b_beta              |        32000 |               2.36 |               0.42 |               2.3  |               0.43 |               0.89 |

</details>




## Reference

- Getting the most out of your tokenizer for pre-training and domain adaptation
- Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca
- https://huggingface.co/spaces/Xenova/the-tokenizer-playground