update

.gitattributes

@@ -1,35 +1,37 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+stats/iter_vocab/*.jsonl filter=lfs diff=lfs merge=lfs -text
+stats/compression_rate/*.json filter=lfs diff=lfs merge=lfs -text

README.2.md

@@ -1,136 +0,0 @@
-
-https://arxiv.org/abs/2308.16692 SpeechTokenizer
-
-对于OpenAI的模型而言，英文的Token效率是中文的8-12倍，
-之前三百字中文以上时Turbo 3.5 16k就会出现逻辑颠倒问题，提示词换成英文后该问题没有出现过。
-
-## 词典构建
-
-bert词典
-gpt词典
-gpt-neox词典
-
-## encode
-
-
-## decode
-
-bert词典有个特殊字符 #
-
-gpt-neox词典呢？
-  - _开头表示空格或句首
-
-
-## 关于分词粒度
-
-
-## ss
-
-
-
-bert-chinese  vocab_size: 21128
-bert-en
-clue
-glm
-chatglm
-bloom
-
-
-## 最小词典
-
-mobilenet
-
-
-## ss
-
-
-## bert
-
-```
-[PAD]
-...
-[unused99]
-[UNK]
-[CLS]
-[SEP]
-[MASK]
-<S>
-<T>
-!
-...
-
-big
-##ut
-ftp
-carol
-##vi
-```
-
-
-## @@
-
-https://github.com/pytorch/fairseq/blob/master/tests/test_noising.py#L37
-
-```
-"he@@", "llo", "n@@", "ew", "y@@", "or@@", "k"
-```
-
-跟BERT类似，只不过BERT是词后缀，这里是词前缀。
-
-这种应该是 https://github.com/rsennrich/subword-nmt
-
-
-## GPT2
-
-词典见：https://huggingface.co/gpt2/raw/main/vocab.json
-
-
-```
-['What', "'s", 'Ġup', 'Ġwith', 'Ġthe', 'Ġtoken', 'izer', '?']
-```
-跟BERT不同，BERT用特殊符号表示 “连接”，GPT2用特殊符号表示 “空格”。
-
-详见 gpt2/README.md
-
-- 功能符号： `<|endoftext|>` 表示换行。tab？ 空格？
-- 很多数字独立编码，几乎上千个。
-
-- 类似的还有：moss
-
-
-### Ġ是什么
-
-It's a feature of byte-level BPE(an encoded space character). 
-Ġ 表示空格，有的版本用Ä代替Ġ。
-
-
-```sh
-What's up with the tokenizer?
-# BPE后
-['What', "'s", 'Ġup', 'Ġwith', 'Ġthe', 'Ġtoken', 'izer', '?']
-# 经过vocab.json编码后
-[ 2061,   338,  510,    351,    262,    11241,    7509,   30]
-# 经过dict.txt编码后（fairseq特有）
-[           其他数字                                         ]
-```
-<>
-疑问：up会加Ġ，为什么what不加Ġ，因为有个pre
-
-- https://github.com/pytorch/fairseq/issues/1716
-- https://github.com/huggingface/transformers/issues/1083
-
-
-## 空格、tab、换行 
-
-
-
-
-
-## reversible and lossless
-
-It's reversible and lossless, so you can convert tokens back into the original text
-
-
-## diff
-
-

README.md

@@ -1,15 +1,17 @@
----
-title: Tokenizer Arena
-emoji: 📚
-colorFrom: red
-colorTo: gray
-sdk: gradio
-sdk_version: 4.31.4
-app_file: app.py
-pinned: false
-datasets:
-  - cc100
----
-
-
-Please visit our GitHub repo for more information: https://github.com/xu-song/tokenizer-arena
+---
+title: Tokenizer Arena
+emoji: ⚔
+colorFrom: red
+colorTo: gray
+sdk: gradio
+sdk_version: 4.32.2
+app_file: app.py
+pinned: false
+datasets:
+  - cc100
+---
+
+
+
+
+Please visit our GitHub repo for more information: https://github.com/xu-song/tokenizer-arena

app.py

@@ -1,24 +1,39 @@
-import os
-from playground_app import demo as playground_tab
-from compression_app import demo as compression_tab
-from character_app import demo as character_tab
-from patcher.gr_interface import TabbedInterface
-from huggingface_hub import login
-
-auth_token = os.environ.get('HF_TOKEN', None)
-if auth_token:
-    login(token=auth_token)
-
-
-# 编码速度，解码速度，字符分类(zh、num等，支持正则)，支持的语言，。
-demo = TabbedInterface(
-    [playground_tab, compression_tab, character_tab],
-    [" ⚔️ Playground", "🏆 Compression Leaderboard", "📊 Character Statistics"],
-    title='<div align="center">Tokenizer Arena ⚔️</div>',
-    css="css/style.css"
-)
-
-demo.load(js=open("js/onload.js", "r", encoding="utf-8").read())
-
-if __name__ == "__main__":
-    demo.queue(max_size=1024, default_concurrency_limit=80).launch()
+"""Gradio app to showcase the LLM tokenization."""
+
+import os
+import gradio as gr
+from huggingface_hub import login
+from playground_app import demo as playground_tab
+from compression_app import demo as compression_tab
+from character_app import demo as character_tab
+
+auth_token = os.environ.get('HF_TOKEN', None)
+if auth_token:
+    login(token=auth_token)
+
+
+title = '<div align="center">Tokenizer Arena ⚔️</div>'
+interface_list = [playground_tab, compression_tab, character_tab]
+tab_names = [" ⚔️ Playground", "🏆 Compression Leaderboard", "📊 Character Statistics"]
+
+with gr.Blocks(css="css/style.css", js="js/onload.js") as demo:
+    gr.HTML(
+        f"<h1 style='text-align: center; margin-bottom: 1rem'>{title}</h1>"
+    )
+    with gr.Tabs():
+        for interface, tab_name in zip(interface_list, tab_names):
+            with gr.Tab(label=tab_name):
+                interface.render()
+
+    model_name = gr.Textbox(
+        placeholder="🔍 Add tokenizer from Hugging Face (e.g. Xenova/gpt-4o) and press ENTER...",
+        show_label=False,
+    )
+
+    model_name.submit()
+
+# demo.load(js=open("js/onload.js", "r", encoding="utf-8").read())
+
+if __name__ == "__main__":
+    demo.launch()
+    # demo.queue(max_size=1024, default_concurrency_limit=80).launch()

character_app.py

@@ -1,79 +1,83 @@
-import gradio as gr
-from character_util import get_character_table, default_columns
-
-all_columns = [
-    ("digit", "digit"),
-    ("space", "space"),
-    ("lang-chinese", 'zh'),
-    ("lang-korea", 'ko'),
-    ("lang-japanese", 'ja'),
-    # ("byte", "byte"),
-    # ("oov", "oov")
-]
-
-
-# columns = ["lang-zh", "lang-korea", "lang-japanese", "number", "space", "bytes", "oov"]
-
-abbr2name = {column[1]: column[0].split('-')[-1] for column in all_columns}
-
-
-def get_column_info(columns):
-    markdown = ""
-    for column in columns:
-        markdown += f"- `num({column})`: num of tokens containing {abbr2name[column]} characters\n" \
-                    f"- `len({column})`: `min,median,max` length of tokens containing {abbr2name[column]} characters\n"
-    return markdown
-
-
-with gr.Blocks() as demo:
-    gr.Markdown("## 🛠️ Setting")  # ⚙
-    with gr.Accordion("Please select the type of character you want to count.", open=True):
-        # file size 💽 🖴, tokens 🧮
-        with gr.Row():
-            with gr.Column():
-                columns = gr.Checkboxgroup(
-                    all_columns,
-                    value=default_columns,
-                    label="character type",
-                    # info=""
-                )
-                gr.Markdown(
-                    "To count other types of characters, you can modify [character_util.py]"
-                    "(https://huggingface.co/spaces/eson/tokenizer-arena/blob/main/character_util.py). "
-                )
-            column_info = gr.Markdown(
-                get_column_info(default_columns)
-            )
-
-        gr.Markdown("## 📊 Character Statistics")
-        search_bar = gr.Textbox(
-            placeholder="🔍 Search by tokenizer or organization (e.g., 'llama', 'openai') and press ENTER...",
-            show_label=False,
-            elem_id="search-bar",
-        )
-        compress_rate_table = gr.Dataframe(datatype="html", wrap=True)
-
-        search_bar.submit(
-            get_character_table,
-            inputs=[search_bar, columns],
-            outputs=compress_rate_table
-        )
-        columns.change(
-            get_character_table,
-            inputs=[search_bar, columns],
-            outputs=compress_rate_table
-        )
-        columns.change(
-            get_column_info,
-            inputs=[columns],
-            outputs=column_info
-        )
-
-        demo.load(
-            get_character_table,
-            inputs=[search_bar, columns],
-            outputs=compress_rate_table
-        )
-
-    if __name__ == "__main__":
-        demo.launch()
+"""
+##
+"""
+
+import gradio as gr
+from character_util import get_character_table, default_columns
+
+all_columns = [
+    ("digit", "digit"),
+    ("space", "space"),
+    ("lang-chinese", 'zh'),
+    ("lang-korea", 'ko'),
+    ("lang-japanese", 'ja'),
+    # ("byte", "byte"),
+    # ("oov", "oov")
+]
+
+
+# columns = ["lang-zh", "lang-korea", "lang-japanese", "number", "space", "bytes", "oov"]
+
+abbr2name = {column[1]: column[0].split('-')[-1] for column in all_columns}
+
+
+def get_column_info(columns):
+    markdown = ""
+    for column in columns:
+        markdown += f"- `num({column})`: num of tokens containing {abbr2name[column]} characters\n" \
+                    f"- `len({column})`: `min,median,max` length of tokens containing {abbr2name[column]} characters\n"
+    return markdown
+
+
+with gr.Blocks() as demo:
+    gr.Markdown("## 🛠️ Setting")  # ⚙
+    with gr.Accordion("Please select the type of character you want to count.", open=True):
+        # file size 💽 🖴, tokens 🧮
+        with gr.Row():
+            with gr.Column():
+                columns = gr.Checkboxgroup(
+                    all_columns,
+                    value=default_columns,
+                    label="character type",
+                    # info=""
+                )
+                gr.Markdown(
+                    "To count other types of characters, you can modify [lang_util.py]"
+                    "(https://huggingface.co/spaces/eson/tokenizer-arena/blob/main/utils/lang_util.py). "
+                )
+            column_info = gr.Markdown(
+                get_column_info(default_columns)
+            )
+
+        gr.Markdown("## 📊 Character Statistics")
+        search_bar = gr.Textbox(
+            placeholder="🔍 Search by tokenizer or organization (e.g., 'llama', 'openai') and press ENTER...",
+            show_label=False,
+            elem_id="search-bar",
+        )
+        compress_rate_table = gr.Dataframe(datatype="html", wrap=True)
+
+        search_bar.submit(
+            get_character_table,
+            inputs=[search_bar, columns],
+            outputs=compress_rate_table
+        )
+        columns.change(
+            get_character_table,
+            inputs=[search_bar, columns],
+            outputs=compress_rate_table
+        )
+        columns.change(
+            get_column_info,
+            inputs=[columns],
+            outputs=column_info
+        )
+
+        demo.load(
+            get_character_table,
+            inputs=[search_bar, columns],
+            outputs=compress_rate_table
+        )
+
+    if __name__ == "__main__":
+        demo.launch()

character_util.py

@@ -1,216 +1,216 @@
-"""
-TODO:
-1. add more language
-2. check space count of bert
-3. add token_impl
-4.
-"""
-import os
-import json
-import numpy as np
-import pandas as pd
-from collections import Counter, defaultdict
-from vocab import tokenizer_factory
-from typing import Optional, Union, Literal
-from utils.log_util import logger
-from utils.text_util import contains_digit, get_space_count
-from utils.lang_util import detect_language, language_ranges
-
-CURRENT_DIR = os.path.dirname(os.path.abspath(__file__))
-
-default_columns = ["digit", "zh"]
-
-def _to_unicode(text):
-    return ''.join(r'\u{:04X}'.format(ord(chr)) for chr in text)
-
-
-def _get_coding_length(tokenizer, vocab, filter=None):
-    """
-    oov character may be tokenized into more than one token.
-    """
-    all_length = []
-    for word in vocab:
-        if len(word) > 1:
-            continue
-        if filter is not None and filter(word):
-            continue
-        try:
-            tokens = tokenizer.encode(word)
-        except Exception as e:
-            print(e)
-
-        all_length.append(len(tokens))
-        # if len(tokens.ids) > 1:
-        # if len(tokens) > 3:
-        #     print(word, tokens)
-
-    dist_length = Counter(all_length)
-    mean_length = round(sum(all_length) / len(all_length), 2)
-    return dist_length, mean_length
-
-
-cache = {}
-
-
-def _dist(token_lens):
-    """
-    :param token_lens:
-    :return: min,median,max of token_lens
-    """
-    if not token_lens:
-        return "-"
-    return f"{min(token_lens)},{round(np.median(token_lens))},{max(token_lens)}"
-
-
-def iter_vocab(
-        tokenizer_name: str,
-        from_cache: bool = True,
-        cache_dir: str = "stats",
-) -> Union[pd.DataFrame, dict]:
-    """
-    :param tokenizer_name:
-    :param from_cache:
-    :param cache_dir:
-    :return:
-    """
-    tokenizer_config = tokenizer_factory.get_tokenizer_config(tokenizer_name)
-
-    cache_dir = os.path.join(CURRENT_DIR, cache_dir)
-    os.makedirs(cache_dir, exist_ok=True)
-
-    # load from cache
-    cache_path = os.path.join(cache_dir, "character_stats.json")
-    if not cache and os.path.exists(cache_path):
-        with open(cache_path, "r", encoding="utf-8") as f_tmp:
-            cache.update(json.load(f_tmp))
-    if from_cache and tokenizer_name in cache:
-        # logger.info(f"load {tokenizer_config.name_or_path} from cache")
-        return cache[tokenizer_name]
-
-    tokenizer = tokenizer_factory.get_tokenizer(tokenizer_name)
-
-    tokens_by_lang = {lang[1]: [] for lang in language_ranges.keys()}
-    digit_tokens = []
-    space_tokens = []
-    byte_tokens = []
-
-    buffer = []
-    for token_id in range(tokenizer.vocab_size):
-        # for token_id in tokenizer.get_vocab():
-        # for token_id in range(len(tokenizer)):
-        decode_str = tokenizer.decode([token_id], skip_special_tokens=False)
-        token = tokenizer.convert_ids_to_tokens([token_id], skip_special_tokens=False)[0]
-        tags = []
-        if token is None:  # 有些词典有空的id（不连续）
-            continue
-        if isinstance(token, bytes):
-            token = token.decode("utf-8", errors="ignore")
-
-        if hasattr(tokenizer, "sp_model"):  # 基于 sentencepiece 包
-            if tokenizer.sp_model.is_byte(token_id):
-                tags.append("is_byte")
-                byte_tokens.append(token)
-
-        language_tags = detect_language(decode_str)
-        for language in language_tags:
-            tokens_by_lang[language[1]].append(decode_str)
-
-        if contains_digit(decode_str):
-            tags.append("digit")
-            digit_tokens.append(decode_str)
-
-        space_count = get_space_count(decode_str)
-        if space_count > 0:
-            space_tokens.append(decode_str)
-
-        buffer.append(json.dumps(
-            {
-                "id": token_id,
-                "token": token,
-                "token_decode": decode_str,
-                "token_dumps": json.dumps(token),
-                "token_unicode": _to_unicode(token),
-                "token_len": len(decode_str),
-            },
-            ensure_ascii=False) + "\n")
-
-    result = {
-        "tokenizer": tokenizer_factory.get_name_with_hyperlink(tokenizer_name),
-        "organization": tokenizer_config.org,
-        # "impl": str(tokenizer.__class__),
-        # "vocab_size-": tokenizer.vocab_size,  # vocab_size_without_added_token
-        "vocab_size": len(tokenizer),
-
-        # "中文汉字编码长度均值": mean_length,   # 不用统计，因为字典包含中文字符多，一般就意味着 中文汉字编码长度短。
-        # "中文汉字编码长度分布": json.dumps(dist_length),
-
-        "num(digit)": len(digit_tokens),
-        "len(digit)": _dist([len(token) for token in digit_tokens]),
-        "num(space)": len(space_tokens),
-        "len(space)": _dist([len(token) for token in space_tokens]),
-
-        # "num(byte)": len(byte_tokens)
-    }
-
-    for lang, tokens in tokens_by_lang.items():
-        result[f"num({lang})"] = len(tokens)
-        result["len(" + lang + ")"] = _dist([len(token) for token in tokens])
-
-    out_path = os.path.join(cache_dir, f"iter_vocab/{tokenizer_name.replace('/', '_')}.vocab.jsonl")
-    with open(out_path, "w", encoding="utf-8") as f_out:
-        for line in buffer:
-            f_out.write(line)
-    len_before = len(cache)
-    cache[tokenizer_name] = result
-    len_after = len(cache)
-    logger.info(f"saving {tokenizer_name} to memory and file cache: {len_before}->{len_after}")
-    with open(cache_path, "w", encoding="utf-8") as f_out:
-        f_out.write(json.dumps(cache, ensure_ascii=False, indent=2))
-    return result
-
-
-def to_dataframe(stats, columns):
-    table = []
-    for stat in stats.values():
-        filtered_stat = {}
-        for k, v in stat.items():
-            if not k.startswith("num") and not k.startswith("len"):
-                filtered_stat[k] = v
-            if any(column in k for column in columns):
-                k = k.replace("ja-kana", "kana")
-                filtered_stat[k] = v
-        table.append(filtered_stat)
-    df = pd.DataFrame(table)
-    return df
-
-
-def get_character_table(
-        tokenizer_filter: Optional[str] = None,
-        columns: Optional[list] = None,
-        return_type: Optional[Literal["dict", "dataframe"]] = "dataframe"
-) -> Union[pd.DataFrame, dict]:
-    """
-    """
-    logger.info(f"columns: {columns}, tokenizer_filter: {tokenizer_filter}")
-    stats = {}
-    if columns is None:
-        columns = default_columns
-    if tokenizer_filter is not None:
-        tokenizer_names = [tokenizer_config.name_or_path for tokenizer_config in tokenizer_factory.all_tokenizer_configs
-                           if tokenizer_filter.lower() in tokenizer_config.name_or_path.lower()]
-    else:
-        tokenizer_names = tokenizer_factory.all_tokenizer_names
-
-    for tokenizer_name in tokenizer_names:
-        stat = iter_vocab(tokenizer_name)
-        stats[tokenizer_name] = stat
-
-    if return_type == "dataframe":
-        stats = to_dataframe(stats, columns)
-    return stats
-
-
-if __name__ == "__main__":
-    # aa = get_character_table(tokenizer_filter="baichuan")
-    df = get_character_table()
-    logger.info(f"\n{df.to_markdown(index=False)}")
+"""
+TODO:
+1. add more language
+2. check space count of bert
+3. add token_impl
+4.
+"""
+import os
+import json
+import numpy as np
+import pandas as pd
+from collections import Counter, defaultdict
+from vocab import tokenizer_factory
+from typing import Optional, Union, Literal
+from utils.log_util import logger
+from utils.text_util import contains_digit, get_space_count
+from utils.lang_util import detect_language_by_unicode, language_ranges
+
+CURRENT_DIR = os.path.dirname(os.path.abspath(__file__))
+
+default_columns = ["digit", "zh"]
+
+def _to_unicode(text):
+    return ''.join(r'\u{:04X}'.format(ord(chr)) for chr in text)
+
+
+def _get_coding_length(tokenizer, vocab, filter=None):
+    """
+    oov character may be tokenized into more than one token.
+    """
+    all_length = []
+    for word in vocab:
+        if len(word) > 1:
+            continue
+        if filter is not None and filter(word):
+            continue
+        try:
+            tokens = tokenizer.encode(word)
+        except Exception as e:
+            print(e)
+
+        all_length.append(len(tokens))
+        # if len(tokens.ids) > 1:
+        # if len(tokens) > 3:
+        #     print(word, tokens)
+
+    dist_length = Counter(all_length)
+    mean_length = round(sum(all_length) / len(all_length), 2)
+    return dist_length, mean_length
+
+
+cache = {}
+
+
+def _dist(token_lens):
+    """
+    :param token_lens:
+    :return: min,median,max of token_lens
+    """
+    if not token_lens:
+        return "-"
+    return f"{min(token_lens)},{round(np.median(token_lens))},{max(token_lens)}"
+
+
+def iter_vocab(
+        tokenizer_name: str,
+        from_cache: bool = True,
+        cache_dir: str = "stats",
+) -> Union[pd.DataFrame, dict]:
+    """
+    :param tokenizer_name:
+    :param from_cache:
+    :param cache_dir:
+    :return:
+    """
+    tokenizer_config = tokenizer_factory.get_tokenizer_config(tokenizer_name)
+
+    cache_dir = os.path.join(CURRENT_DIR, cache_dir)
+    os.makedirs(cache_dir, exist_ok=True)
+
+    # load from cache
+    cache_path = os.path.join(cache_dir, "character_stats.json")
+    if not cache and os.path.exists(cache_path):
+        with open(cache_path, "r", encoding="utf-8") as f_tmp:
+            cache.update(json.load(f_tmp))
+    if from_cache and tokenizer_name in cache:
+        # logger.info(f"load {tokenizer_config.name_or_path} from cache")
+        return cache[tokenizer_name]
+
+    tokenizer = tokenizer_factory.get_tokenizer(tokenizer_name)
+
+    tokens_by_lang = {lang[1]: [] for lang in language_ranges.keys()}
+    digit_tokens = []
+    space_tokens = []
+    byte_tokens = []
+
+    buffer = []
+    for token_id in range(tokenizer.vocab_size):
+        # for token_id in tokenizer.get_vocab():
+        # for token_id in range(len(tokenizer)):
+        decode_str = tokenizer.decode([token_id], skip_special_tokens=False)
+        token = tokenizer.convert_ids_to_tokens([token_id], skip_special_tokens=False)[0]
+        tags = []
+        if token is None:  # 有些词典有空的id（不连续）
+            continue
+        if isinstance(token, bytes):
+            token = token.decode("utf-8", errors="ignore")
+
+        if hasattr(tokenizer, "sp_model"):  # 基于 sentencepiece 包
+            if tokenizer.sp_model.is_byte(token_id):
+                tags.append("is_byte")
+                byte_tokens.append(token)
+
+        language_tags = detect_language_by_unicode(decode_str)
+        for language in language_tags:
+            tokens_by_lang[language[1]].append(decode_str)
+
+        if contains_digit(decode_str):
+            tags.append("digit")
+            digit_tokens.append(decode_str)
+
+        space_count = get_space_count(decode_str)
+        if space_count > 0:
+            space_tokens.append(decode_str)
+
+        buffer.append(json.dumps(
+            {
+                "id": token_id,
+                "token": token,
+                "token_decode": decode_str,
+                "token_dumps": json.dumps(token),
+                "token_unicode": _to_unicode(token),
+                "token_len": len(decode_str),
+            },
+            ensure_ascii=False) + "\n")
+
+    result = {
+        "tokenizer": tokenizer_factory.get_name_with_hyperlink(tokenizer_name),
+        "organization": tokenizer_config.org,
+        # "impl": str(tokenizer.__class__),
+        # "vocab_size-": tokenizer.vocab_size,  # vocab_size_without_added_token
+        "vocab_size": len(tokenizer),
+
+        # "中文汉字编码长度均值": mean_length,   # 不用统计，因为字典包含中文字符多，一般就意味着 中文汉字编码长度短。
+        # "中文汉字编码长度分布": json.dumps(dist_length),
+
+        "num(digit)": len(digit_tokens),
+        "len(digit)": _dist([len(token) for token in digit_tokens]),
+        "num(space)": len(space_tokens),
+        "len(space)": _dist([len(token) for token in space_tokens]),
+
+        # "num(byte)": len(byte_tokens)
+    }
+
+    for lang, tokens in tokens_by_lang.items():
+        result[f"num({lang})"] = len(tokens)
+        result["len(" + lang + ")"] = _dist([len(token) for token in tokens])
+
+    out_path = os.path.join(cache_dir, f"iter_vocab/{tokenizer_name.replace('/', '_')}.vocab.jsonl")
+    with open(out_path, "w", encoding="utf-8") as f_out:
+        for line in buffer:
+            f_out.write(line)
+    len_before = len(cache)
+    cache[tokenizer_name] = result
+    len_after = len(cache)
+    logger.info(f"saving {tokenizer_name} to memory and file cache: {len_before}->{len_after}")
+    with open(cache_path, "w", encoding="utf-8") as f_out:
+        f_out.write(json.dumps(cache, ensure_ascii=False, indent=2))
+    return result
+
+
+def to_dataframe(stats, columns):
+    table = []
+    for stat in stats.values():
+        filtered_stat = {}
+        for k, v in stat.items():
+            if not k.startswith("num") and not k.startswith("len"):
+                filtered_stat[k] = v
+            if any(column in k for column in columns):
+                k = k.replace("ja-kana", "kana")
+                filtered_stat[k] = v
+        table.append(filtered_stat)
+    df = pd.DataFrame(table)
+    return df
+
+
+def get_character_table(
+        tokenizer_filter: Optional[str] = None,
+        columns: Optional[list] = None,
+        return_type: Optional[Literal["dict", "dataframe"]] = "dataframe"
+) -> Union[pd.DataFrame, dict]:
+    """
+    """
+    logger.info(f"columns: {columns}, tokenizer_filter: {tokenizer_filter}")
+    stats = {}
+    if columns is None:
+        columns = default_columns
+    if tokenizer_filter is not None:
+        tokenizer_names = [tokenizer_config.name_or_path for tokenizer_config in tokenizer_factory.all_tokenizer_configs
+                           if tokenizer_filter.lower() in tokenizer_config.name_or_path.lower()]
+    else:
+        tokenizer_names = tokenizer_factory.all_tokenizer_names
+
+    for tokenizer_name in tokenizer_names:
+        stat = iter_vocab(tokenizer_name)
+        stats[tokenizer_name] = stat
+
+    if return_type == "dataframe":
+        stats = to_dataframe(stats, columns)
+    return stats
+
+
+if __name__ == "__main__":
+    # aa = get_character_table(tokenizer_filter="baichuan")
+    df = get_character_table()
+    logger.info(f"\n{df.to_markdown(index=False)}")

compression_app.py

@@ -1,130 +1,187 @@
-"""
-TODO:
-- 统计 tokenizer_impl
-- 统计 OOV
-- 统计 reversal
-- 增加 math，code
-"""
-
-import gradio as gr
-from compression_util import get_compression_leaderboard, common_corpuses
-
-
-with gr.Blocks() as demo:
-    # gr.Markdown("## Convertor")
-    # with gr.Accordion("Convertor", open=False):
-    #     gr.Markdown("Tokenize {} corpus")
-    #     with gr.Row(elem_classes="no-border"):
-    #         gr.Button("File Size", min_width=50)
-    #         file_size = gr.Textbox(
-    #             show_label=False,
-    #             min_width=50,
-    #             # elem_classes="textbox-as-text"
-    #         )
-    #         gr.Dropdown(
-    #             choices=['MB', 'GB', 'TB'],
-    #             show_label=False,
-    #             min_width=15,
-    #             # elem_classes="textbox-as-text"
-    #         )
-    #         # gr.Markdown('<h2 align="center">≈</h2>')
-    #         # gr.HTML('<h2 style="margin: auto;">≈</h2>')
-    #         gr.Button(
-    #             "≈",
-    #             min_width=10,
-    #             elem_classes="button-white h2-font"
-    #
-    #         )
-    #
-    #         gr.Button(
-    #             "Tokens",
-    #             min_width=50
-    #         )
-    #         gr.Textbox(
-    #             show_label=False,
-    #             min_width=50
-    #         )
-    #         gr.Dropdown(
-    #             ['million', 'billion', 'trillion'],
-    #             show_label=False,
-    #             min_width=15,
-    #             elem_classes="button-white"
-    #         )
-
-    gr.Markdown("## 🛠️ Setting")  # ⚙
-    with gr.Accordion("Please select the corpus and measure of compression rate.", open=True):
-        # file size 💽 🖴, tokens 🧮
-        # Total amount of disk used
-        with gr.Row():
-            with gr.Column():
-                compress_rate_corpus = gr.Dropdown(
-                    common_corpuses,  # , "code"
-                    value=["cc100/en", "cc100/zh-Hans", "cc100/fr", "cc100/es"],
-                    label="corpus",
-                    multiselect=True
-                    # info=""
-                )
-
-                # unit of file_size: gigabyte terabyte
-                # unit of token_num: million billion trillion
-                # The most common units of measurement include length (meter, inch, foot), weight (gram, kilogram, pound), volume (liter, gallon, milliliter), time (second, minute, hour)
-                compress_rate_unit = gr.Radio(
-                    ["b_tokens/g_bytes", "t_tokens/t_bytes"],
-                    value="b_tokens/g_bytes",
-                    label="measure",  # evaluation metric
-                )
-
-            gr.Markdown(
-                # "Note:\n\n"
-                "- `corpus`: tokenization is performed on the selected subsets of [cc100](https://huggingface.co/datasets/cc100) corpus.\n"
-                "- `b_tokens/g_bytes` measures how many billion tokens per gigabytes corpus.\n"
-                "- `t_tokens/t_bytes` measures how many trillion tokens per terabytes corpus.\n"
-                # "- `g_bytes/b_tokens` measures how many gigabytes corpus per billion tokens.\n"
-                # "- `t_bytes/t_tokens` measures how many terabytes corpus per trillion tokens.\n"
-                "- `char/token` measures how many chars per token on the tokenized corpus.\n"
-                "- `oov_ratio`: out-of-vocabulary ratio on the selected corpus, 👉 get [oov charset](https://huggingface.co/spaces/eson/tokenizer-arena/raw/main/stats/compression_rate.json)\n\n"
-                "You can reproduce this procedure with [compression_util.py](https://huggingface.co/spaces/eson/tokenizer-arena/blob/main/compression_util.py)."
-            )
-
-    gr.Markdown("## 🏆 Compression Rate Leaderboard")
-    search_bar = gr.Textbox(
-        placeholder="🔍 Search by tokenizer or organization (e.g., 'llama', 'openai') and press ENTER...",
-        show_label=False,
-        elem_id="search-bar",
-    )
-    compress_rate_table = gr.Dataframe(datatype="html")
-
-    # func call
-    compress_rate_corpus.change(
-        get_compression_leaderboard,
-        inputs=[compress_rate_corpus, compress_rate_unit, search_bar],
-        outputs=compress_rate_table
-    )
-    compress_rate_unit.change(
-        get_compression_leaderboard,
-        inputs=[compress_rate_corpus, compress_rate_unit, search_bar],
-        outputs=compress_rate_table
-    )
-    # file_size.change(
-    #     get_all_compress_rate,
-    #     outputs=compress_rate_table
-    # )
-
-    search_bar.submit(
-        get_compression_leaderboard,
-        inputs=[
-            compress_rate_corpus,
-            compress_rate_unit,
-            search_bar,
-        ],
-        outputs=compress_rate_table
-    )
-
-    demo.load(
-        get_compression_leaderboard,
-        inputs=[compress_rate_corpus, compress_rate_unit],
-        outputs=compress_rate_table
-    )
-
-if __name__ == "__main__":
-    demo.launch()
+"""
+TODO:
+- 统计 tokenizer_impl
+- 统计 OOV
+- 统计 reversal
+- 增加 math，code
+
+
+
+## balance
+
+- 高压缩率 VS vocab_size:
+    - 高压缩率，就意味着，编码后的token数少，那么 token长度 就会长，--> vocab_size 就会太大
+- 高压缩率 VS 无损
+    - s
+- OOV
+    - OOV 多，那么生成的 UNK 可能多(一个char 一个UNK) --> token 数目多 -> 压缩率低
+    - OOV 多，那么生成的 UNK 可能少（） --> token 数目多 -> 压缩率低
+
+"""
+
+import gradio as gr
+from compression_util import get_compression_leaderboard, common_corpuses
+
+
+# From the perspective of compression
+# exactly reconstructed from compressed tokens
+docs = """## 📖 What is a good tokenizer?
+
+From a compression perspective, a good tokenizer should be lossless, and keep high compression rate (less tokens).
+The encoding and decoding process can be formulated as
+```python
+    token_ids = tokenizer.encode(input_text)    # compressed tokens
+    decoded_text = tokenizer.decode(token_ids)  # reconstructed text
+```
+
+- **Lossless** <br>
+Lossless tokenization preserves the exact original text, i.e. `decoded_text = input_text`.
+
+  - Most lossy tokenizers get many out-of-vocabulary tokens. 👉 Check the [oov of bert-base-uncased](https://huggingface.co/spaces/eson/tokenizer-arena/raw/main/stats/google-bert.bert-base-casedcc100.zh-Hans.diff.json).
+  - Some other tokenizers have no oov, but still be lossy due to text normalization. For example qwen performs [unicode normalization](https://github.com/huggingface/transformers/blob/v4.42.3/src/transformers/models/qwen2/tokenization_qwen2.py#L338),
+    which may bring some [slight difference](https://huggingface.co/spaces/eson/tokenizer-arena/raw/main/stats/compression_rate/Qwen.Qwen1.5-1.8B%20@%20cc100.ja.diff.jsonn) to the reconstructed text. 
+
+- **Compression Rate** <br>
+There are mainly two types of metric to represent the `input_text`:
+  - `byte-level`: the number of bytes in the given text
+  - `char-level`: the number of characters in the given text.
+
+To evaluate compression rate, simple metrics can be "how many bytes per token" or "how many chars per token". <br>
+In this leaderboard, we adopt more frequently used metric: "how many billion tokens per gigabytes corpus" and "how many chars 
+per token", i.e. `b_tokens/g_bytes` and `char/token`.
+
+
+💬 [Discussions is Welcome](https://huggingface.co/spaces/eson/tokenizer-arena/discussions)
+"""
+
+
+
+# theme = gr.themes.Monochrome()
+theme = gr.themes.Default()
+# theme.set(accordion_text_weight=600)  # 暂不支持
+with gr.Blocks(theme=theme) as demo:
+    # gr.Markdown("## Convertor")
+    # with gr.Accordion("Convertor", open=False):
+    #     gr.Markdown("Tokenize {} corpus")
+    #     with gr.Row(elem_classes="no-border"):
+    #         gr.Button("File Size", min_width=50)
+    #         file_size = gr.Textbox(
+    #             show_label=False,
+    #             min_width=50,
+    #             # elem_classes="textbox-as-text"
+    #         )
+    #         gr.Dropdown(
+    #             choices=['MB', 'GB', 'TB'],
+    #             show_label=False,
+    #             min_width=15,
+    #             # elem_classes="textbox-as-text"
+    #         )
+    #         # gr.Markdown('<h2 align="center">≈</h2>')
+    #         # gr.HTML('<h2 style="margin: auto;">≈</h2>')
+    #         gr.Button(
+    #             "≈",
+    #             min_width=10,
+    #             elem_classes="button-white h2-font"
+    #
+    #         )
+    #
+    #         gr.Button(
+    #             "Tokens",
+    #             min_width=50
+    #         )
+    #         gr.Textbox(
+    #             show_label=False,
+    #             min_width=50
+    #         )
+    #         gr.Dropdown(
+    #             ['million', 'billion', 'trillion'],
+    #             show_label=False,
+    #             min_width=15,
+    #             elem_classes="button-white"
+    #         )
+
+
+
+    gr.Markdown(docs)
+    gr.Markdown("## 🛠️ Setting")  # ⚙
+    gr.Markdown("We perform tokenization on different corpus, and calculate the compression rate."
+                "")
+    with gr.Accordion("Please select the corpus and measure of compression rate.", open=True):
+        # file size 💽 🖴, tokens 🧮
+        # Total amount of disk used
+        with gr.Row():
+            with gr.Column():
+                compress_rate_corpus = gr.Dropdown(
+                    common_corpuses,  # , "code"
+                    value=["cc100/en", "cc100/zh-Hans", "cc100/fr", "cc100/es"],
+                    label="corpus",
+                    multiselect=True
+                    # info=""
+                )
+
+                # unit of file_size: gigabyte terabyte
+                # unit of token_num: million billion trillion
+                # The most common units of measurement include length (meter, inch, foot), weight (gram, kilogram, pound), volume (liter, gallon, milliliter), time (second, minute, hour)
+                compress_rate_unit = gr.Radio(
+                    ["b_tokens/g_bytes", "t_tokens/t_bytes"],
+                    value="b_tokens/g_bytes",
+                    label="measure",  # evaluation metric
+                )
+
+            gr.Markdown(
+                # "Note:\n\n  explanation"
+                # "Supported languages are (20): arabic (ar), bulgarian (bg), german (de), modern greek (el), english (en), spanish (es), french (fr), hindi (hi), italian (it), japanese (ja), dutch (nl), polish (pl), portuguese (pt), russian (ru), swahili (sw), thai (th), turkish (tr), urdu (ur), vietnamese (vi), and chinese (zh)."
+                # " arabic (ar), english (en), spanish (es), french (fr), italian (it), japanese (ja), portuguese (pt), russian (ru), and chinese (zh)."
+                "- `corpus`: tokenization is performed on the selected subsets of [cc100](https://huggingface.co/datasets/statmt/cc100) corpus.\n"
+                "- measure\n"
+                "  - `b_tokens/g_bytes` measures how many billion tokens per gigabytes corpus.\n"
+                "  - `t_tokens/t_bytes` measures how many trillion tokens per terabytes corpus.\n"
+                # "- `g_bytes/b_tokens` measures how many gigabytes corpus per billion tokens.\n"
+                # "- `t_bytes/t_tokens` measures how many terabytes corpus per trillion tokens.\n"
+                "  - `char/token` measures how many chars per token on the tokenized corpus.\n"
+                "  - `oov_ratio`: out-of-vocabulary ratio on the selected corpus, 👉 get [oov charset](https://huggingface.co/spaces/eson/tokenizer-arena/raw/main/stats/compression_rate.json)\n\n"
+                "You can reproduce this procedure with [compression_util.py](https://huggingface.co/spaces/eson/tokenizer-arena/blob/main/compression_util.py)."
+            )
+
+    gr.Markdown("## 🏆 Compression Rate Leaderboard")
+    search_bar = gr.Textbox(
+        placeholder="🔍 Search by tokenizer or organization (e.g., 'llama', 'openai') and press ENTER...",
+        show_label=False,
+        elem_id="search-bar",
+    )
+    compress_rate_table = gr.Dataframe(datatype="html")
+
+    # func call
+    compress_rate_corpus.change(
+        get_compression_leaderboard,
+        inputs=[compress_rate_corpus, compress_rate_unit, search_bar],
+        outputs=compress_rate_table
+    )
+    compress_rate_unit.change(
+        get_compression_leaderboard,
+        inputs=[compress_rate_corpus, compress_rate_unit, search_bar],
+        outputs=compress_rate_table
+    )
+    # file_size.change(
+    #     get_all_compress_rate,
+    #     outputs=compress_rate_table
+    # )
+
+    search_bar.submit(
+        get_compression_leaderboard,
+        inputs=[
+            compress_rate_corpus,
+            compress_rate_unit,
+            search_bar,
+        ],
+        outputs=compress_rate_table
+    )
+
+    demo.load(
+        get_compression_leaderboard,
+        inputs=[compress_rate_corpus, compress_rate_unit],
+        outputs=compress_rate_table
+    )
+
+if __name__ == "__main__":
+    demo.launch()

compression_util.py

@@ -1,302 +1,320 @@
-"""
-
-## TODO
-code:
-math：
-whitespace:
-
-"""
-
-import json
-import os
-import sys
-import pandas as pd
-from datasets import load_dataset
-from utils.log_util import logger
-from vocab import tokenizer_factory, TokenizerConfig
-from typing import List, Optional, Union, Literal
-
-CURRENT_DIR = os.path.dirname(os.path.abspath(__file__))
-
-common_units = ["g_bytes/b_tokens", "b_tokens/g_bytes", "t_bytes/t_tokens", "t_tokens/t_bytes", "n_chars/n_tokens", ]
-
-common_corpuses = sorted(["cc100/en", "cc100/zh-Hans", "cc100/es", "cc100/fr", "cc100/de", "cc100/ko",
-                          "cc100/fa", "cc100/ar", "cc100/ja"])
-
-VALID_CODES_CC100 = [
-    "am", "ar", "as", "az", "be", "bg", "bn", "bn_rom", "br", "bs", "ca", "cs", "cy", "da", "de",
-    "el", "en", "eo", "es", "et", "eu", "fa", "ff", "fi", "fr", "fy", "ga", "gd", "gl", "gn", "gu",
-    "ha", "he", "hi", "hi_rom", "hr", "ht", "hu", "hy", "id", "ig", "is", "it", "ja", "jv", "ka",
-    "kk", "km", "kn", "ko", "ku", "ky", "la", "lg", "li", "ln", "lo", "lt", "lv", "mg", "mk", "ml",
-    "mn", "mr", "ms", "my", "my_zaw", "ne", "nl", "no", "ns", "om", "or", "pa", "pl", "ps", "pt",
-    "qu", "rm", "ro", "ru", "sa", "si", "sc", "sd", "sk", "sl", "so", "sq", "sr", "ss", "su", "sv",
-    "sw", "ta", "ta_rom", "te", "te_rom", "th", "tl", "tn", "tr", "ug", "uk", "ur", "ur_rom", "uz",
-    "vi", "wo", "xh", "yi", "yo", "zh-Hans", "zh-Hant", "zu",
-]
-
-
-# code: https://huggingface.co/datasets/codeparrot/github-code-clean  python java c sql html
-# math:
-
-def get_n_bytes_of_string(string_text):
-    n_bytes = len(string_text.encode("utf-8"))
-    return n_bytes
-
-
-def unit_convertor(stat, unit):
-    n_tokens = stat["_n_tokens"]
-    n_chars = stat["_n_chars"]
-    n_bytes = stat["_n_bytes"]
-
-    if n_tokens is None:
-        return None
-
-    n_tokens_in_billion = n_tokens / (1000 * 1000 * 1000)
-    n_tokens_in_trillion = n_tokens / (1000 * 1000 * 1000 * 1000)
-    n_bytes_in_mb = n_bytes / (1024 * 1024)
-    n_bytes_in_gb = n_bytes_in_mb / 1024
-    n_bytes_in_tb = n_bytes_in_gb / 1024
-    # n_chars_in_billion = n_chars / (1000 * 1000 * 1000)
-
-    if unit == "n_tokens/n_bytes":
-        value = n_tokens / n_bytes
-    elif unit in ["char/token", "chars_per_token"]:  # 重要：平均一个token包含多少个字符。
-        value = n_chars / n_tokens
-    elif unit in ["token/char", "tokens_per_char"]:  # 一个中文汉字需要几个token？
-        value = n_tokens / n_chars
-    elif unit == "g_bytes/b_tokens":
-        value = n_bytes_in_gb / n_tokens_in_billion
-    elif unit == "b_tokens/g_bytes":
-        value = n_tokens_in_billion / n_bytes_in_gb
-    elif unit == "t_bytes/t_tokens":  # 重要：
-        value = n_bytes_in_tb / n_tokens_in_trillion
-    elif unit == "t_tokens/t_bytes":
-        value = n_tokens_in_trillion / n_bytes_in_tb
-    else:
-        raise "measure not support"
-    return round(value, 3)
-
-
-def _merge_stats_by_corpus(stats_by_corpus, oov_threshold=0.3):
-    """
-    """
-    all_stats = list(stats_by_corpus.values())
-    assert len(set([stats["tokenizer"] for stats in all_stats])) == 1
-    reversible = all(stat['reversible'] for stat in all_stats)
-    is_support = all(stat['oov_ratio'] < oov_threshold for stat in all_stats)
-
-    merged_stats = {
-        "tokenizer": all_stats[0]["tokenizer"],
-        "organization": all_stats[0]["organization"],
-        "vocab_size": all_stats[0]["vocab_size"],
-        "_n_bytes": 0,
-        "_n_tokens": 0 if is_support else None,
-        "_n_chars": 0,
-        "_n_oov_chars": 0,
-        "reversible": True,
-    }
-    for stats in all_stats:
-        merged_stats["_n_bytes"] += stats["_n_bytes"]
-        merged_stats["_n_chars"] += stats["_n_chars"]
-        if is_support:  # The number of tokens cannot be accurately counted, when there are too many UNKs.
-            merged_stats["_n_tokens"] += stats["_n_tokens"]
-        merged_stats["_n_oov_chars"] += stats["_n_oov_chars"]
-        merged_stats["reversible"] &= stats['reversible']
-
-    merged_stats.update({
-        "oov_ratio": float("%.4g" % (stats["_n_oov_chars"] / stats["_n_chars"])),
-        "reversible": reversible
-    })
-    return merged_stats
-
-
-def to_dataframe(stats, units=None):
-    if units is None:
-        units = common_units
-    elif not isinstance(units, list):
-        units = [units]
-    table = []
-
-    for stat in stats.values():
-        columns = {k: v for k, v in stat.items() if not k.startswith("_")}
-        for unit in units:
-            if unit not in stat:
-                columns[unit] = unit_convertor(stat, unit)
-            else:
-                logger.error(f"unit {unit} not support")
-        table.append(columns)
-    df = pd.DataFrame(table)
-    return df
-
-
-cache = {}
-
-
-def tokenize_corpus(
-        tokenizer_name: str,
-        corpuses: List[str],
-        cache_dir: str = "stats"
-) -> dict:
-    """
-    :param tokenizer_name:
-    :param corpuses:
-    :param cache_dir:
-    :return:
-    """
-
-    def _char_based_oov(src_text, decode_text):
-        oov_chars = []
-        for char in src_text:
-            if char not in decode_text:
-                oov_chars.append(char)
-
-        n_oov_chars = len(oov_chars)
-        oov_charset = list(dict.fromkeys(oov_chars))
-        return n_oov_chars, oov_charset
-
-    def _tokenize(tokenizer, datasets, detail_path=None):
-        """
-        :param tokenizer:
-        :param datasets:
-        :param detail_path:
-        :return:
-        """
-        n_bytes = 0
-        n_tokens = 0
-        n_chars = 0
-        n_oov_chars = 0
-        diff_details = []
-        oov_charset = set()
-        unk_token_id = None
-        if hasattr(tokenizer, "unk_token"):
-            unk_token_id = tokenizer.unk_token_id
-        for dataset in datasets:
-            for item in dataset:
-                text = item["text"]
-                n_bytes += get_n_bytes_of_string(text)
-                n_chars += len(text)
-                ids = tokenizer.encode(text, add_special_tokens=False)
-
-                # detect oov
-                decode_text = tokenizer.decode(ids)
-                decode_text_without_unk = tokenizer.decode([token_id for token_id in ids if token_id != unk_token_id])
-                if decode_text != text:
-                    _n_oov_chars, _oov_charset = _char_based_oov(text, decode_text_without_unk)
-                    diff_details.append(
-                        {
-                            "text": text,
-                            "decode_text": decode_text,
-                            "decode_text_without_unk": decode_text_without_unk,
-                            "n_oov_chars": _n_oov_chars,
-                            'oov_ratio': _n_oov_chars / len(text),
-                            'oov_charset': json.dumps(_oov_charset, ensure_ascii=False),
-                        }
-                    )
-                    n_oov_chars += _n_oov_chars
-                    oov_charset.update(_oov_charset)
-                n_tokens += len(ids)
-        stat = {
-            "_n_bytes": n_bytes,
-            "_n_tokens": n_tokens,
-            "_n_chars": n_chars,
-            "_n_oov_chars": n_oov_chars,
-            "oov_ratio": n_oov_chars / n_chars,
-            '_oov_charset': json.dumps(list(oov_charset), ensure_ascii=False),
-            "reversible": len(diff_details) == 0
-        }
-
-        if detail_path and diff_details:
-            logger.info(f"saving tokenization detail to '{detail_path}'")
-            with open(detail_path, "w", encoding="utf-8") as f:
-                f.write(json.dumps(diff_details, ensure_ascii=False, indent=2))
-                # print(f"{tokenizer_config.name_or_path}, {infer_tokenizer_type(tokenizer_config)}\n"
-                #       f"reversible: false; unk_token: {get_unk(tokenizer_config)},"
-                #       f" unk_ratio: {unk_count / len(encoding):.4f}; oov: []")
-                # for diff_detail in diff_details:
-                #         # print(f"text[{i}]     = {str(bytes(text[i:], 'utf-8'))}\n"
-                #         #       f"decoding[{i}] = {str(bytes(decoding[i:], 'utf-8'))}")
-                #         f.write(f"text= {json.dumps(text[i:], ensure_ascii=False)}, \n"
-                #               f"decoding[{i}] = {json.dumps(decoding[i:], ensure_ascii=False)}")
-        return stat
-
-    # load from cache
-    cache_id = f"{tokenizer_name} @ {'.'.join(corpuses)}"
-    cache_path = os.path.join(cache_dir, "compression_rate.json")
-    if not cache and os.path.exists(cache_path):
-        with open(cache_path, "r", encoding="utf-8") as f_tmp:
-            cache.update(json.load(f_tmp))
-    if cache_id in cache:
-        # logger.info(f"loading {cache_id} from in-memory cache")
-        return cache[cache_id]
-
-    # tokenize corpus
-    tokenizer = tokenizer_factory.get_tokenizer(tokenizer_name)
-    datasets = [load_dataset("eson/cc100-samples", corpus.replace("cc100/", ""), split="train") for corpus in corpuses]
-
-    stat = {
-        "tokenizer": tokenizer_factory.get_name_with_hyperlink(tokenizer_name),
-        "organization": tokenizer_factory.get_tokenizer_config(tokenizer_name).org,
-        "vocab_size": len(tokenizer),
-    }
-    tokenize_detail_dir = os.path.join(cache_dir, "compression_rate")
-    os.makedirs(tokenize_detail_dir, exist_ok=True)
-    tokenize_detail_path = os.path.join(tokenize_detail_dir, cache_id.replace("/", ".") + ".diff.json")
-    stat.update(_tokenize(tokenizer, datasets, detail_path=tokenize_detail_path))
-    # add basic info
-
-    # save to cache
-    len_before = len(cache)
-    cache[cache_id] = stat
-    len_after = len(cache)
-    logger.info(f"saving '{cache_id}' to memory and file cache '{cache_path}': {len_before}->{len_after}")
-    with open(cache_path, "w", encoding="utf-8") as f_tmp:
-        json.dump(cache, f_tmp, ensure_ascii=False, indent=2)
-    return stat
-
-
-def get_compression_leaderboard(
-        corpuses: List[str] = ['cc100/en'],
-        unit: str = "b_tokens/g_bytes",
-        tokenizer_filter: Optional[str] = None,
-        return_type: Optional[Literal["dict", "dataframe"]] = "dataframe"
-) -> Union[pd.DataFrame, dict]:
-    """
-    """
-    logger.info(f"corpuses: {corpuses}; unit: {unit}; tokenizer_filter: {tokenizer_filter}")
-    stats = {}
-    if tokenizer_filter is not None:
-        tokenizer_names = [tokenizer_name for tokenizer_name in tokenizer_factory.all_tokenizer_names
-                           if tokenizer_filter.lower() in tokenizer_name.lower()]
-    else:
-        tokenizer_names = tokenizer_factory.all_tokenizer_names
-    for tokenizer_name in tokenizer_names:
-        stats_by_corpus = {}
-        for corpus in corpuses:
-            stats_by_corpus[corpus] = tokenize_corpus(tokenizer_name, [corpus])
-        stats[tokenizer_name] = _merge_stats_by_corpus(stats_by_corpus)
-
-    if return_type == "dataframe":
-        token_number_unit, file_size_unit = unit.split("/")
-        reverse_unit = f"{file_size_unit}/{token_number_unit}"
-        stats = to_dataframe(stats, [unit, reverse_unit, "char/token"])
-        stats = stats.sort_values(["oov_ratio", unit], ascending=[True, True])
-        stats = stats.rename(columns={"oov_ratio": f' ⬆️oov_ratio'}).rename(columns={unit: f' ⬆️{unit}'})  # ⬇
-    return stats
-
-
-def main():
-    if len(sys.argv) == 3:
-        tokenizer_filter = [sys.argv[1]]
-        corpuses = [sys.argv[2]]
-    else:
-        tokenizer_filter = None
-        corpuses = common_corpuses
-        # tokenizer_filter = "openai"
-        # corpuses = ["cc100/en", "cc100/zh-Hans"]
-    df = get_compression_leaderboard(corpuses, tokenizer_filter=tokenizer_filter)
-    # print(df.to_markdown(index=False, tablefmt='fancy_grid'))
-    logger.info(f"\n{df.to_markdown(index=False)}")
-
-
-if __name__ == "__main__":
-    main()
+"""
+## more statistics
+code:
+math：
+digit:
+whitespace:
+top_oov: most frequent oov chars
+ranking: thumb_up thumb_down
+"""
+
+import json
+import os
+import sys
+from difflib import SequenceMatcher
+import pandas as pd
+from datasets import load_dataset
+from utils.log_util import logger
+from vocab import tokenizer_factory, TokenizerConfig
+from typing import List, Optional, Union, Literal
+
+CURRENT_DIR = os.path.dirname(os.path.abspath(__file__))
+
+common_units = ["g_bytes/b_tokens", "b_tokens/g_bytes", "t_bytes/t_tokens", "t_tokens/t_bytes", "n_chars/n_tokens", ]
+
+common_corpuses = sorted(["cc100/en", "cc100/zh-Hans", "cc100/es", "cc100/fr", "cc100/de", "cc100/ko",
+                          "cc100/fa", "cc100/ar", "cc100/ja"])
+
+VALID_CODES_CC100 = [
+    "am", "ar", "as", "az", "be", "bg", "bn", "bn_rom", "br", "bs", "ca", "cs", "cy", "da", "de",
+    "el", "en", "eo", "es", "et", "eu", "fa", "ff", "fi", "fr", "fy", "ga", "gd", "gl", "gn", "gu",
+    "ha", "he", "hi", "hi_rom", "hr", "ht", "hu", "hy", "id", "ig", "is", "it", "ja", "jv", "ka",
+    "kk", "km", "kn", "ko", "ku", "ky", "la", "lg", "li", "ln", "lo", "lt", "lv", "mg", "mk", "ml",
+    "mn", "mr", "ms", "my", "my_zaw", "ne", "nl", "no", "ns", "om", "or", "pa", "pl", "ps", "pt",
+    "qu", "rm", "ro", "ru", "sa", "si", "sc", "sd", "sk", "sl", "so", "sq", "sr", "ss", "su", "sv",
+    "sw", "ta", "ta_rom", "te", "te_rom", "th", "tl", "tn", "tr", "ug", "uk", "ur", "ur_rom", "uz",
+    "vi", "wo", "xh", "yi", "yo", "zh-Hans", "zh-Hant", "zu",
+]
+
+
+# code: https://huggingface.co/datasets/codeparrot/github-code-clean  python java c sql html
+# math:
+
+def get_n_bytes_of_string(string_text):
+    n_bytes = len(string_text.encode("utf-8"))
+    return n_bytes
+
+
+def unit_convertor(stat, unit):
+    n_tokens = stat["_n_tokens"]
+    n_chars = stat["_n_chars"]
+    n_bytes = stat["_n_bytes"]
+
+    if n_tokens is None:
+        return None
+
+    n_tokens_in_billion = n_tokens / (1000 * 1000 * 1000)
+    n_tokens_in_trillion = n_tokens / (1000 * 1000 * 1000 * 1000)
+    n_bytes_in_mb = n_bytes / (1024 * 1024)
+    n_bytes_in_gb = n_bytes_in_mb / 1024
+    n_bytes_in_tb = n_bytes_in_gb / 1024
+    # n_chars_in_billion = n_chars / (1000 * 1000 * 1000)
+
+    if unit == "n_tokens/n_bytes":
+        value = n_tokens / n_bytes
+    elif unit in ["char/token", "chars_per_token"]:  # 重要：平均一个token包含多少个字符。
+        value = n_chars / n_tokens
+    elif unit in ["token/char", "tokens_per_char"]:  # 一个中文汉字需要几个token？
+        value = n_tokens / n_chars
+    elif unit == "g_bytes/b_tokens":
+        value = n_bytes_in_gb / n_tokens_in_billion
+    elif unit == "b_tokens/g_bytes":
+        value = n_tokens_in_billion / n_bytes_in_gb
+    elif unit == "t_bytes/t_tokens":  # 重要：
+        value = n_bytes_in_tb / n_tokens_in_trillion
+    elif unit == "t_tokens/t_bytes":
+        value = n_tokens_in_trillion / n_bytes_in_tb
+    else:
+        raise "measure not support"
+    return round(value, 3)
+
+
+def _merge_stats_by_corpus(stats_by_corpus, oov_threshold=0.3):
+    """
+    """
+    all_stats = list(stats_by_corpus.values())
+    assert len(set([stats["tokenizer"] for stats in all_stats])) == 1
+    lossless = all(stat['lossless'] for stat in all_stats)
+    is_support = all(stat['oov_ratio'] < oov_threshold for stat in all_stats)
+
+    merged_stats = {
+        "tokenizer": all_stats[0]["tokenizer"],
+        "organization": all_stats[0]["organization"],
+        "vocab_size": all_stats[0]["vocab_size"],
+        "_n_bytes": 0,
+        "_n_tokens": 0 if is_support else None,
+        "_n_chars": 0,
+        "_n_oov_chars": 0,
+        "lossless": True,
+    }
+    for stats in all_stats:
+        merged_stats["_n_bytes"] += stats["_n_bytes"]
+        merged_stats["_n_chars"] += stats["_n_chars"]
+        if is_support:  # The number of tokens cannot be accurately counted, when there are too many UNKs.
+            merged_stats["_n_tokens"] += stats["_n_tokens"]
+        merged_stats["_n_oov_chars"] += stats["_n_oov_chars"]
+        merged_stats["lossless"] &= stats['lossless']
+
+    merged_stats.update({
+        "oov_ratio": float("%.4g" % (stats["_n_oov_chars"] / stats["_n_chars"])),
+        "lossless": lossless
+    })
+    return merged_stats
+
+
+def to_dataframe(stats, units=None):
+    if units is None:
+        units = common_units
+    elif not isinstance(units, list):
+        units = [units]
+    table = []
+
+    for stat in stats.values():
+        columns = {k: v for k, v in stat.items() if not k.startswith("_")}
+        for unit in units:
+            if unit not in stat:
+                columns[unit] = unit_convertor(stat, unit)
+            else:
+                logger.error(f"unit {unit} not support")
+        table.append(columns)
+    df = pd.DataFrame(table)
+    return df
+
+
+cache = {}
+
+
+def tokenize_corpus(
+        tokenizer_name: str,
+        corpuses: List[str],
+        cache_dir: str = "stats"
+) -> dict:
+    """
+    :param tokenizer_name:
+    :param corpuses:
+    :param cache_dir:
+    :return:
+    """
+
+    def _assert_oov(tokenizer, oov_candidate):
+
+        tokenizer.encode()
+
+    def _char_based_oov(src_text, decoded_text, tokenizer):
+        oov_charset = []  # keep the order in src_text
+        decoded_charset = set(decoded_text)
+        for char in dict.fromkeys(src_text):
+            if char not in decoded_charset \
+                    and char != tokenizer.decode(tokenizer.encode(char, add_special_tokens=False)):
+                oov_charset.append(char)
+
+        n_oov_chars = sum([1 for char in src_text if char in oov_charset])
+        return n_oov_chars, oov_charset
+
+    def _diff_path(src_text, decoded_text):
+        s = SequenceMatcher(a=src_text, b=decoded_text)
+        changes = []
+        for tag, i1, i2, j1, j2 in s.get_opcodes():
+            if tag != "equal":
+                changes.append('{:7}   text[{}:{}] --> decoded_text[{}:{}] {!r:>8} --> {!r}'.format(
+                    tag, i1, i2, j1, j2, src_text[i1:i2], decoded_text[j1:j2]))
+        return changes
+
+    def _tokenize(tokenizer, datasets, detail_path=None):
+        """
+        :param tokenizer:
+        :param datasets:
+        :param detail_path:
+        :return:
+        """
+        n_bytes = 0
+        n_tokens = 0
+        n_chars = 0
+        n_oov_chars = 0
+        diff_details = []
+        oov_charset = set()
+        unk_token_id = None
+        if hasattr(tokenizer, "unk_token"):
+            unk_token_id = tokenizer.unk_token_id
+        for dataset in datasets:
+            for item in dataset:
+                text = item["text"]
+                n_bytes += get_n_bytes_of_string(text)
+                n_chars += len(text)
+                ids = tokenizer.encode(text, add_special_tokens=False)
+
+                # detect oov
+                decoded_text = tokenizer.decode(ids)
+                decoded_text_without_unk = tokenizer.decode([token_id for token_id in ids if token_id != unk_token_id])
+                if decoded_text != text:
+                    _n_oov_chars, _oov_charset = _char_based_oov(text, decoded_text_without_unk, tokenizer)
+                    diffs = _diff_path(text, decoded_text)
+                    diff_details.append(
+                        {
+                            "text": text,
+                            "decoded_text": decoded_text,
+                            "diff": diffs,
+                            "n_oov_chars": _n_oov_chars,
+                            'oov_ratio': _n_oov_chars / len(text),
+                            'oov_charset': json.dumps(_oov_charset, ensure_ascii=False),
+                        }
+                    )
+                    n_oov_chars += _n_oov_chars
+                    oov_charset.update(_oov_charset)
+                n_tokens += len(ids)
+        stat = {
+            "_n_bytes": n_bytes,
+            "_n_tokens": n_tokens,
+            "_n_chars": n_chars,
+            "_n_oov_chars": n_oov_chars,
+            "oov_ratio": n_oov_chars / n_chars,
+            '_oov_charset': json.dumps(list(oov_charset), ensure_ascii=False),
+            "lossless": len(diff_details) == 0
+        }
+
+        if detail_path and diff_details:
+            logger.info(f"saving tokenization detail to '{detail_path}'")
+            with open(detail_path, "w", encoding="utf-8") as f:
+                f.write(json.dumps(diff_details, ensure_ascii=False, indent=2))
+                # print(f"{tokenizer_config.name_or_path}, {infer_tokenizer_type(tokenizer_config)}\n"
+                #       f"lossless: false; unk_token: {get_unk(tokenizer_config)},"
+                #       f" unk_ratio: {unk_count / len(encoding):.4f}; oov: []")
+                # for diff_detail in diff_details:
+                #         # print(f"text[{i}]     = {str(bytes(text[i:], 'utf-8'))}\n"
+                #         #       f"decoding[{i}] = {str(bytes(decoding[i:], 'utf-8'))}")
+                #         f.write(f"text= {json.dumps(text[i:], ensure_ascii=False)}, \n"
+                #               f"decoding[{i}] = {json.dumps(decoding[i:], ensure_ascii=False)}")
+        return stat
+
+    # load from cache
+    cache_id = f"{tokenizer_name} @ {'.'.join(corpuses)}"
+    cache_path = os.path.join(cache_dir, "compression_rate.json")
+    if not cache and os.path.exists(cache_path):
+        with open(cache_path, "r", encoding="utf-8") as f_tmp:
+            cache.update(json.load(f_tmp))
+    if cache_id in cache:
+        # logger.info(f"loading {cache_id} from in-memory cache")
+        return cache[cache_id]
+
+    # tokenize corpus
+    tokenizer = tokenizer_factory.get_tokenizer(tokenizer_name)
+    datasets = [load_dataset("eson/cc100-samples", corpus.replace("cc100/", ""), split="train") for corpus in corpuses]
+
+    stat = {
+        "tokenizer": tokenizer_factory.get_name_with_hyperlink(tokenizer_name),
+        "organization": tokenizer_factory.get_tokenizer_config(tokenizer_name).org,
+        "vocab_size": len(tokenizer),
+    }
+    tokenize_detail_dir = os.path.join(cache_dir, "compression_rate")
+    os.makedirs(tokenize_detail_dir, exist_ok=True)
+    tokenize_detail_path = os.path.join(tokenize_detail_dir, cache_id.replace("/", ".") + ".diff.json")
+    stat.update(_tokenize(tokenizer, datasets, detail_path=tokenize_detail_path))
+    # add basic info
+
+    # save to cache
+    len_before = len(cache)
+    cache[cache_id] = stat
+    len_after = len(cache)
+    logger.info(f"saving '{cache_id}' to memory and file cache '{cache_path}': {len_before}->{len_after}")
+    with open(cache_path, "w", encoding="utf-8") as f_tmp:
+        json.dump(cache, f_tmp, ensure_ascii=False, indent=2)
+    return stat
+
+
+def get_compression_leaderboard(
+        corpuses: List[str] = ['cc100/en'],
+        unit: str = "b_tokens/g_bytes",
+        tokenizer_filter: Optional[str] = None,
+        return_type: Optional[Literal["dict", "dataframe"]] = "dataframe"
+) -> Union[pd.DataFrame, dict]:
+    """
+    """
+    logger.info(f"corpuses: {corpuses}; unit: {unit}; tokenizer_filter: {tokenizer_filter}")
+    stats = {}
+    if tokenizer_filter is not None:
+        tokenizer_names = [tokenizer_name for tokenizer_name in tokenizer_factory.all_tokenizer_names
+                           if tokenizer_filter.lower() in tokenizer_name.lower()]
+    else:
+        tokenizer_names = tokenizer_factory.all_tokenizer_names
+    for tokenizer_name in tokenizer_names:
+        stats_by_corpus = {}
+        for corpus in corpuses:
+            stats_by_corpus[corpus] = tokenize_corpus(tokenizer_name, [corpus])
+        stats[tokenizer_name] = _merge_stats_by_corpus(stats_by_corpus)
+
+    if return_type == "dataframe":
+        token_number_unit, file_size_unit = unit.split("/")
+        reverse_unit = f"{file_size_unit}/{token_number_unit}"
+        stats = to_dataframe(stats, [unit, reverse_unit, "char/token"])
+        stats = stats.sort_values(["oov_ratio", unit], ascending=[True, True])
+        stats = stats.rename(columns={"oov_ratio": f' ⬆️oov_ratio'}).rename(columns={unit: f' ⬆️{unit}'})  # ⬇
+    return stats
+
+
+def main():
+    if len(sys.argv) == 3:
+        tokenizer_filter = [sys.argv[1]]
+        corpuses = [sys.argv[2]]
+    else:
+        tokenizer_filter, corpuses = None, common_corpuses
+        # tokenizer_filter, corpuses = "openai", ["cc100/en", "cc100/zh-Hans"]
+        # tokenizer_filter, corpuses = "Qwen/Qwen1.5-14B", ["cc100/de"]
+        # tokenizer_filter, corpuses = "Qwen/Qwen1.5-14B", ["cc100/ja"]  # oov 特别多
+        # tokenizer_filter, corpuses = "google-bert/bert-base-uncased", ["cc100/ja", "cc100/zh-Hans"]  # oov 特别多
+    df = get_compression_leaderboard(corpuses, tokenizer_filter=tokenizer_filter)
+    # print(df.to_markdown(index=False, tablefmt='fancy_grid'))
+    logger.info(f"\n{df.to_markdown(index=False)}")
+
+
+if __name__ == "__main__":
+    main()

css/style.css

@@ -1,59 +1,62 @@
-
-/* 显示空格：https://blog.csdn.net/liuxiao723846/article/details/118994673 */
-.space-show {
-	white-space: pre-wrap;
-}
-
-.cell-wrap {
-	white-space: pre-wrap;
-}
-
-
-/* white button */
-.button-as-text {
-	background: #fff;
-    border-color: #fff;
-}
-
-.textbox-as-text {
-    border-style: hidden;
-    background: #fff;
-    border-color: #fff;
-}
-
-
-.h2-font {
-    font-size: 30px;
-}
-
-.no-border {
-    border: 0px none;
-}
-
-/* 隐藏legend */
-.category-legend {
-	display: none !important;
-}
-
-.statistics {
-	min-width: min(50px, 100%) !important;
-}
-
-.statistics textarea {
-	min-width: min(50px, 100%) !important;
-	font-size: 20px !important;
-	font-weight: 600 !important;
-	text-align: center !important;
-	border: none !important;
-}
-
-.statistics label {
-	text-align: center !important;
-}
-
-/* align-self: flex-end; */
-.example-style {
-	max-width: 150px;
-	align-self: self-end;
-}
-
+/* hidde legend of HighlightText, has been integrated in gradio.4.32.0 with `show_inline_category`
+.category-legend {
+	display: none !important;
+}
+*/
+
+/* show space in HighlightText：https://blog.csdn.net/liuxiao723846/article/details/118994673
+TODO: integrate in gradio with `show_single_whitespace=True` or `strip_token=False`
+*/
+.space-show {
+	white-space: pre-wrap;
+}
+
+.cell-wrap {
+	white-space: pre-wrap;
+}
+
+
+/* white button */
+.button-as-text {
+	background: #fff;
+    border-color: #fff;
+}
+
+.textbox-as-text {
+    border-style: hidden;
+    background: #fff;
+    border-color: #fff;
+}
+
+
+.h2-font {
+    font-size: 30px;
+}
+
+.no-border {
+    border: 0px none;
+}
+
+
+.statistics {
+	min-width: min(50px, 100%) !important;
+}
+
+.statistics textarea {
+	min-width: min(50px, 100%) !important;
+	font-size: 20px !important;
+	font-weight: 600 !important;
+	text-align: center !important;
+	border: none !important;
+}
+
+.statistics label {
+	text-align: center !important;
+}
+
+/* align-self: flex-end; */
+.example-style {
+	max-width: 150px;
+	align-self: self-end;
+}
+

playground_app.py

@@ -1,264 +1,233 @@
-# coding=utf-8
-# author: xusong
-# time: 2022/8/23 16:06
-
-"""
-## TODO:
-- i18 国际化  https://blog.csdn.net/qq_26212731/article/details/78457198   request.header中也有language
-- iter_vocab 的 warmup
-- 开关
-  - add_special_token 开关
-  - theme 开关 light/dark
-  - token_id/tokens/bytes 开关
-  - 中文字词统计，是否要包括 _ G 等字符
-- 评测
-  - OOV评测
-- 通过 javascript 添加 hover_text
-- 英文 utf-8编码
-- 词典支持下载，借用image下载的标签，
-- baichuan的单字数量怎么两万多个？
-- qwen:  ValueError: Unclosed image token
-- 路径修改为全path  meta-llama/Llama-2-13b-hf
-
-plots
-
-table
-
-## related demo
-- [](http://text-processing.com/demo/tokenize/)
-- [gpt-tokenizer](https://gpt-tokenizer.dev/)
-- [llama-tokenizer-js](https://belladoreai.github.io/llama-tokenizer-js/example-demo/build/)
-- [](https://huggingface.co/spaces/Xenova/the-tokenizer-playground)
-
-## 可视化
-
-[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone ]
-"""
-
-import gradio as gr
-from vocab import tokenizer_factory
-from playground_examples import example_types, example_fn
-from playground_util import tokenize, tokenize_pair, basic_count, get_overlap_token_size, on_load
-
-
-
-
-get_window_url_params = """
-    function(url_params) {
-        const params = new URLSearchParams(window.location.search);
-        url_params = JSON.stringify(Object.fromEntries(params));
-        return url_params;
-        }
-    """
-
-all_tokenizer_name = [(config.name_display, config.name_or_path) for config in tokenizer_factory.all_tokenizer_configs]
-
-with gr.Blocks() as demo:
-    # links: https://www.coderstool.com/utf8-encoding-decoding
-    # 功能：输入文本，进行分词
-    # 分词器：常见的分词器有集中，
-    # 背景：方便分词、看词粒度、对比
-
-    with gr.Row():
-        gr.Markdown("## Input Text")
-        dropdown_examples = gr.Dropdown(
-            example_types,
-            value="Examples",
-            type="index",
-            allow_custom_value=True,
-            show_label=False,
-            container=False,
-            scale=0,
-            elem_classes="example-style"
-        )
-    user_input = gr.Textbox(
-        # value=default_user_input,
-        label="Input Text",
-        lines=5,
-        show_label=False,
-    )
-    gr.Markdown("## Tokenization")
-
-    # compress rate setting TODO: 将 这个模块调整到下面
-    # with gr.Accordion("Compress Rate Setting", open=True):
-    #     gr.Markdown(
-    #         "Please select corpus and unit of compress rate, get more details at [github](https://github.com/xu-song/tokenizer-arena/). ")
-    #     with gr.Row():
-    #         compress_rate_corpus = gr.CheckboxGroup(
-    #             common_corpuses,  # , "code"
-    #             value=["cc100-en", "cc100-zh-Hans"],
-    #             label="corpus",
-    #             # info=""
-    #         )
-    #         compress_rate_unit = gr.Radio(
-    #             common_units,
-    #             value="b_tokens/g_bytes",
-    #             label="unit",
-    #         )
-    # TODO: Token Setting
-    # with gr.Accordion("Token Filter Setting", open=False):
-    #     gr.Markdown(
-    #         "Get total number of tokens which contain the following character)")
-    #     gr.Radio(
-    #         ["zh-Hans", "", "number", "space"],
-    #         value="zh",
-    #     )
-
-    with gr.Row():
-        with gr.Column(scale=6):
-            with gr.Group():
-                tokenizer_name_1 = gr.Dropdown(
-                    all_tokenizer_name,
-                    label="Tokenizer 1",
-                )
-                with gr.Group():
-                    with gr.Row():
-                        organization_1 = gr.TextArea(
-                            label="Organization",
-                            lines=1,
-                            elem_classes="statistics",
-                        )
-                        stats_vocab_size_1 = gr.TextArea(
-                            label="Vocab Size",
-                            lines=1,
-                            elem_classes="statistics"
-                        )
-                        # stats_zh_token_size_1 = gr.TextArea(
-                        #     label="ZH char/word",
-                        #     lines=1,
-                        #     elem_classes="statistics",
-                        # )
-                        # stats_compress_rate_1 = gr.TextArea(
-                        #     label="Compress Rate",
-                        #     lines=1,
-                        #     elem_classes="statistics",
-                        # )
-                        stats_overlap_token_size_1 = gr.TextArea(
-                            # value=default_stats_overlap_token_size,
-                            label="Overlap Tokens",
-                            lines=1,
-                            elem_classes="statistics"
-                        )
-                        # stats_3 = gr.TextArea(
-                        #     label="Compress Rate",
-                        #     lines=1,
-                        #     elem_classes="statistics"
-                        # )
-        # https://www.onlinewebfonts.com/icon/418591
-        gr.Image("images/VS.svg", scale=1, show_label=False,
-                 show_download_button=False, container=False,
-                 show_share_button=False)
-        with gr.Column(scale=6):
-            with gr.Group():
-                tokenizer_name_2 = gr.Dropdown(
-                    all_tokenizer_name,
-                    label="Tokenizer 2",
-                )
-                with gr.Group():
-                    with gr.Row():
-                        organization_2 = gr.TextArea(
-                            label="Organization",
-                            lines=1,
-                            elem_classes="statistics",
-                        )
-                        stats_vocab_size_2 = gr.TextArea(
-                            label="Vocab Size",
-                            lines=1,
-                            elem_classes="statistics"
-                        )
-                        # stats_zh_token_size_2 = gr.TextArea(
-                        #     label="ZH char/word",  # 中文字/词
-                        #     lines=1,
-                        #     elem_classes="statistics",
-                        # )
-                        # stats_compress_rate_2 = gr.TextArea(
-                        #     label="Compress Rate",
-                        #     lines=1,
-                        #     elem_classes="statistics"
-                        # )
-                        stats_filtered_token_2 = gr.TextArea(
-                            label="filtered tokens",
-                            lines=1,
-                            elem_classes="statistics",
-                            visible=False
-                        )
-                        stats_overlap_token_size_2 = gr.TextArea(
-                            label="Overlap Tokens",
-                            lines=1,
-                            elem_classes="statistics"
-                        )
-
-    # TODO: 图 表 压缩率
-    with gr.Row():
-        # dynamic change label
-        with gr.Column():
-            output_text_1 = gr.Highlightedtext(
-                show_legend=True,
-                elem_classes="space-show"
-            )
-        with gr.Column():
-            output_text_2 = gr.Highlightedtext(
-                show_legend=True,
-                elem_classes="space-show"
-            )
-
-    with gr.Row():
-        output_table_1 = gr.Dataframe()
-        output_table_2 = gr.Dataframe()
-
-    # setting
-    # compress_rate_unit.change(compress_rate_unit_change, [compress_rate_unit],
-    #                             [stats_compress_rate_1, stats_compress_rate_2])
-
-    tokenizer_name_1.change(tokenize, [user_input, tokenizer_name_1],
-                            [output_text_1, output_table_1])
-    tokenizer_name_1.change(basic_count, [tokenizer_name_1], [stats_vocab_size_1, organization_1])
-    tokenizer_name_1.change(get_overlap_token_size, [tokenizer_name_1, tokenizer_name_2],
-                            [stats_overlap_token_size_1, stats_overlap_token_size_2])
-    # tokenizer_type_1.change(get_compress_rate, [tokenizer_type_1, compress_rate_corpus, compress_rate_unit],
-    #                         [stats_compress_rate_1])
-
-    # TODO: every=3
-    user_input.change(tokenize_pair,
-                      [user_input, tokenizer_name_1, tokenizer_name_2],
-                      [output_text_1, output_table_1, output_text_2, output_table_2])  # , pass_request=1
-
-    tokenizer_name_2.change(tokenize, [user_input, tokenizer_name_2],
-                            [output_text_2, output_table_2])
-    tokenizer_name_2.change(basic_count, [tokenizer_name_2], [stats_vocab_size_2, organization_2])
-    tokenizer_name_2.change(get_overlap_token_size, [tokenizer_name_1, tokenizer_name_2],
-                            [stats_overlap_token_size_1, stats_overlap_token_size_2])
-    # tokenizer_type_2.change(get_compress_rate,
-    #                         [tokenizer_type_2, compress_rate_corpus, compress_rate_unit],
-    #                         [stats_compress_rate_2])
-    #
-    # compress_rate_unit.change(get_compress_rate,
-    #                           [tokenizer_type_1, compress_rate_corpus, compress_rate_unit],
-    #                           [stats_compress_rate_1])
-    # compress_rate_unit.change(get_compress_rate,
-    #                           [tokenizer_type_2, compress_rate_corpus, compress_rate_unit],
-    #                           [stats_compress_rate_2])
-    # compress_rate_corpus.change(get_compress_rate,
-    #                             [tokenizer_type_1, compress_rate_corpus, compress_rate_unit],
-    #                             [stats_compress_rate_1])
-    # compress_rate_corpus.change(get_compress_rate,
-    #                             [tokenizer_type_2, compress_rate_corpus, compress_rate_unit],
-    #                             [stats_compress_rate_2])
-
-    dropdown_examples.change(
-        example_fn,
-        dropdown_examples,
-        [user_input, tokenizer_name_1, tokenizer_name_2]
-    )
-
-    demo.load(
-        fn=on_load,
-        inputs=[user_input],  # 这里只需要传个空object即可。
-        outputs=[user_input, tokenizer_name_1, tokenizer_name_2],
-        js=get_window_url_params
-    )
-
-if __name__ == "__main__":
-    # demo.queue(max_size=20).launch()
-    demo.launch()
-    # demo.launch(share=True)
+# coding=utf-8
+# author: xusong
+# time: 2022/8/23 16:06
+
+import gradio as gr
+from vocab import tokenizer_factory
+from playground_examples import example_types, example_fn
+from playground_util import tokenize, tokenize_pair, basic_count, get_overlap_token_size, on_load
+
+
+
+
+get_window_url_params = """
+    function(url_params) {
+        const params = new URLSearchParams(window.location.search);
+        url_params = JSON.stringify(Object.fromEntries(params));
+        return url_params;
+        }
+    """
+
+all_tokenizer_name = [(config.name_display, config.name_or_path) for config in tokenizer_factory.all_tokenizer_configs]
+
+with gr.Blocks() as demo:
+    # links: https://www.coderstool.com/utf8-encoding-decoding
+    # 功能：输入文本，进行分词
+    # 分词器：常见的分词器有集中，
+    # 背景：方便分词、看词粒度、对比
+
+    with gr.Row():
+        gr.Markdown("## Input Text")
+        dropdown_examples = gr.Dropdown(
+            example_types,
+            value="Examples",
+            type="index",
+            allow_custom_value=True,
+            show_label=False,
+            container=False,
+            scale=0,
+            elem_classes="example-style"
+        )
+    user_input = gr.Textbox(
+        # value=default_user_input,
+        label="Input Text",
+        lines=5,
+        show_label=False,
+    )
+    gr.Markdown("## Tokenization")
+
+    # compress rate setting TODO: 将 这个模块调整到下面
+    # with gr.Accordion("Compress Rate Setting", open=True):
+    #     gr.Markdown(
+    #         "Please select corpus and unit of compress rate, get more details at [github](https://github.com/xu-song/tokenizer-arena/). ")
+    #     with gr.Row():
+    #         compress_rate_corpus = gr.CheckboxGroup(
+    #             common_corpuses,  # , "code"
+    #             value=["cc100-en", "cc100-zh-Hans"],
+    #             label="corpus",
+    #             # info=""
+    #         )
+    #         compress_rate_unit = gr.Radio(
+    #             common_units,
+    #             value="b_tokens/g_bytes",
+    #             label="unit",
+    #         )
+    # TODO: Token Setting
+    # with gr.Accordion("Token Filter Setting", open=False):
+    #     gr.Markdown(
+    #         "Get total number of tokens which contain the following character)")
+    #     gr.Radio(
+    #         ["zh-Hans", "", "number", "space"],
+    #         value="zh",
+    #     )
+
+    with gr.Row():
+        with gr.Column(scale=6):
+            with gr.Group():
+                tokenizer_name_1 = gr.Dropdown(
+                    all_tokenizer_name,
+                    label="Tokenizer 1",
+                )
+                with gr.Group():
+                    with gr.Row():
+                        organization_1 = gr.TextArea(
+                            label="Organization",
+                            lines=1,
+                            elem_classes="statistics",
+                        )
+                        stats_vocab_size_1 = gr.TextArea(
+                            label="Vocab Size",
+                            lines=1,
+                            elem_classes="statistics"
+                        )
+                        # stats_zh_token_size_1 = gr.TextArea(
+                        #     label="ZH char/word",
+                        #     lines=1,
+                        #     elem_classes="statistics",
+                        # )
+                        # stats_compress_rate_1 = gr.TextArea(
+                        #     label="Compress Rate",
+                        #     lines=1,
+                        #     elem_classes="statistics",
+                        # )
+                        stats_overlap_token_size_1 = gr.TextArea(
+                            # value=default_stats_overlap_token_size,
+                            label="Overlap Tokens",
+                            lines=1,
+                            elem_classes="statistics"
+                        )
+                        # stats_3 = gr.TextArea(
+                        #     label="Compress Rate",
+                        #     lines=1,
+                        #     elem_classes="statistics"
+                        # )
+        # https://www.onlinewebfonts.com/icon/418591
+        gr.Image("images/VS.svg", scale=1, show_label=False,
+                 show_download_button=False, container=False,
+                 show_share_button=False)
+        with gr.Column(scale=6):
+            with gr.Group():
+                tokenizer_name_2 = gr.Dropdown(
+                    all_tokenizer_name,
+                    label="Tokenizer 2",
+                )
+                with gr.Group():
+                    with gr.Row():
+                        organization_2 = gr.TextArea(
+                            label="Organization",
+                            lines=1,
+                            elem_classes="statistics",
+                        )
+                        stats_vocab_size_2 = gr.TextArea(
+                            label="Vocab Size",
+                            lines=1,
+                            elem_classes="statistics"
+                        )
+                        # stats_zh_token_size_2 = gr.TextArea(
+                        #     label="ZH char/word",  # 中文字/词
+                        #     lines=1,
+                        #     elem_classes="statistics",
+                        # )
+                        # stats_compress_rate_2 = gr.TextArea(
+                        #     label="Compress Rate",
+                        #     lines=1,
+                        #     elem_classes="statistics"
+                        # )
+                        stats_filtered_token_2 = gr.TextArea(
+                            label="filtered tokens",
+                            lines=1,
+                            elem_classes="statistics",
+                            visible=False
+                        )
+                        stats_overlap_token_size_2 = gr.TextArea(
+                            label="Overlap Tokens",
+                            lines=1,
+                            elem_classes="statistics"
+                        )
+
+    # TODO: 图 表 压缩率
+    with gr.Row():
+        # dynamic change label
+        with gr.Column():
+            output_text_1 = gr.Highlightedtext(
+                show_legend=False,
+                show_inline_category=False,
+                elem_classes="space-show"
+            )
+        with gr.Column():
+            output_text_2 = gr.Highlightedtext(
+                show_legend=False,
+                show_inline_category=False,
+                elem_classes="space-show"
+            )
+
+    with gr.Row():
+        output_table_1 = gr.Dataframe()
+        output_table_2 = gr.Dataframe()
+
+    # setting
+    # compress_rate_unit.change(compress_rate_unit_change, [compress_rate_unit],
+    #                             [stats_compress_rate_1, stats_compress_rate_2])
+
+    tokenizer_name_1.change(tokenize, [user_input, tokenizer_name_1],
+                            [output_text_1, output_table_1])
+    tokenizer_name_1.change(basic_count, [tokenizer_name_1], [stats_vocab_size_1, organization_1])
+    tokenizer_name_1.change(get_overlap_token_size, [tokenizer_name_1, tokenizer_name_2],
+                            [stats_overlap_token_size_1, stats_overlap_token_size_2])
+    # tokenizer_type_1.change(get_compress_rate, [tokenizer_type_1, compress_rate_corpus, compress_rate_unit],
+    #                         [stats_compress_rate_1])
+
+    # TODO: every=3
+    user_input.change(tokenize_pair,
+                      [user_input, tokenizer_name_1, tokenizer_name_2],
+                      [output_text_1, output_table_1, output_text_2, output_table_2])  # , pass_request=1
+
+    tokenizer_name_2.change(tokenize, [user_input, tokenizer_name_2],
+                            [output_text_2, output_table_2])
+    tokenizer_name_2.change(basic_count, [tokenizer_name_2], [stats_vocab_size_2, organization_2])
+    tokenizer_name_2.change(get_overlap_token_size, [tokenizer_name_1, tokenizer_name_2],
+                            [stats_overlap_token_size_1, stats_overlap_token_size_2])
+    # tokenizer_type_2.change(get_compress_rate,
+    #                         [tokenizer_type_2, compress_rate_corpus, compress_rate_unit],
+    #                         [stats_compress_rate_2])
+    #
+    # compress_rate_unit.change(get_compress_rate,
+    #                           [tokenizer_type_1, compress_rate_corpus, compress_rate_unit],
+    #                           [stats_compress_rate_1])
+    # compress_rate_unit.change(get_compress_rate,
+    #                           [tokenizer_type_2, compress_rate_corpus, compress_rate_unit],
+    #                           [stats_compress_rate_2])
+    # compress_rate_corpus.change(get_compress_rate,
+    #                             [tokenizer_type_1, compress_rate_corpus, compress_rate_unit],
+    #                             [stats_compress_rate_1])
+    # compress_rate_corpus.change(get_compress_rate,
+    #                             [tokenizer_type_2, compress_rate_corpus, compress_rate_unit],
+    #                             [stats_compress_rate_2])
+
+    dropdown_examples.change(
+        example_fn,
+        dropdown_examples,
+        [user_input, tokenizer_name_1, tokenizer_name_2]
+    )
+
+    demo.load(
+        fn=on_load,
+        inputs=[user_input],  # 这里只需要传个空object即可。
+        outputs=[user_input, tokenizer_name_1, tokenizer_name_2],
+        js=get_window_url_params
+    )
+
+if __name__ == "__main__":
+    # demo.queue(max_size=20).launch()
+    demo.launch()
+    # demo.launch(share=True)

playground_util.py

@@ -1,181 +1,181 @@
-import gradio as gr
-import json
-import copy
-import pandas as pd
-from vocab import tokenizer_factory
-from character_util import iter_vocab
-from utils.log_util import logger
-from functools import lru_cache
-
-default_user_input = """\
-Replace this text in the input field to see how tokenization works.
-Buenos días!
-华为发布Mate60手机。
-ラグビーワールドカップ2023フランス"""
-# default_tokenizer_name_1 = "Meta/llama3"
-default_tokenizer_name_1 = "gradientai/Llama-3-8B-Instruct-Gradient-1048k"
-default_tokenizer_name_2 = "openai/gpt-4"
-
-
-@lru_cache
-def _tokenize(
-        text: str,
-        tokenizer_name: str,
-        color_num: int = 5,
-        add_special_token: bool = False
-):
-    logger.info("param=" + json.dumps({"text": text, "tokenizer_type": tokenizer_name}, ensure_ascii=False))
-    pos_tokens = []
-    tokenizer = tokenizer_factory.get_tokenizer(tokenizer_name)
-    if add_special_token:
-        encoding = tokenizer.encode(text, add_special_tokens=True)
-    else:
-        encoding = tokenizer.encode(text, add_special_tokens=False)
-
-    table = []
-
-    for idx, token_id in enumerate(encoding):
-        decode_text = tokenizer.decode([token_id])  # 特殊字符解码后会统一变成 �，对应 "\ufffd"
-        pos_tokens.extend([(decode_text, str(idx % color_num))])
-
-        # token  "Byte":  # 这是 utf-8编码吧？
-        token = tokenizer.convert_ids_to_tokens([token_id], skip_special_tokens=False)[0]
-        if isinstance(token, bytes):
-            try:
-                token_str = token.decode("utf-8")
-            except:
-                token_str = token.decode("utf-8", errors="ignore")
-                logger.error(f"{idx}: decode_error: " + json.dumps(  # gpt_35_turbo 经常有token会decode error，这里用来记录一下
-                    {"tokenizer_type": tokenizer_name, "token": str(token), "token_str": token_str},
-                    ensure_ascii=False))
-
-            token_bytes = token
-            # json_dumps = json.dumps(token_str)
-        elif isinstance(token, str):
-            token_str = token
-            token_bytes = bytes(token_str, "utf-8")
-            # json_dumps = json.dumps(token_str)
-        else:
-            logger.error(f"{idx}: wrong type for token {token_id} {type(token)} " + json.dumps(
-                {"text": text, "tokenizer_type": tokenizer_name}, ensure_ascii=False))
-            token_str = token
-            token_bytes = token
-            # continue
-
-        # ⭐
-        # TODO: gpt3.5_turbo错误： 只有id和text是对的，token和 utf8都是错的。说明 convert_ids_to_tokens 出错了。
-        table.append(
-            {"TokenID": token_id,
-             "Token": token_str,  # utf-8解码后的字符串，为什么有些是 <0xE7>，表示什么？比如llama
-             "Text": decode_text,  #
-             # "Bytes": token_bytes,  # bytes类型在gradio前端页面被解码成字符串，比如   b'\xe4\xb8\xad' 仍然显示成 "中"。因此 str(token_bytes)
-             "UTF8 Bytes": str(token_bytes),
-             # "Unicode": json_dumps  # unicode, 如果是ascii码，就直接显示。如果不是ascii码，就显示unicode
-             }
-        )
-
-    table_df = pd.DataFrame(table)
-    logger.info(f"tokenizer_type={tokenizer_name}, Tokens={table[:4]}")
-    return pos_tokens, len(encoding), table_df
-
-
-def tokenize(
-        text: str,
-        tokenizer_name: str,
-        color_num: int = 5,
-        add_special_token: bool = False
-):
-    """ tokenize wrapper
-    As gr.Update would be overwritten after passing to frontend, we apply lru_cache in _tokenize.
-    """
-    pos_tokens, num_tokens, table_df = _tokenize(text, tokenizer_name, color_num, add_special_token)
-    return gr.update(value=pos_tokens, label=f"Tokens: {num_tokens}"), table_df
-
-
-def tokenize_pair(text, tokenizer_type_1, tokenizer_type_2):
-    """
-    input_text.change
-    """
-    pos_tokens_1, table_df_1 = tokenize(text, tokenizer_type_1)
-    pos_tokens_2, table_df_2 = tokenize(text, tokenizer_type_2)
-    return pos_tokens_1, table_df_1, pos_tokens_2, table_df_2
-
-
-@lru_cache
-def basic_count(tokenizer_name):
-    stats = iter_vocab(tokenizer_name)
-    return stats['vocab_size'], f'{stats["organization"]}'
-    # return tokenizer.vocab_size, f'{stats["中文汉字数"]["中文单字"]}/{stats["中文汉字数"]["中文多字"]}'
-
-
-# def get_compress_rate(tokenizer_name, all_corpus, unit):
-#     tokenizer = tokenizer_factory.get_tokenizer(tokenizer_name)
-#     compress_rate_stats = tokenize_corpus(tokenizer, all_corpus)
-#     compress_rate = unit_convertor(compress_rate_stats, unit)
-#     return compress_rate
-
-
-@lru_cache
-def get_overlap_token_size(tokenizer_name_1, tokenizer_name_2):
-    tokenizer1 = tokenizer_factory.get_tokenizer(tokenizer_name_1)
-    tokenizer2 = tokenizer_factory.get_tokenizer(tokenizer_name_2)
-
-    vocab_set_1 = tokenizer1.get_vocab().keys()
-    vocab_set_2 = tokenizer2.get_vocab().keys()
-
-    token1 = next(iter(vocab_set_1))
-    token2 = next(iter(vocab_set_2))
-    if type(token1) != type(token2):  # bytes  str
-        if isinstance(token1, str):
-            vocab_set_1 = set([token.encode("utf-8") for token in vocab_set_1])
-        if isinstance(token2, str):
-            vocab_set_2 = set([token.encode("utf-8") for token in vocab_set_2])
-
-    overlap_tokens = vocab_set_1 & vocab_set_2
-    overlap_token_size = len(overlap_tokens)
-    logger.info(
-        f"{overlap_token_size} OverlapTokens of {tokenizer_name_1} {tokenizer_name_2}: {list(overlap_tokens)[:10]}")
-    return overlap_token_size, overlap_token_size
-
-
-def on_load(url_params, request: gr.Request):
-    """
-    onLoad
-    """
-    text = None
-    tokenizer_type_1 = None
-    tokenizer_type_2 = None
-    try:
-        url_params = json.loads(url_params)
-    except:
-        url_params = {}
-    if request:
-        logger.info(str(request.headers))
-        client_ip = request.client.host
-        # local_ip = socket.gethostbyname(socket.gethostbyname(""))
-        # headers = request.kwargs['headers']
-        # if headers and 'x-forwarded-for' in headers:
-        #     x_forwarded_for = headers['x-forwarded-for']
-        #     client_ip = x_forwarded_for.split(' ')[0] if x_forwarded_for else ""
-        # if "referer" in request.headers:   # not work for huggingface-space
-        #     url_params = parse_qs(urlparse(request.headers["referer"]).query)
-        #     url_params = {k: v[0] for k, v in url_params.items() if len(v) > 0}
-        tokenizer_type_1 = url_params.get("tokenizer1", default_tokenizer_name_1)
-        tokenizer_type_2 = url_params.get("tokenizer2", default_tokenizer_name_2)
-        text = url_params.get("text", default_user_input)
-        logger.info(f"client_ip: {client_ip}; params: {url_params}")
-    return text, tokenizer_type_1, tokenizer_type_2
-
-
-# def compress_rate_unit_change(unit):
-#     return gr.update(label=f"Compress Rate: {unit}"), gr.update(label=f"Compress Rate: {unit}"),
-
-
-def test_coding():
-    bytes1 = b'\xe4\xb8\xad'
-    print(bytes1)  # b'\xe4\xb8\xad'
-
-
-if __name__ == "__main__":
-    print(get_overlap_token_size("gpt-35-turbo", "gpt-4"))
-    # print(basic_count("internlm_chat_7b"))
+import gradio as gr
+import json
+import copy
+import pandas as pd
+from vocab import tokenizer_factory
+from character_util import iter_vocab
+from utils.log_util import logger
+from functools import lru_cache
+
+default_user_input = """\
+Replace this text in the input field to see how tokenization works.
+Buenos días!
+华为发布Mate60手机。
+ラグビーワールドカップ2023フランス"""
+# default_tokenizer_name_1 = "Meta/llama3"
+default_tokenizer_name_1 = "gradientai/Llama-3-8B-Instruct-Gradient-1048k"
+default_tokenizer_name_2 = "openai/gpt-4o"
+
+
+@lru_cache
+def _tokenize(
+        text: str,
+        tokenizer_name: str,
+        color_num: int = 5,
+        add_special_token: bool = False
+):
+    logger.info("param=" + json.dumps({"text": text, "tokenizer_type": tokenizer_name}, ensure_ascii=False))
+    pos_tokens = []
+    tokenizer = tokenizer_factory.get_tokenizer(tokenizer_name)
+    if add_special_token:
+        encoding = tokenizer.encode(text, add_special_tokens=True)
+    else:
+        encoding = tokenizer.encode(text, add_special_tokens=False)
+
+    table = []
+
+    for idx, token_id in enumerate(encoding):
+        decoded_text = tokenizer.decode([token_id])  # 特殊字符解码后会统一变成 �，对应 "\ufffd"
+        pos_tokens.extend([(decoded_text, str(idx % color_num))])
+
+        # token  "Byte":  # 这是 utf-8编码吧？
+        token = tokenizer.convert_ids_to_tokens([token_id], skip_special_tokens=False)[0]
+        if isinstance(token, bytes):
+            try:
+                token_str = token.decode("utf-8")
+            except:
+                token_str = token.decode("utf-8", errors="ignore")
+                logger.error(f"{idx}: decode_error: " + json.dumps(  # gpt_35_turbo 经常有token会decode error，这里用来记录一下
+                    {"tokenizer_type": tokenizer_name, "token": str(token), "token_str": token_str},
+                    ensure_ascii=False))
+
+            token_bytes = token
+            # json_dumps = json.dumps(token_str)
+        elif isinstance(token, str):
+            token_str = token
+            token_bytes = bytes(token_str, "utf-8")
+            # json_dumps = json.dumps(token_str)
+        else:
+            logger.error(f"{idx}: wrong type for token {token_id} {type(token)} " + json.dumps(
+                {"text": text, "tokenizer_type": tokenizer_name}, ensure_ascii=False))
+            token_str = token
+            token_bytes = token
+            # continue
+
+        # ⭐
+        # TODO: gpt3.5_turbo错误： 只有id和text是对的，token和 utf8都是错的。说明 convert_ids_to_tokens 出错了。
+        table.append(
+            {"TokenID": token_id,
+             "Token": token_str,  # utf-8解码后的字符串，为什么���些是 <0xE7>，表示什么？比如llama
+             "Text": decoded_text,  #
+             # "Bytes": token_bytes,  # bytes类型在gradio前端页面被解码成字符串，比如   b'\xe4\xb8\xad' 仍然显示成 "中"。因此 str(token_bytes)
+             "UTF8 Bytes": str(token_bytes),
+             # "Unicode": json_dumps  # unicode, 如果是ascii码，就直接显示。如果不是ascii码，就显示unicode
+             }
+        )
+
+    table_df = pd.DataFrame(table)
+    logger.info(f"tokenizer_type={tokenizer_name}, Tokens={table[:4]}")
+    return pos_tokens, len(encoding), table_df
+
+
+def tokenize(
+        text: str,
+        tokenizer_name: str,
+        color_num: int = 5,
+        add_special_token: bool = False
+):
+    """ tokenize wrapper
+    As gr.Update would be overwritten after passing to frontend, we apply lru_cache in _tokenize.
+    """
+    pos_tokens, num_tokens, table_df = _tokenize(text, tokenizer_name, color_num, add_special_token)
+    return gr.update(value=pos_tokens, label=f"Tokens: {num_tokens}"), table_df
+
+
+def tokenize_pair(text, tokenizer_type_1, tokenizer_type_2):
+    """
+    input_text.change
+    """
+    pos_tokens_1, table_df_1 = tokenize(text, tokenizer_type_1)
+    pos_tokens_2, table_df_2 = tokenize(text, tokenizer_type_2)
+    return pos_tokens_1, table_df_1, pos_tokens_2, table_df_2
+
+
+@lru_cache
+def basic_count(tokenizer_name):
+    stats = iter_vocab(tokenizer_name)
+    return stats['vocab_size'], f'{stats["organization"]}'
+    # return tokenizer.vocab_size, f'{stats["中文汉字数"]["中文单字"]}/{stats["中文汉字数"]["中文多字"]}'
+
+
+# def get_compress_rate(tokenizer_name, all_corpus, unit):
+#     tokenizer = tokenizer_factory.get_tokenizer(tokenizer_name)
+#     compress_rate_stats = tokenize_corpus(tokenizer, all_corpus)
+#     compress_rate = unit_convertor(compress_rate_stats, unit)
+#     return compress_rate
+
+
+@lru_cache
+def get_overlap_token_size(tokenizer_name_1, tokenizer_name_2):
+    tokenizer1 = tokenizer_factory.get_tokenizer(tokenizer_name_1)
+    tokenizer2 = tokenizer_factory.get_tokenizer(tokenizer_name_2)
+
+    vocab_set_1 = tokenizer1.get_vocab().keys()
+    vocab_set_2 = tokenizer2.get_vocab().keys()
+
+    token1 = next(iter(vocab_set_1))
+    token2 = next(iter(vocab_set_2))
+    if type(token1) != type(token2):  # bytes  str
+        if isinstance(token1, str):
+            vocab_set_1 = set([token.encode("utf-8") for token in vocab_set_1])
+        if isinstance(token2, str):
+            vocab_set_2 = set([token.encode("utf-8") for token in vocab_set_2])
+
+    overlap_tokens = vocab_set_1 & vocab_set_2
+    overlap_token_size = len(overlap_tokens)
+    logger.info(
+        f"{overlap_token_size} OverlapTokens of {tokenizer_name_1} {tokenizer_name_2}: {list(overlap_tokens)[:10]}")
+    return overlap_token_size, overlap_token_size
+
+
+def on_load(url_params, request: gr.Request):
+    """
+    onLoad
+    """
+    text = None
+    tokenizer_type_1 = None
+    tokenizer_type_2 = None
+    try:
+        url_params = json.loads(url_params)
+    except:
+        url_params = {}
+    if request:
+        logger.info(str(request.headers))
+        client_ip = request.client.host
+        # local_ip = socket.gethostbyname(socket.gethostbyname(""))
+        # headers = request.kwargs['headers']
+        # if headers and 'x-forwarded-for' in headers:
+        #     x_forwarded_for = headers['x-forwarded-for']
+        #     client_ip = x_forwarded_for.split(' ')[0] if x_forwarded_for else ""
+        # if "referer" in request.headers:   # not work for huggingface-space
+        #     url_params = parse_qs(urlparse(request.headers["referer"]).query)
+        #     url_params = {k: v[0] for k, v in url_params.items() if len(v) > 0}
+        tokenizer_type_1 = url_params.get("tokenizer1", default_tokenizer_name_1)
+        tokenizer_type_2 = url_params.get("tokenizer2", default_tokenizer_name_2)
+        text = url_params.get("text", default_user_input)
+        logger.info(f"client_ip: {client_ip}; params: {url_params}")
+    return text, tokenizer_type_1, tokenizer_type_2
+
+
+# def compress_rate_unit_change(unit):
+#     return gr.update(label=f"Compress Rate: {unit}"), gr.update(label=f"Compress Rate: {unit}"),
+
+
+def test_coding():
+    bytes1 = b'\xe4\xb8\xad'
+    print(bytes1)  # b'\xe4\xb8\xad'
+
+
+if __name__ == "__main__":
+    print(get_overlap_token_size("gpt-35-turbo", "gpt-4"))
+    # print(basic_count("internlm_chat_7b"))

requirements.txt

@@ -1,11 +1,12 @@
-transformers
-sentencepiece
-tiktoken
-icetk
-torch
-nltk
-boto3
-protobuf==4.25.3
-ai2-olmo==0.2.4
-ipadic
+gradio>=4.32.0
+transformers
+sentencepiece
+tiktoken
+icetk
+torch
+nltk
+boto3
+protobuf==4.25.3
+ai2-olmo==0.2.4
+ipadic
 fugashi

stats/compression_rate/ClassCat.gpt2-base-french @ cc100.ar.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bfb1c2be8bf13e5989a95b5f401f92aaad6cadde8ecc704ebaf9b9578bb359a2
+size 2145294

stats/compression_rate/ClassCat.gpt2-base-french @ cc100.de.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:554a365ce0da76ae5d93642b496bb1bc3d8d78c1112523545a2219f7fe213a91
+size 10978507

stats/compression_rate/ClassCat.gpt2-base-french @ cc100.en.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21c349b2602379affd0aa388d75addece67a14d0afaaf5b4980c90e9cc875e8e
+size 5261108

stats/compression_rate/ClassCat.gpt2-base-french @ cc100.es.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e829c4c544a9e8d98701b3d3bf1e3593b63e59ab5ba244c1ab376f6002fbd0f9
+size 6853004

stats/compression_rate/ClassCat.gpt2-base-french @ cc100.fa.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:908327a56262f721590d9479faa579156ba8bd155242262943797be697bc2655
+size 1058478

stats/compression_rate/ClassCat.gpt2-base-french @ cc100.fr.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f02e17dfe25c4c1526c8adee812a7141d92ccbd3b1160e7c73fc325d9fbfe4e
+size 6385085

stats/compression_rate/ClassCat.gpt2-base-french @ cc100.ja.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0abf3a91ddeeaa12d4732eaf1b4ff2a207b3d85fc54a079b4ac853696d831148
+size 2529096

stats/compression_rate/ClassCat.gpt2-base-french @ cc100.ko.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8fd64f035328b88bb4389ee820bb6d2bed510e0e4259cc4f38a0f573d2c003c2
+size 2491144

stats/compression_rate/ClassCat.gpt2-base-french @ cc100.zh-Hans.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7781b5bc9b2c3f45764842bf551a6e039ddef4f1bafd85ce12446834a26dd241
+size 10841058

stats/compression_rate/ClassCat.gpt2-base-spanish @ cc100.ar.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bfb1c2be8bf13e5989a95b5f401f92aaad6cadde8ecc704ebaf9b9578bb359a2
+size 2145294

stats/compression_rate/ClassCat.gpt2-base-spanish @ cc100.de.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:554a365ce0da76ae5d93642b496bb1bc3d8d78c1112523545a2219f7fe213a91
+size 10978507

stats/compression_rate/ClassCat.gpt2-base-spanish @ cc100.en.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21c349b2602379affd0aa388d75addece67a14d0afaaf5b4980c90e9cc875e8e
+size 5261108

stats/compression_rate/ClassCat.gpt2-base-spanish @ cc100.es.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e829c4c544a9e8d98701b3d3bf1e3593b63e59ab5ba244c1ab376f6002fbd0f9
+size 6853004

stats/compression_rate/ClassCat.gpt2-base-spanish @ cc100.fa.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:908327a56262f721590d9479faa579156ba8bd155242262943797be697bc2655
+size 1058478

stats/compression_rate/ClassCat.gpt2-base-spanish @ cc100.fr.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f02e17dfe25c4c1526c8adee812a7141d92ccbd3b1160e7c73fc325d9fbfe4e
+size 6385085

stats/compression_rate/ClassCat.gpt2-base-spanish @ cc100.ja.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0abf3a91ddeeaa12d4732eaf1b4ff2a207b3d85fc54a079b4ac853696d831148
+size 2529096

stats/compression_rate/ClassCat.gpt2-base-spanish @ cc100.ko.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8fd64f035328b88bb4389ee820bb6d2bed510e0e4259cc4f38a0f573d2c003c2
+size 2491144

stats/compression_rate/ClassCat.gpt2-base-spanish @ cc100.zh-Hans.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7781b5bc9b2c3f45764842bf551a6e039ddef4f1bafd85ce12446834a26dd241
+size 10841058

stats/compression_rate/ClueAI.ChatYuan-large-v2 @ cc100.ar.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b56af2e07e0c6ae80ed6c212d92a11eaad7dc654c187c7471738ba3c830a588
+size 20780798

stats/compression_rate/ClueAI.ChatYuan-large-v2 @ cc100.de.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:00492605965dd0637b79fe80e3d2428065cba551a9a7198bd7a0b505ce85d81b
+size 2751629

stats/compression_rate/ClueAI.ChatYuan-large-v2 @ cc100.en.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e823bebc4f8f42e03b8e621baa23b07072a4199eb0fd293e92d11c96003f3433
+size 163424

stats/compression_rate/ClueAI.ChatYuan-large-v2 @ cc100.es.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3003793b062ae28b5b4f202b8f0d9f725e46f024acc38f7f9ef08e8b3381fc0
+size 2030664

stats/compression_rate/ClueAI.ChatYuan-large-v2 @ cc100.fa.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c07f75c1eb80e59bab44b7b6ced9aec1404dbf56a5abd85779846c83974a7de
+size 18041636

stats/compression_rate/ClueAI.ChatYuan-large-v2 @ cc100.fr.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:341e747d07dd8276b90de8c7d725a45e10d39084bc819ffd54cab6460ddcba63
+size 3129632

stats/compression_rate/ClueAI.ChatYuan-large-v2 @ cc100.ja.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b46c604a75d0288f253c3439a2a7333c38e900ebb42ba39dd1c2ecbe4229f304
+size 6425383

stats/compression_rate/ClueAI.ChatYuan-large-v2 @ cc100.ko.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eeab167e9f566512c3065d362e720f1930bd51ca5b9c14c207a252fa9380e7fa
+size 15893128

stats/compression_rate/ClueAI.ChatYuan-large-v2 @ cc100.zh-Hans.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e851ffd1f4f9bf8949cb0e77cc15ea65223fe4a54ac5a13ec9e43c27a550388f
+size 10563259

stats/compression_rate/ClueAI.PromptCLUE-base @ cc100.ar.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b56af2e07e0c6ae80ed6c212d92a11eaad7dc654c187c7471738ba3c830a588
+size 20780798

stats/compression_rate/ClueAI.PromptCLUE-base @ cc100.de.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:00492605965dd0637b79fe80e3d2428065cba551a9a7198bd7a0b505ce85d81b
+size 2751629

stats/compression_rate/ClueAI.PromptCLUE-base @ cc100.en.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e823bebc4f8f42e03b8e621baa23b07072a4199eb0fd293e92d11c96003f3433
+size 163424

stats/compression_rate/ClueAI.PromptCLUE-base @ cc100.es.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3003793b062ae28b5b4f202b8f0d9f725e46f024acc38f7f9ef08e8b3381fc0
+size 2030664

stats/compression_rate/ClueAI.PromptCLUE-base @ cc100.fa.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c07f75c1eb80e59bab44b7b6ced9aec1404dbf56a5abd85779846c83974a7de
+size 18041636

stats/compression_rate/ClueAI.PromptCLUE-base @ cc100.fr.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:341e747d07dd8276b90de8c7d725a45e10d39084bc819ffd54cab6460ddcba63
+size 3129632

stats/compression_rate/ClueAI.PromptCLUE-base @ cc100.ja.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b46c604a75d0288f253c3439a2a7333c38e900ebb42ba39dd1c2ecbe4229f304
+size 6425383

stats/compression_rate/ClueAI.PromptCLUE-base @ cc100.ko.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eeab167e9f566512c3065d362e720f1930bd51ca5b9c14c207a252fa9380e7fa
+size 15893128

stats/compression_rate/ClueAI.PromptCLUE-base @ cc100.zh-Hans.diff.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e851ffd1f4f9bf8949cb0e77cc15ea65223fe4a54ac5a13ec9e43c27a550388f
+size 10563259