Kyudan
/

TeXBLEU-Tokenizer

Model card Files Files and versions Community

YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

from huggingface_hub import snapshot_download

def load_tokenizer_from_hub(repo_name: str = "Kyudan/TeXBLEU-Tokenizer") -> Tokenizer: # Hugging Face Hub에서 토크나이저 다운로드 snapshot_dir = snapshot_download(repo_id=repo_name)

# 다운로드한 토크나이저 불러오기
    tokenizer = Tokenizer.from_file(f"{snapshot_dir}/tokenizer.json")
    return tokenizer

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference API

Unable to determine this model's library. Check the docs .