Update tokenizer

#11
by alvarobartt HF staff - opened
Hugging Quants org
β€’
edited Jul 28

Description

Updates both tokenizer.json and special_tokens_map.json as in https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct/commit/e04e3022cdc89bfed0db69f5ac1d249e21ee2d30

Closes #9

alvarobartt changed pull request title from Update special_tokens_map.json to Update tokenizer
alvarobartt changed pull request status to merged

Sign up or log in to comment