Spaces:

xu-song
/

tokenizer-arena

Running

add more tokenizers

f4973d4 12 months ago

136 Bytes




	from transformers import AutoTokenizer
	tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", trust_remote_code=True)