What are the start and stop tokens of this model?
#40
by
aryaash
- opened
Hi. I would like to know what are the start and stop tokens of this model. I'm unable to find it in any of the config files.
@aryaash
that info is in the tokenizer_config.json file. The start (bos) token is <|begin_of_text|> and the stop (eos) token is <|end_of_text|>
"bos_token": "<|begin_of_text|>",
"eos_token": "<|end_of_text|>",
https://huggingface.co/meta-llama/Llama-3.2-1B/blob/main/tokenizer_config.json