uses '<|eot_id|>' instead of '<|end_of_text|>'
#2
by
fbirlik
- opened
uses '<|eot_id|>' instead of '<|end_of_text|>' unlike default llama 3 causing '<|eot_id|>' to appear in the end of responses.
Updating 'tokenizer_config.json' as follows would fix the issue.
...
"eos_token": "<|eot_id|>"
...
I also checked individual merged fine tunes, but I couldn't pinpoint where this comes from.