eos token different from base model

#52
by metric-space - opened

Hey there folks

Maybe this is nothing but noticed the eos token
2024-05-23-134753_1913x385_scrot.png
in the tokenizer_config.json being different from the one in the base 70b model
I'm wondering is this is a bug?

Sign up or log in to comment