Vocab size of tokenizer is not equal to vocab size of the model
#2
by
dinhanhx
- opened
I notice that in config.json
, the vocab size of the model is 256512. However, the vocab size of the tokenizer is 256000.
dinhanhx
changed discussion title from
Vocab size is 256512 or 25600?
to Vocab size of tokenizer is not equal to vocab size of the model
jbochi
changed discussion status to
closed
Ah I see thanks for clarifying