Use the right tokenizer
#10
by
Nick088
- opened
As its a finetune of the Google Flan T5 Small model, the tokenizer its the same, so there isn't really a reason to use the base model repo tokenizer when you can just use the one of your own model as its like a duplicate one
Good catch, ty!
roborovski
changed pull request status to
merged