update tokenizer code for compatibility with latest transformers c936d25 verified katuni4ka commited on Mar 26
Make _decode compatible with PreTrainedTokenizerBase (#8) d4dc9dd rooa Vinno97 commited on Aug 18, 2023