about tokenizer padding
#2
by
mmjwxbc
- opened
rna_tokens = rna_tokenizer(train_rnas, padding=True, truncation=True, return_tensors='pt', max_length=50)['input_ids']
why this instruction can not pad my token
The tokenizer auto pads all sequences to the longest sequence in a batch.max_length applies to truncation only.