How to use the tokenizer? Is the tokenizer same as the original CLIP?

#2
by ir1d - opened

CLIPTokenizer.from_pretrained('laion/CLIP-ViT-B-32-laion2B-s34B-b79K') generates error

@ir1d

CLIPTokenizer.from_pretrained('laion/CLIP-ViT-B-32-laion2B-s34B-b79K') generates error

Use:

tokenizer = CLIPProcessor.from_pretrained('laion/CLIP-ViT-B-32-laion2B-s34B-b79K').tokenizer

I actually find the encoded input_ids is the same as "openai/clip-vit-base-patch32", and both tokenizers has same vocal. But there's no way to ensure.

Sign up or log in to comment