JustinLin610 commited on
Commit
2f13d6c
1 Parent(s): 63b2141

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +61 -0
README.md ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ ---
4
+
5
+ # OFA-large-caption
6
+ This is the **large** version of OFA model finetuned for **image captioning**. OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework.
7
+
8
+ The directory includes 4 files, namely `config.json` which consists of model configuration, `vocab.json` and `merge.txt` for our OFA tokenizer, and lastly `pytorch_model.bin` which consists of model weights. There is no need to worry about the mismatch between Fairseq and transformers, since we have addressed the issue yet.
9
+
10
+ To use it in transformers, please refer to https://github.com/OFA-Sys/OFA/tree/feature/add_transformers. Install the transformers and download the models as shown below.
11
+ ```
12
+ git clone --single-branch --branch feature/add_transformers https://github.com/OFA-Sys/OFA.git
13
+ pip install OFA/transformers/
14
+ git clone https://huggingface.co/OFA-Sys/OFA-large-caption
15
+ ```
16
+ After, refer the path to OFA-large to `ckpt_dir`, and prepare an image for the testing example below. Also, ensure that you have pillow and torchvision in your environment.
17
+
18
+ ```
19
+ >>> from PIL import Image
20
+ >>> from torchvision import transforms
21
+ >>> from transformers import OFATokenizer, OFAModel
22
+ >>> from generate import sequence_generator
23
+
24
+ >>> mean, std = [0.5, 0.5, 0.5], [0.5, 0.5, 0.5]
25
+ >>> resolution = 480
26
+ >>> patch_resize_transform = transforms.Compose([
27
+ lambda image: image.convert("RGB"),
28
+ transforms.Resize((resolution, resolution), interpolation=Image.BICUBIC),
29
+ transforms.ToTensor(),
30
+ transforms.Normalize(mean=mean, std=std)
31
+ ])
32
+
33
+
34
+ >>> tokenizer = OFATokenizer.from_pretrained(ckpt_dir)
35
+
36
+ >>> txt = " what does the image describe?"
37
+ >>> inputs = tokenizer([txt], return_tensors="pt").input_ids
38
+ >>> img = Image.open(path_to_image)
39
+ >>> patch_img = patch_resize_transform(img).unsqueeze(0)
40
+
41
+
42
+ >>> # using the generator of fairseq version
43
+ >>> model = OFAModel.from_pretrained(ckpt_dir, use_cache=True)
44
+ >>> generator = sequence_generator.SequenceGenerator(
45
+ tokenizer=tokenizer,
46
+ beam_size=5,
47
+ max_len_b=16,
48
+ min_len=0,
49
+ no_repeat_ngram_size=3,
50
+ )
51
+ >>> data = {}
52
+ >>> data["net_input"] = {"input_ids": inputs, 'patch_images': patch_img, 'patch_masks':torch.tensor([True])}
53
+ >>> gen_output = generator.generate([model], data)
54
+ >>> gen = [gen_output[i][0]["tokens"] for i in range(len(gen_output))]
55
+
56
+ >>> # using the generator of huggingface version
57
+ >>> model = OFAModel.from_pretrained(ckpt_dir, use_cache=False)
58
+ >>> gen = model.generate(inputs, patch_images=patch_img, num_beams=5, no_repeat_ngram_size=3)
59
+
60
+ >>> print(tokenizer.batch_decode(gen, skip_special_tokens=True))
61
+ ```