JustinLin610
commited on
Commit
•
24d4292
1
Parent(s):
f7b9c28
Update README.md
Browse files
README.md
CHANGED
@@ -2,10 +2,18 @@
|
|
2 |
license: apache-2.0
|
3 |
---
|
4 |
|
5 |
-
# OFA-
|
6 |
This is the **medium** version of OFA pretrained model. OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework.
|
7 |
|
8 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
9 |
|
10 |
```
|
11 |
>>> from PIL import Image
|
@@ -29,6 +37,6 @@ To use it in Transformers, please refer to https://github.com/OFA-Sys/OFA/tree/f
|
|
29 |
>>> img = Image.open(path_to_image)
|
30 |
>>> patch_img = patch_resize_transform(img).unsqueeze(0)
|
31 |
|
32 |
-
>>> gen = model.generate(inputs,
|
33 |
-
>>> print(tokenizer.
|
34 |
```
|
|
|
2 |
license: apache-2.0
|
3 |
---
|
4 |
|
5 |
+
# OFA-tiny
|
6 |
This is the **medium** version of OFA pretrained model. OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework.
|
7 |
|
8 |
+
The directory includes 4 files, namely `config.json` which consists of model configuration, `vocab.json` and `merge.txt` for our OFA tokenizer, and lastly `pytorch_model.bin` which consists of model weights. There is no need to worry about the mismatch between Fairseq and transformers, since we have addressed the issue yet.
|
9 |
+
|
10 |
+
To use it in transformers, please refer to https://github.com/OFA-Sys/OFA/tree/feature/add_transformers. Install the transformers and download the models as shown below.
|
11 |
+
```
|
12 |
+
git clone --single-branch --branch feature/add_transformers https://github.com/OFA-Sys/OFA.git
|
13 |
+
pip install OFA/transformers/
|
14 |
+
it clone https://huggingface.co/OFA-Sys/OFA-medium
|
15 |
+
```
|
16 |
+
After, refer the path to OFA-medium to `ckpt_dir`, and prepare an image for the testing example below. Also, ensure that you have pillow and torchvision in your environment.
|
17 |
|
18 |
```
|
19 |
>>> from PIL import Image
|
|
|
37 |
>>> img = Image.open(path_to_image)
|
38 |
>>> patch_img = patch_resize_transform(img).unsqueeze(0)
|
39 |
|
40 |
+
>>> gen = model.generate(inputs, patch_images=patch_img, num_beams=4)
|
41 |
+
>>> print(tokenizer.batch_decode(gen, skip_special_tokens=True))
|
42 |
```
|