JustinLin610
commited on
Commit
•
fe49e57
1
Parent(s):
2f13d6c
Update README.md
Browse files
README.md
CHANGED
@@ -3,10 +3,14 @@ license: apache-2.0
|
|
3 |
---
|
4 |
|
5 |
# OFA-large-caption
|
|
|
|
|
6 |
This is the **large** version of OFA model finetuned for **image captioning**. OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework.
|
7 |
|
8 |
The directory includes 4 files, namely `config.json` which consists of model configuration, `vocab.json` and `merge.txt` for our OFA tokenizer, and lastly `pytorch_model.bin` which consists of model weights. There is no need to worry about the mismatch between Fairseq and transformers, since we have addressed the issue yet.
|
9 |
|
|
|
|
|
10 |
To use it in transformers, please refer to https://github.com/OFA-Sys/OFA/tree/feature/add_transformers. Install the transformers and download the models as shown below.
|
11 |
```
|
12 |
git clone --single-branch --branch feature/add_transformers https://github.com/OFA-Sys/OFA.git
|
@@ -15,7 +19,7 @@ git clone https://huggingface.co/OFA-Sys/OFA-large-caption
|
|
15 |
```
|
16 |
After, refer the path to OFA-large to `ckpt_dir`, and prepare an image for the testing example below. Also, ensure that you have pillow and torchvision in your environment.
|
17 |
|
18 |
-
```
|
19 |
>>> from PIL import Image
|
20 |
>>> from torchvision import transforms
|
21 |
>>> from transformers import OFATokenizer, OFAModel
|
|
|
3 |
---
|
4 |
|
5 |
# OFA-large-caption
|
6 |
+
|
7 |
+
## Introduction
|
8 |
This is the **large** version of OFA model finetuned for **image captioning**. OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework.
|
9 |
|
10 |
The directory includes 4 files, namely `config.json` which consists of model configuration, `vocab.json` and `merge.txt` for our OFA tokenizer, and lastly `pytorch_model.bin` which consists of model weights. There is no need to worry about the mismatch between Fairseq and transformers, since we have addressed the issue yet.
|
11 |
|
12 |
+
|
13 |
+
## How to use
|
14 |
To use it in transformers, please refer to https://github.com/OFA-Sys/OFA/tree/feature/add_transformers. Install the transformers and download the models as shown below.
|
15 |
```
|
16 |
git clone --single-branch --branch feature/add_transformers https://github.com/OFA-Sys/OFA.git
|
|
|
19 |
```
|
20 |
After, refer the path to OFA-large to `ckpt_dir`, and prepare an image for the testing example below. Also, ensure that you have pillow and torchvision in your environment.
|
21 |
|
22 |
+
```python
|
23 |
>>> from PIL import Image
|
24 |
>>> from torchvision import transforms
|
25 |
>>> from transformers import OFATokenizer, OFAModel
|