|
--- |
|
language: en |
|
datasets: |
|
- laion2b |
|
--- |
|
|
|
# OpenFlamingo-3B (CLIP ViT-L/14, MPT-1B-Dolly) |
|
|
|
[Blog post]() | [Code](https://github.com/mlfoundations/open_flamingo) | [Demo]() |
|
|
|
OpenFlamingo is an open source implementation of DeepMind's [Flamingo](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model) models. |
|
This 3B-parameter model uses a [CLIP ViT-L/14](https://huggingface.co/openai/clip-vit-large-patch14) vision encoder and an instruction-tuned [MPT-1B](https://huggingface.co/mosaicml/mpt-1b-redpajama-200b-dolly) language model. |
|
|
|
## Model Details |
|
We follow the Flamingo modeling paradigm, outfitting the layers of a pretrained, frozen language model such that they cross-attend to visual features when decoding. Following Flamingo, we freeze the vision encoder and language model but train the connecting modules on web-scraped image-text sequences. Specifically, we trained this model on a mixture of [LAION-2B](https://arxiv.org/abs/2210.08402) and [Multimodal C4](https://arxiv.org/abs/2304.06939). |
|
|
|
This model has cross-attention modules inserted in *every* decoder block. It was trained using DistributedDataParallel across 64 A100 40GB GPUs at FP32 precision. |
|
|
|
The [MPT-1B](https://huggingface.co/mosaicml/mpt-1b-redpajama-200b-dolly) modeling code does not accept the `labels` kwarg and compute cross-entropy loss within `forward()`. To train with the OpenFlamingo codebase, we suggest using a version with the `labels` kwarg [here](https://huggingface.co/anas-awadalla/mpt-1b-redpajama-200b-dolly). |
|
|
|
## Uses |
|
OpenFlamingo models process arbitrarily interleaved sequences of images and text to output text. This allows the models to accept in-context examples and undertake tasks like captioning, visual question answering, and image classification. |
|
### Initialization |
|
|
|
``` python |
|
from open_flamingo import create_model_and_transforms |
|
|
|
model, image_processor, tokenizer = create_model_and_transforms( |
|
clip_vision_encoder_path="ViT-L-14", |
|
clip_vision_encoder_pretrained="openai", |
|
lang_encoder_path="anas-awadalla/mpt-1b-redpajama-200b-dolly", |
|
tokenizer_path="anas-awadalla/mpt-1b-redpajama-200b-dolly", |
|
cross_attn_every_n_layers=1 |
|
) |
|
|
|
# grab model checkpoint from huggingface hub |
|
from huggingface_hub import hf_hub_download |
|
import torch |
|
|
|
checkpoint_path = hf_hub_download("openflamingo/OpenFlamingo-3B-vitl-mpt1b-langinstruct", "checkpoint.pt") |
|
model.load_state_dict(torch.load(checkpoint_path), strict=False) |
|
``` |
|
|
|
### Generation example |
|
Below is an example of generating text conditioned on interleaved images/text. In particular, let's try few-shot image captioning. |
|
|
|
``` python |
|
from PIL import Image |
|
import requests |
|
|
|
""" |
|
Step 1: Load images |
|
""" |
|
demo_image_one = Image.open( |
|
requests.get( |
|
"http://images.cocodataset.org/val2017/000000039769.jpg", stream=True |
|
).raw |
|
) |
|
|
|
demo_image_two = Image.open( |
|
requests.get( |
|
"http://images.cocodataset.org/test-stuff2017/000000028137.jpg", |
|
stream=True |
|
).raw |
|
) |
|
|
|
query_image = Image.open( |
|
requests.get( |
|
"http://images.cocodataset.org/test-stuff2017/000000028352.jpg", |
|
stream=True |
|
).raw |
|
) |
|
|
|
|
|
""" |
|
Step 2: Preprocessing images |
|
Details: For OpenFlamingo, we expect the image to be a torch tensor of shape |
|
batch_size x num_media x num_frames x channels x height x width. |
|
In this case batch_size = 1, num_media = 3, num_frames = 1, |
|
channels = 3, height = 224, width = 224. |
|
""" |
|
vision_x = [image_processor(demo_image_one).unsqueeze(0), image_processor(demo_image_two).unsqueeze(0), image_processor(query_image).unsqueeze(0)] |
|
vision_x = torch.cat(vision_x, dim=0) |
|
vision_x = vision_x.unsqueeze(1).unsqueeze(0) |
|
|
|
""" |
|
Step 3: Preprocessing text |
|
Details: In the text we expect an <image> special token to indicate where an image is. |
|
We also expect an <|endofchunk|> special token to indicate the end of the text |
|
portion associated with an image. |
|
""" |
|
tokenizer.padding_side = "left" # For generation padding tokens should be on the left |
|
lang_x = tokenizer( |
|
["<image>An image of two cats.<|endofchunk|><image>An image of a bathroom sink.<|endofchunk|><image>An image of"], |
|
return_tensors="pt", |
|
) |
|
|
|
|
|
""" |
|
Step 4: Generate text |
|
""" |
|
generated_text = model.generate( |
|
vision_x=vision_x, |
|
lang_x=lang_x["input_ids"], |
|
attention_mask=lang_x["attention_mask"], |
|
max_new_tokens=20, |
|
num_beams=3, |
|
) |
|
|
|
print("Generated text: ", tokenizer.decode(generated_text[0])) |
|
``` |
|
|
|
### Bias, Risks, and Limitations |
|
OpenFlamingo models inherit the risks of their parent models, especially the language model. As an open-source research effort, we highly value open, accessible, reproducible multimodal model research; however, it is crucial to be aware that these models are trained on web data, have not been finetuned for safety, and thus may produce unintended, inappropriate, unreliable, and/or inaccurate outputs. Please use caution before deploying OpenFlamingo models in real applications. We also hope that OpenFlamingo enables further safety and reliability research to address these issues. |
|
|
|
In an effort to mitigate current potential biases and harms, we have deployed a text content filter on model outputs in the OpenFlamingo demo. We continue to red-team the model to understand and improve its safety. |
|
|
|
## Evaluation |
|
<table> |
|
<tr> |
|
<th></th> |
|
<th>0-shot</th> |
|
<th>4-shot</th> |
|
<th>8-shot</th> |
|
<th>16-shot</th> |
|
<th>32-shot</th> |
|
</tr> |
|
<tr> |
|
<th>COCO (CIDEr)</th> |
|
<td>74.4 (0.6)</td> |
|
<td>82.7 (0.7)</td> |
|
<td>87.8 (0.5)</td> |
|
<td>91.9 (0.3)</td> |
|
<td>94.8 (0.3)</td> |
|
</tr> |
|
<tr> |
|
<th>VQAv2 (Accuracy)</th> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
</tr> |
|
<tr> |
|
<th>Flickr-30K (CIDEr)</th> |
|
<td>51.2 (0.2)</td> |
|
<td>59.1 (0.3)</td> |
|
<td>60.7 (0.6)</td> |
|
<td>63.0 (0.4)</td> |
|
<td>64.5 (1.3)</td> |
|
</tr> |
|
<tr> |
|
<th>OK-VQA (Accuracy)</th> |
|
<td>26.2 (0.3)</td> |
|
<td>31.9 (0.2)</td> |
|
<td>31.4 (0.4)</td> |
|
<td>31.6 (0.3)</td> |
|
<td>31.0 (0.1)</td> |
|
</tr> |
|
<tr> |
|
<th>TextVQA (Accuracy)</th> |
|
<td>23.1 (0.2)</td> |
|
<td>28.1 (0.4)</td> |
|
<td>29.1 (0.1)</td> |
|
<td>29.1 (0.1)</td> |
|
<td>28.5 (0.1)</td> |
|
</tr> |
|
<tr> |
|
<th>Vizwiz (Accuracy)</th> |
|
<td>18.0 (0.6)</td> |
|
<td>22.0 (0.8)</td> |
|
<td>28.8 (1.3)</td> |
|
<td>35.5 (0.8)</td> |
|
<td>41.3 (0.5)</td> |
|
</tr> |
|
<tr> |
|
<th>Hateful Memes (ROC AUC)</th> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
</tr> |
|
</table> |
|
|
