Irena Gao
commited on
Commit
•
daeaf8e
1
Parent(s):
dfb2f2c
update README
Browse files
README.md
CHANGED
@@ -20,6 +20,26 @@ The [MPT-1B](https://huggingface.co/mosaicml/mpt-1b-redpajama-200b-dolly) modeli
|
|
20 |
|
21 |
## Uses
|
22 |
OpenFlamingo models process arbitrarily interleaved sequences of images and text to output text. This allows the models to accept in-context examples and undertake tasks like captioning, visual question answering, and image classification.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
23 |
|
24 |
### Generation example
|
25 |
Below is an example of generating text conditioned on interleaved images/text. In particular, let's try few-shot image captioning.
|
|
|
20 |
|
21 |
## Uses
|
22 |
OpenFlamingo models process arbitrarily interleaved sequences of images and text to output text. This allows the models to accept in-context examples and undertake tasks like captioning, visual question answering, and image classification.
|
23 |
+
### Initialization
|
24 |
+
|
25 |
+
``` python
|
26 |
+
from open_flamingo import create_model_and_transforms
|
27 |
+
|
28 |
+
model, image_processor, tokenizer = create_model_and_transforms(
|
29 |
+
clip_vision_encoder_path="ViT-L-14",
|
30 |
+
clip_vision_encoder_pretrained="openai",
|
31 |
+
lang_encoder_path="anas-awadalla/mpt-1b-redpajama-200b-dolly",
|
32 |
+
tokenizer_path="anas-awadalla/mpt-1b-redpajama-200b-dolly",
|
33 |
+
cross_attn_every_n_layers=1
|
34 |
+
)
|
35 |
+
|
36 |
+
# grab model checkpoint from huggingface hub
|
37 |
+
from huggingface_hub import hf_hub_download
|
38 |
+
import torch
|
39 |
+
|
40 |
+
checkpoint_path = hf_hub_download("openflamingo/OpenFlamingo-3B-vitl-mpt1b-langinstruct", "checkpoint.pt")
|
41 |
+
model.load_state_dict(torch.load(checkpoint_path), strict=False)
|
42 |
+
```
|
43 |
|
44 |
### Generation example
|
45 |
Below is an example of generating text conditioned on interleaved images/text. In particular, let's try few-shot image captioning.
|