Mantis
Collection
Mantis model family optimized for multi-image reasoning with interleaved text/image format
β’
11 items
β’
Updated
β’
8
Paper | Website | Github | Models | Demo | Wandb
Excited to announce Mantis-Idefics2, with enhanced ability in multi-image scenarios! It's fine-tuned on Mantis-Instruct from Idefics2-8b
Models | Size | Format | NLVR2 | Q-Bench | Mantis-Eval | BLINK | MVBench | Avg |
---|---|---|---|---|---|---|---|---|
GPT-4V | - | sequence | 88.80 | 76.52 | 62.67 | 51.14 | 43.50 | 64.5 |
Open Source Models | ||||||||
Random | - | - | 48.93 | 40.20 | 23.04 | 38.09 | 27.30 | 35.5 |
Kosmos2 | 1.6B | merge | 49.00 | 35.10 | 30.41 | 37.50 | 21.62 | 34.7 |
LLaVA-v1.5 | 7B | merge | 53.88 | 49.32 | 31.34 | 37.13 | 36.00 | 41.5 |
LLava-V1.6 | 7B | merge | 58.88 | 54.80 | 45.62 | 39.55 | 40.90 | 48.0 |
Qwen-VL-Chat | 7B | merge | 58.72 | 45.90 | 39.17 | 31.17 | 42.15 | 43.4 |
Fuyu | 8B | merge | 51.10 | 49.15 | 27.19 | 36.59 | 30.20 | 38.8 |
BLIP-2 | 13B | merge | 59.42 | 51.20 | 49.77 | 39.45 | 31.40 | 46.2 |
InstructBLIP | 13B | merge | 60.26 | 44.30 | 45.62 | 42.24 | 32.50 | 45.0 |
CogVLM | 17B | merge | 58.58 | 53.20 | 45.16 | 41.54 | 37.30 | 47.2 |
OpenFlamingo | 9B | sequence | 36.41 | 19.60 | 12.44 | 39.18 | 7.90 | 23.1 |
Otter-Image | 9B | sequence | 49.15 | 17.50 | 14.29 | 36.26 | 15.30 | 26.5 |
Idefics1 | 9B | sequence | 54.63 | 30.60 | 28.11 | 24.69 | 26.42 | 32.9 |
VideoLLaVA | 7B | sequence | 56.48 | 45.70 | 35.94 | 38.92 | 44.30 | 44.3 |
Emu2-Chat | 37B | sequence | 58.16 | 50.05 | 37.79 | 36.20 | 39.72 | 44.4 |
Vila | 8B | sequence | 76.45 | 45.70 | 51.15 | 39.30 | 49.40 | 52.4 |
Idefics2 | 8B | sequence | 86.87 | 57.00 | 48.85 | 45.18 | 29.68 | 53.5 |
Mantis-CLIP | 8B | sequence | 84.66 | 66.00 | 55.76 | 47.06 | 48.30 | 60.4 |
Mantis-SIGLIP | 8B | sequence | 87.43 | 69.90 | 59.45 | 46.35 | 50.15 | 62.7 |
Mantis-Flamingo | 9B | sequence | 52.96 | 46.80 | 32.72 | 38.00 | 40.83 | 42.3 |
Mantis-Idefics2 | 8B | sequence | 89.71 | 75.20 | 57.14 | 49.05 | 51.38 | 64.5 |
$\Delta$ over SOTA | - | - | +2.84 | +18.20 | +8.30 | +3.87 | +1.98 | +11.0 |
Model | Size | TextVQA | VQA | MMB | MMMU | OKVQA | SQA | MathVista | Avg |
---|---|---|---|---|---|---|---|---|---|
OpenFlamingo | 9B | 46.3 | 58.0 | 32.4 | 28.7 | 51.4 | 45.7 | 18.6 | 40.2 |
Idefics1 | 9B | 39.3 | 68.8 | 45.3 | 32.5 | 50.4 | 51.6 | 21.1 | 44.1 |
InstructBLIP | 7B | 33.6 | 75.2 | 38.3 | 30.6 | 45.2 | 70.6 | 24.4 | 45.4 |
Yi-VL | 6B | 44.8 | 72.5 | 68.4 | 39.1 | 51.3 | 71.7 | 29.7 | 53.9 |
Qwen-VL-Chat | 7B | 63.8 | 78.2 | 61.8 | 35.9 | 56.6 | 68.2 | 15.5 | 54.3 |
LLaVA-1.5 | 7B | 58.2 | 76.6 | 64.8 | 35.3 | 53.4 | 70.4 | 25.6 | 54.9 |
Emu2-Chat | 37B | 66.6 | 84.9 | 63.6 | 36.3 | 64.8 | 65.3 | 30.7 | 58.9 |
CogVLM | 17B | 70.4 | 82.3 | 65.8 | 32.1 | 64.8 | 65.6 | 35.0 | 59.4 |
Idefics2 | 8B | 70.4 | 79.1 | 75.7 | 43.0 | 53.5 | 86.5 | 51.4 | 65.7 |
Mantis-CLIP | 8B | 56.4 | 73.0 | 66.0 | 38.1 | 53.0 | 73.8 | 31.7 | 56.0 |
Mantis-SigLIP | 8B | 59.2 | 74.9 | 68.7 | 40.1 | 55.4 | 74.9 | 34.4 | 58.2 |
Mantis-Idefics2 | 8B | 63.5 | 77.6 | 75.7 | 41.1 | 52.6 | 81.3 | 40.4 | 61.7 |
import requests
import torch
from PIL import Image
from io import BytesIO
from transformers import AutoProcessor, AutoModelForVision2Seq
from transformers.image_utils import load_image
processor = AutoProcessor.from_pretrained("TIGER-Lab/Mantis-8B-Idefics2") # do_image_splitting is False by default
model = AutoModelForVision2Seq.from_pretrained(
"TIGER-Lab/Mantis-8B-Idefics2",
device_map="auto"
)
generation_kwargs = {
"max_new_tokens": 1024,
"num_beams": 1,
"do_sample": False
}
# Note that passing the image urls (instead of the actual pil images) to the processor is also possible
image1 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg")
image2 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")
image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg")
images = [image1, image2, image3]
query1 = "What cities image 1, image 2, and image 3 belong to respectively? Answer me in order."
query2 = "Which one do you recommend for a visit? and why?"
query3 = "Which picture has most cars in it?"
### Chat
### Round 1
messages = [
{
"role": "user",
"content": [
{"type": "image"},
{"type": "image"},
{"type": "image"},
{"type": "text", "text": query1},
]
}
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=images, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
# Generate
generated_ids = model.generate(**inputs, **generation_kwargs)
response = processor.batch_decode(generated_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print("User: ", query1)
print("ASSISTANT: ", response[0])
### Round 2
messages.append(
{
"role": "assistant",
"content": [
{"type": "text", "text": response[0]},
]
}
)
messages.append(
{
"role": "user",
"content": [
{"type": "text", "text": query2},
]
}
)
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=images, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
generated_ids = model.generate(**inputs, **generation_kwargs)
response = processor.batch_decode(generated_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print("User: ", query2)
print("ASSISTANT: ", response[0])
### Round 3
messages.append(
{
"role": "assistant",
"content": [
{"type": "text", "text": response[0]},
]
}
)
messages.append(
{
"role": "user",
"content": [
{"type": "text", "text": query3},
]
}
)
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=images, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
generated_ids = model.generate(**inputs, **generation_kwargs)
response = processor.batch_decode(generated_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print("User: ", query3)
print("ASSISTANT: ", response[0])
"""
User: What cities image 1, image 2, and image 3 belong to respectively? Answer me in order.
ASSISTANT: Chicago, New York, San Francisco
User: Which one do you recommend for a visit? and why?
ASSISTANT: New York - because it's a bustling metropolis with iconic landmarks like the Statue of Liberty and the Empire State Building.
User: Which picture has most cars in it?
ASSISTANT: Image 3
"""
See mantis/train for details
See mantis/benchmark for details
Please cite our paper or give a star to out Github repo if you find this model useful
@inproceedings{Jiang2024MANTISIM,
title={MANTIS: Interleaved Multi-Image Instruction Tuning},
author={Dongfu Jiang and Xuan He and Huaye Zeng and Cong Wei and Max W.F. Ku and Qian Liu and Wenhu Chen},
publisher={arXiv2405.01483}
year={2024},
}