task_categories:visual-question-answering size_categories:10K<n<100K language:en license:cc-by-nc-4.0 region:us
---
license: cc-by-nc-4.0
task_categories:
- visual-question-answering
language:
- en
pretty_name: SEED-Bench
size_categories:
- 10K<n<100K
---



# SEED-Bench Card

## Benchmark details

**Benchmark type:**
SEED-Bench is a large-scale benchmark to evaluate Multimodal Large Language Models (MLLMs).
It consists of 19K multiple choice questions with accurate human annotations, which
covers 12 evaluation dimensions including the comprehension of both the image and video modality.


**Benchmark date:**
SEED-Bench was collected in July 2023.

**Paper or resources for more information:**
https://github.com/AILab-CVC/SEED-Bench

**License:**
Attribution-NonCommercial 4.0 International. It should abide by the policy of OpenAI: https://openai.com/policies/terms-of-use.

For the images of SEED-Bench, we use the data from Conceptual Captions Dataset (https://ai.google.com/research/ConceptualCaptions/)
following its license (https://github.com/google-research-datasets/conceptual-captions/blob/master/LICENSE).
Tencent does not hold the copyright for these images and the copyright belongs to the original owner of Conceptual Captions Dataset. 

For the videos of SEED-Bench, we use tha data from Something-Something v2 (https://developer.qualcomm.com/software/ai-datasets/something-something),
Epic-kitchen 100 (https://epic-kitchens.github.io/2023) and 
Breakfast (https://serre-lab.clps.brown.edu/resource/breakfast-actions-dataset/). We only provide the video name. Please download them in their official websites.

**Where to send questions or comments about the benchmark:**
https://github.com/AILab-CVC/SEED-Bench/issues

## Intended use
**Primary intended uses:**
The primary use of SEED-Bench is evaluate Multimodal Large Language Models on spatial and temporal understanding.

**Primary intended users:**
The primary intended users of the Benchmark are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.