constants.py

import os

HF_TOKEN = os.environ.get("HF_TOKEN")

LEADERBOARD_INTRODUCTION = """# MEGA-Bench Leaderboard

## 🚀 Introduction

[MEGA-Bench](https://tiger-ai-lab.github.io/MEGA-Bench/) is a comprehensive benchmark scaling multimodal evaluation to 500+ real-world tasks! 

We aim to provide cost-effective and accurate evaluation for multimodal models, covering a wide range of real-world tasks. You don't have to run models on dozens of benchmarks -- MEGA-Bench delivers a comprehensive performance report in a single benchmark.

## 🧐 Highlights of MEGA-Bench

- 505 diverse tasks evaluating multimodal models across 8 grand application types, 7 input visual formats, 6 output formats, and 10 general multimodal skills, covering single-image, multi-image, and video tasks
- Moves beyond multiple-choice questions, offering diverse output formats like numbers, code, LATEX, phrases, free-form responses, and more. We developed 45 customized metrics to accurately evaluate these diverse outputs
- Focuses on task diversity rather than repetitive examples, ensuring cost-efficient evaluation
- Provides fine-grained capability reports across application type, input/output formats, and required skills


## 🔨 Systematic Annotation Process

- Guided by an initial application-driven taxonomy tree
- 16 expert annotators contributing to a 2-round process to develop 505 tasks
- Utilizes advanced tools for task design, review, and quality control
- Ensures high-quality data through continuous refinement and balanced task distribution


## 📊🔍 Results & Takeaways from Evaluating Top Models

- GPT4o leads the benchmark, outperforming others by 3.5% over Claude3.5
- Qwen2VL stands out among open-source models, nearing flagship-level performance
- Chain-of-Thought (CoT) improves proprietary models but has limited impact on open-source models
- Efficiency models like Gemini 1.5 Flash perform well but struggle with UI and document tasks
- Many open-source models face challenges in adhering to output format instructions

## 🎯 Interactive Visualization 

Visit our [project page](https://tiger-ai-lab.github.io/MEGA-Bench/) to explore the interactive task taxonomy and radar maps, offering deep insights into model capabilities across multiple dimensions. Discover a comprehensive breakdown far beyond single-score evaluations.


## 📚 More Information

- Our evaluation pipeline will soon be available on our GitHub: https://github.com/TIGER-AI-Lab/MEGA-Bench.
- Check full details of our paper at [https://arxiv.org/abs/2410.10563](https://arxiv.org/abs/2410.10563)
- Hugging Face Datasets: [https://huggingface.co/datasets/TIGER-Lab/MEGA-Bench](https://huggingface.co/datasets/TIGER-Lab/MEGA-Bench)
- GitHub: [https://github.com/TIGER-AI-Lab/MEGA-Bench](https://github.com/TIGER-AI-Lab/MEGA-Bench)

"""

TABLE_INTRODUCTION = """
"""

DATA_INFO = """
### Data Sources
The data source of MEGA-Bench tasks have three main types:
- **Purely Self-designed:** The task is designed entirely by the annotator, and the annotator looks for the image or video resources from the Internet or even using code/simulator.
- **Inspired and adapted from existing benchmarks:** The task is inspired by existing benchmarks or datasets. The annotator collects the raw image/video data from existing datasets but does not use the original annotation. The annotator redesigns/repurposes the data by writing concrete task descriptions and creating new questions and answers, or using scripts to re-process the data for the designed task.  
- **Directly converted from existing benchmarks:** The task is directly converted from existing benchmarks or datasets. The annotator randomly samples a subset from the existing benchmark, directly using its image/video and the annotation without redesign.

In our annotation process, the first two task types are encouraged. The task reviewers strictly control the number of the third type and reject the task if an annotator submits many tasks of the third type.

Please refer to Table 17 of our [paper](https://arxiv.org/abs/2410.10563) for the detailed data source of all tasks in MEGA-Bench.
"""

CITATION_BUTTON_LABEL = "Copy the following snippet to cite our paper and evaluation results below"
CITATION_BUTTON_TEXT = r"""
@article{chen2024mega-bench,
        title={MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks},
        author={Chen, Jiacheng and Liang, Tianhao and Siu, Sherman and Wang, Zhengqing and Wang, Kai and Wang, Yubo and Ni, Yuansheng and Zhu, Wang and Jiang, Ziyan and Lyu, Bohan and Jiang, Dongfu and He, Xuan and Liu, Yuan and Hu, Hexiang and Yue, Xiang and Chen, Wenhu},
        journal={arXiv preprint arXiv:2410.10563},
        year={2024},
}
"""

SUBMIT_INTRODUCTION = """# Submit on MEGA-Bench Leaderboard

We will provide details on how to submit third-party evaluation results once our evaluation pipeline is released on our [GitHub repository](https://github.com/TIGER-AI-Lab/MEGA-Bench).


"""