|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- laion/laion400m |
|
- kakaobrain/coyo-700m |
|
pipeline_tag: feature-extraction |
|
tags: |
|
- Vision |
|
- LLaVA |
|
--- |
|
|
|
|
|
|
|
|
|
[[Paper]](https://arxiv.org/abs/2407.17331) [[GitHub]](https://github.com/deepglint/unicom) |
|
## Model |
|
We used the same Vision Transformer architecture [ViT-L/14@336px as CLIP](https://huggingface.co/openai/clip-vit-large-patch14-336). |
|
|
|
 |
|
|
|
|
|
## Data |
|
Our model was trained on publicly available image-caption data from the [LAION400M](https://arxiv.org/abs/2111.02114) and [COYO700M](https://github.com/kakaobrain/coyo-dataset) datasets. |
|
|
|
## Performance and Limitations |
|
|
|
### A. MLLMs Evaluation Results |
|
In our experiments, we replaced the CLIP model in [LLaVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT) with the MLCD model to demonstrate the performance of the MLCD model in Multimodal Large Language Models (MLLMs). For the language model, we used [Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B). The evaluation results show that the modified model performs exceptionally well across multiple benchmarks, validating the effectiveness of the MLCD model within MLLMs. |
|
|
|
| Vision Tower | MLCD (ViT_L_14_336px) | CLIP (ViT_L_14_336px) | |
|
|:----------------|:----------------------|:----------------------| |
|
| LLM | Qwen2.5-7B | Qwen2.5-7B | |
|
| AI2D | <span style="color:red">76.98</span> | 73.15 | |
|
| ScienceQA_img | <span style="color:red">78.09</span> | 76.35 | |
|
| GQA | <span style="color:red">64.17</span> | 63.31 | |
|
| InfoVQA_val | <span style="color:red">43.48</span> | 38.88 | |
|
| MMBench_cn_dev | <span style="color:red">74.83</span> | 72.51 | |
|
| MMBench_en_dev | <span style="color:red">76.37</span> | 74.57 | |
|
| MME(cognition) | <span style="color:red">432</span> | 384 | |
|
| MME(perception) | <span style="color:red">1598</span> | 1512 | |
|
| SeedBench | <span style="color:red">68.20</span> | 66.80 | |
|
| SeedBench_img | <span style="color:red">73.75</span> | 72.72 | |
|
| MMStar | <span style="color:red">50.98</span> | 48.98 | |
|
| MMMU | <span style="color:red">44.30</span> | 44.20 | |
|
| OCRBench | <span style="color:red">531.00</span> | 525.00 | |
|
| ChartQA | <span style="color:red">67.84</span> | 66.52 | |
|
| DocVQA_val | <span style="color:red">76.46</span> | 75.21 | |
|
| POPE | 88.69 | <span style="color:red">88.83</span> | |
|
| TextVQA_val | 61.69 | <span style="color:red">62.47</span> | |
|
|
|
|
|
|
|
|
|
### B. Linear Probe Evaluation Results |
|
This table presents the results of linear probe evaluations comparing CLIP and MLCD models on the ViT_L_14_336px architecture across various datasets. The linear probe test freezes the pre-trained model's weights and trains a linear classifier on top to assess how well the model's representations generalize to different tasks. |
|
|
|
| Dataset | MLCD (ViT_L_14_336px) | CLIP (ViT_L_14_336px) | |
|
|:---------------|:----------------------|:----------------------| |
|
| AVG | <span style="color:red">87.15</span> | 85.35 | |
|
| Food101 | <span style="color:red">96.21</span> | 95.90 | |
|
| CIFAR-10 | <span style="color:red">99.36</span> | 97.90 | |
|
| CIFAR-100 | <span style="color:red">93.69</span> | 87.40 | |
|
| Birdsnap | <span style="color:red">88.18</span> | 79.90 | |
|
| SUN397 | <span style="color:red">87.96</span> | 82.20 | |
|
| Stanford Cars | <span style="color:red">95.16</span> | 91.50 | |
|
| FGVC Aircraft | <span style="color:red">86.38</span> | 71.60 | |
|
| Describable Textures Dataset | <span style="color:red">86.70</span> | 83.00 | |
|
| Oxford-IIIT Pets | <span style="color:red">96.27</span> | 95.10 | |
|
| Caltech-101 | <span style="color:red">97.92</span> | 96.00 | |
|
| Flowers102 | <span style="color:red">99.58</span> | 99.20 | |
|
| MNIST | 98.67 | <span style="color:red">99.20</span> | |
|
| STL-10 | 99.28 | <span style="color:red">99.70</span> | |
|
| EuroSAT | <span style="color:red">99.06</span> | 98.10 | |
|
| RESISC45 | <span style="color:red">95.48</span> | 94.90 | |
|
| GTSRB | 92.32 | <span style="color:red">92.40</span> | |
|
| KITTI | <span style="color:red">75.39</span> | 69.20 | |
|
| Country211 | 38.12 | <span style="color:red">46.40</span> | |
|
| PatchCamelyon | <span style="color:red">88.00</span> | 85.60 | |
|
| UCF101 | <span style="color:red">92.86</span> | 92.00 | |
|
| Kinetics-700 | <span style="color:red">73.35</span> | 73.00 | |
|
| CLEVR | <span style="color:red">64.40</span> | 60.30 | |
|
| Hateful Memes | 72.00 | <span style="color:red">77.30</span> | |
|
| SST-2 | 76.33 | <span style="color:red">80.50</span> | |
|
| ImageNet | <span style="color:red">86.30</span> | 85.40 | |
|
|
|
|
|
### C. Limitations |
|
|
|
Models with higher resolution are more friendly to OCR results. We are currently training such models and will soon make them available. |
|
|
|
|
|
## Acknowledgments |
|
|
|
We would like to express our gratitude to [Xie Yin](https://huggingface.co/Yin-Xie) and [Yumeng Wang](https://huggingface.co/devymex) for their significant contributions to the experimental validation in MLLMs. |
