Add new benchmark
Browse files
README.md
CHANGED
|
@@ -9,7 +9,7 @@ tags:
|
|
| 9 |
# Omnivision
|
| 10 |
|
| 11 |
## 🔥 Latest Update
|
| 12 |
-
- [Nov 22, 2024] **Major Model Improvements:**
|
| 13 |
- Enhanced Art Descriptions
|
| 14 |
- Better Complex Image Understanding
|
| 15 |
- Improved Anime Recognition
|
|
@@ -53,15 +53,15 @@ Below we demonstrate a figure to show how Omnivision performs against nanollava.
|
|
| 53 |
|
| 54 |
We have conducted a series of experiments on benchmark datasets, including MM-VET, ChartQA, MMMU, ScienceQA, POPE to evaluate the performance of Omnivision.
|
| 55 |
|
| 56 |
-
| Benchmark | Nexa AI Omnivision | nanoLLAVA |
|
| 57 |
-
|
| 58 |
-
|
|
| 59 |
-
|
|
| 60 |
-
|
|
| 61 |
-
|
|
| 62 |
-
|
|
| 63 |
-
|
|
| 64 |
-
|
|
| 65 |
|
| 66 |
|
| 67 |
## How to Use On Device
|
|
|
|
| 9 |
# Omnivision
|
| 10 |
|
| 11 |
## 🔥 Latest Update
|
| 12 |
+
- [Nov 22, 2024] **Major Model Improvements:** OmniVision v2 model's **GGUF file has been updated** in this Hugging Face Repo! ✨ Key Improvements Include:
|
| 13 |
- Enhanced Art Descriptions
|
| 14 |
- Better Complex Image Understanding
|
| 15 |
- Improved Anime Recognition
|
|
|
|
| 53 |
|
| 54 |
We have conducted a series of experiments on benchmark datasets, including MM-VET, ChartQA, MMMU, ScienceQA, POPE to evaluate the performance of Omnivision.
|
| 55 |
|
| 56 |
+
| Benchmark | Nexa AI Omnivision v2 | Nexa AI Omnivision v1 | nanoLLAVA |
|
| 57 |
+
|-------------------|------------------------|------------------------|-----------|
|
| 58 |
+
| ScienceQA (Eval) | 71.0 | 62.2 | 59.0 |
|
| 59 |
+
| ScienceQA (Test) | 71.0 | 64.5 | 59.0 |
|
| 60 |
+
| POPE | 93.3 | 89.4 | 84.1 |
|
| 61 |
+
| MM-VET | 30.9 | 27.5 | 23.9 |
|
| 62 |
+
| ChartQA (Test) | 61.9 | 59.2 | NA |
|
| 63 |
+
| MMMU (Test) | 42.1 | 41.8 | 28.6 |
|
| 64 |
+
| MMMU (Eval) | 40.0 | 39.9 | 30.4 |
|
| 65 |
|
| 66 |
|
| 67 |
## How to Use On Device
|