jupyterjazz
commited on
Commit
•
b04e355
1
Parent(s):
8bf1330
Update README.md
Browse files
README.md
CHANGED
@@ -270,7 +270,7 @@ embeddings = model.encode(['How is the weather today?'], task_type='retrieval.qu
|
|
270 |
## Performance
|
271 |
|
272 |
### English MTEB
|
273 |
-
| Model | Average |
|
274 |
|:------------------------------:|:-------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|
275 |
| jina-embeddings-v2-en | 58.12 | 68.82| 40.08| 84.44| 55.09| 45.64| 80.00| 30.56|
|
276 |
| jina-embeddings-v3 | **65.60** | **82.58**| 45.27| 84.01| 58.13| 53.87| **85.8** | 30.98|
|
@@ -280,22 +280,13 @@ embeddings = model.encode(['How is the weather today?'], task_type='retrieval.qu
|
|
280 |
|
281 |
### Multilingual MTEB
|
282 |
|
283 |
-
| Model | Average |
|
284 |
|:------------------------------:|:-------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|
285 |
-
| jina-embeddings-v2
|
286 |
| jina-embeddings-v3 | **64.44** | **71.46**| 46.71| 76.91| 63.98| 57.98| **69.83**| - |
|
287 |
| multilingual-e5-large | 59.58 | 65.22| 42.12| 76.95| 63.4 | 52.37| 64.65| - |
|
288 |
| multilingual-e5-large-instruct | 64.25 | 67.45| **52.12**| 77.79| **69.02**| **58.38**| 68.77| - |
|
289 |
|
290 |
-
**Notes:**
|
291 |
-
- CF: Classification Accuracy
|
292 |
-
- CL: Clustering V measure
|
293 |
-
- PC: Pair Classification Average Precision
|
294 |
-
- RR: Reranking MAP
|
295 |
-
- RT: Retrieval nDCG@10
|
296 |
-
- STS: Sentence Similarity Spearman Correlation
|
297 |
-
- SM: Summarization Spearman Correlation
|
298 |
-
- `*`: Jina Embeddings v2 bilingual model suite, only evaluated on Chinese, Spanish and German tasks.
|
299 |
|
300 |
### Long Context Tasks (LongEmbed)
|
301 |
|
@@ -317,6 +308,7 @@ embeddings = model.encode(['How is the weather today?'], task_type='retrieval.qu
|
|
317 |
| Retrieval | 52.54| 58.54| 61.64| 62.72| 63.16| 63.30| 63.35|
|
318 |
| STS | 76.35| 77.03| 77.43| 77.56| 77.59| 77.59| 77.58|
|
319 |
|
|
|
320 |
|
321 |
## Contact
|
322 |
|
|
|
270 |
## Performance
|
271 |
|
272 |
### English MTEB
|
273 |
+
| Model | Average | Classification | Clustering | Pair Classification | Reranking | Retrieval | STS | Summarization |
|
274 |
|:------------------------------:|:-------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|
275 |
| jina-embeddings-v2-en | 58.12 | 68.82| 40.08| 84.44| 55.09| 45.64| 80.00| 30.56|
|
276 |
| jina-embeddings-v3 | **65.60** | **82.58**| 45.27| 84.01| 58.13| 53.87| **85.8** | 30.98|
|
|
|
280 |
|
281 |
### Multilingual MTEB
|
282 |
|
283 |
+
| Model | Average | Classification | Clustering | Pair Classification | Reranking | Retrieval | STS | Summarization |
|
284 |
|:------------------------------:|:-------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|
285 |
+
| jina-embeddings-v2 | 60.54 | 65.69| 39.36| **82.95**| 66.57| 58.24| 66.6 | - |
|
286 |
| jina-embeddings-v3 | **64.44** | **71.46**| 46.71| 76.91| 63.98| 57.98| **69.83**| - |
|
287 |
| multilingual-e5-large | 59.58 | 65.22| 42.12| 76.95| 63.4 | 52.37| 64.65| - |
|
288 |
| multilingual-e5-large-instruct | 64.25 | 67.45| **52.12**| 77.79| **69.02**| **58.38**| 68.77| - |
|
289 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
290 |
|
291 |
### Long Context Tasks (LongEmbed)
|
292 |
|
|
|
308 |
| Retrieval | 52.54| 58.54| 61.64| 62.72| 63.16| 63.30| 63.35|
|
309 |
| STS | 76.35| 77.03| 77.43| 77.56| 77.59| 77.59| 77.58|
|
310 |
|
311 |
+
For a comprehensive evaluation and detailed metrics, please refer to the full paper available here (coming soon).
|
312 |
|
313 |
## Contact
|
314 |
|