metadata
language:
- zh
model-index:
- name: Chuxin-Embedding
results:
- dataset:
config: default
name: MTEB CmedqaRetrieval (default)
revision: cd540c506dae1cf9e9a59c3e06f42030d54e7301
split: dev
type: C-MTEB/CmedqaRetrieval
metrics:
- type: map_at_1
value: 33.391999999999996
- type: map_at_10
value: 48.715
- type: map_at_100
value: 50.381
- type: map_at_1000
value: 50.456
- type: map_at_3
value: 43.708999999999996
- type: map_at_5
value: 46.405
- type: mrr_at_1
value: 48.612
- type: mrr_at_10
value: 58.67099999999999
- type: mrr_at_100
value: 59.38
- type: mrr_at_1000
value: 59.396
- type: mrr_at_3
value: 55.906
- type: mrr_at_5
value: 57.421
- type: ndcg_at_1
value: 48.612
- type: ndcg_at_10
value: 56.581
- type: ndcg_at_100
value: 62.422999999999995
- type: ndcg_at_1000
value: 63.476
- type: ndcg_at_3
value: 50.271
- type: ndcg_at_5
value: 52.79899999999999
- type: precision_at_1
value: 48.612
- type: precision_at_10
value: 11.995000000000001
- type: precision_at_100
value: 1.696
- type: precision_at_1000
value: 0.185
- type: precision_at_3
value: 27.465
- type: precision_at_5
value: 19.675
- type: recall_at_1
value: 33.391999999999996
- type: recall_at_10
value: 69.87100000000001
- type: recall_at_100
value: 93.078
- type: recall_at_1000
value: 99.55199999999999
- type: recall_at_3
value: 50.939
- type: recall_at_5
value: 58.714
- type: main_score
value: 56.581
task:
type: Retrieval
- dataset:
config: default
name: MTEB CovidRetrieval (default)
revision: 1271c7809071a13532e05f25fb53511ffce77117
split: dev
type: C-MTEB/CovidRetrieval
metrics:
- type: map_at_1
value: 71.918
- type: map_at_10
value: 80.609
- type: map_at_100
value: 80.796
- type: map_at_1000
value: 80.798
- type: map_at_3
value: 79.224
- type: map_at_5
value: 79.96
- type: mrr_at_1
value: 72.076
- type: mrr_at_10
value: 80.61399999999999
- type: mrr_at_100
value: 80.801
- type: mrr_at_1000
value: 80.803
- type: mrr_at_3
value: 79.276
- type: mrr_at_5
value: 80.025
- type: ndcg_at_1
value: 72.076
- type: ndcg_at_10
value: 84.286
- type: ndcg_at_100
value: 85.14500000000001
- type: ndcg_at_1000
value: 85.21
- type: ndcg_at_3
value: 81.45400000000001
- type: ndcg_at_5
value: 82.781
- type: precision_at_1
value: 72.076
- type: precision_at_10
value: 9.663
- type: precision_at_100
value: 1.005
- type: precision_at_1000
value: 0.101
- type: precision_at_3
value: 29.398999999999997
- type: precision_at_5
value: 18.335
- type: recall_at_1
value: 71.918
- type: recall_at_10
value: 95.574
- type: recall_at_100
value: 99.473
- type: recall_at_1000
value: 100
- type: recall_at_3
value: 87.82900000000001
- type: recall_at_5
value: 90.991
- type: main_score
value: 84.286
task:
type: Retrieval
- dataset:
config: default
name: MTEB DuRetrieval (default)
revision: a1a333e290fe30b10f3f56498e3a0d911a693ced
split: dev
type: C-MTEB/DuRetrieval
metrics:
- type: map_at_1
value: 25.019999999999996
- type: map_at_10
value: 77.744
- type: map_at_100
value: 80.562
- type: map_at_1000
value: 80.60300000000001
- type: map_at_3
value: 52.642999999999994
- type: map_at_5
value: 67.179
- type: mrr_at_1
value: 86.5
- type: mrr_at_10
value: 91.024
- type: mrr_at_100
value: 91.09
- type: mrr_at_1000
value: 91.093
- type: mrr_at_3
value: 90.558
- type: mrr_at_5
value: 90.913
- type: ndcg_at_1
value: 86.5
- type: ndcg_at_10
value: 85.651
- type: ndcg_at_100
value: 88.504
- type: ndcg_at_1000
value: 88.887
- type: ndcg_at_3
value: 82.707
- type: ndcg_at_5
value: 82.596
- type: precision_at_1
value: 86.5
- type: precision_at_10
value: 41.595
- type: precision_at_100
value: 4.7940000000000005
- type: precision_at_1000
value: 0.48900000000000005
- type: precision_at_3
value: 74.233
- type: precision_at_5
value: 63.68000000000001
- type: recall_at_1
value: 25.019999999999996
- type: recall_at_10
value: 88.114
- type: recall_at_100
value: 97.442
- type: recall_at_1000
value: 99.39099999999999
- type: recall_at_3
value: 55.397
- type: recall_at_5
value: 73.095
- type: main_score
value: 85.651
task:
type: Retrieval
- dataset:
config: default
name: MTEB EcomRetrieval (default)
revision: 687de13dc7294d6fd9be10c6945f9e8fec8166b9
split: dev
type: C-MTEB/EcomRetrieval
metrics:
- type: map_at_1
value: 55.60000000000001
- type: map_at_10
value: 67.891
- type: map_at_100
value: 68.28699999999999
- type: map_at_1000
value: 68.28699999999999
- type: map_at_3
value: 64.86699999999999
- type: map_at_5
value: 66.652
- type: mrr_at_1
value: 55.60000000000001
- type: mrr_at_10
value: 67.891
- type: mrr_at_100
value: 68.28699999999999
- type: mrr_at_1000
value: 68.28699999999999
- type: mrr_at_3
value: 64.86699999999999
- type: mrr_at_5
value: 66.652
- type: ndcg_at_1
value: 55.60000000000001
- type: ndcg_at_10
value: 74.01100000000001
- type: ndcg_at_100
value: 75.602
- type: ndcg_at_1000
value: 75.602
- type: ndcg_at_3
value: 67.833
- type: ndcg_at_5
value: 71.005
- type: precision_at_1
value: 55.60000000000001
- type: precision_at_10
value: 9.33
- type: precision_at_100
value: 1
- type: precision_at_1000
value: 0.1
- type: precision_at_3
value: 25.467000000000002
- type: precision_at_5
value: 16.8
- type: recall_at_1
value: 55.60000000000001
- type: recall_at_10
value: 93.30000000000001
- type: recall_at_100
value: 100
- type: recall_at_1000
value: 100
- type: recall_at_3
value: 76.4
- type: recall_at_5
value: 84
- type: main_score
value: 74.01100000000001
task:
type: Retrieval
- dataset:
config: default
name: MTEB MMarcoRetrieval (default)
revision: 539bbde593d947e2a124ba72651aafc09eb33fc2
split: dev
type: C-MTEB/MMarcoRetrieval
metrics:
- type: map_at_1
value: 66.24799999999999
- type: map_at_10
value: 75.356
- type: map_at_100
value: 75.653
- type: map_at_1000
value: 75.664
- type: map_at_3
value: 73.515
- type: map_at_5
value: 74.67099999999999
- type: mrr_at_1
value: 68.496
- type: mrr_at_10
value: 75.91499999999999
- type: mrr_at_100
value: 76.17399999999999
- type: mrr_at_1000
value: 76.184
- type: mrr_at_3
value: 74.315
- type: mrr_at_5
value: 75.313
- type: ndcg_at_1
value: 68.496
- type: ndcg_at_10
value: 79.065
- type: ndcg_at_100
value: 80.417
- type: ndcg_at_1000
value: 80.72399999999999
- type: ndcg_at_3
value: 75.551
- type: ndcg_at_5
value: 77.505
- type: precision_at_1
value: 68.496
- type: precision_at_10
value: 9.563
- type: precision_at_100
value: 1.024
- type: precision_at_1000
value: 0.105
- type: precision_at_3
value: 28.391
- type: precision_at_5
value: 18.086
- type: recall_at_1
value: 66.24799999999999
- type: recall_at_10
value: 89.97
- type: recall_at_100
value: 96.13199999999999
- type: recall_at_1000
value: 98.551
- type: recall_at_3
value: 80.624
- type: recall_at_5
value: 85.271
- type: main_score
value: 79.065
task:
type: Retrieval
- dataset:
config: default
name: MTEB MedicalRetrieval (default)
revision: 2039188fb5800a9803ba5048df7b76e6fb151fc6
split: dev
type: C-MTEB/MedicalRetrieval
metrics:
- type: map_at_1
value: 61.8
- type: map_at_10
value: 71.101
- type: map_at_100
value: 71.576
- type: map_at_1000
value: 71.583
- type: map_at_3
value: 68.867
- type: map_at_5
value: 70.272
- type: mrr_at_1
value: 61.9
- type: mrr_at_10
value: 71.158
- type: mrr_at_100
value: 71.625
- type: mrr_at_1000
value: 71.631
- type: mrr_at_3
value: 68.917
- type: mrr_at_5
value: 70.317
- type: ndcg_at_1
value: 61.8
- type: ndcg_at_10
value: 75.624
- type: ndcg_at_100
value: 77.702
- type: ndcg_at_1000
value: 77.836
- type: ndcg_at_3
value: 71.114
- type: ndcg_at_5
value: 73.636
- type: precision_at_1
value: 61.8
- type: precision_at_10
value: 8.98
- type: precision_at_100
value: 0.9900000000000001
- type: precision_at_1000
value: 0.1
- type: precision_at_3
value: 25.867
- type: precision_at_5
value: 16.74
- type: recall_at_1
value: 61.8
- type: recall_at_10
value: 89.8
- type: recall_at_100
value: 99
- type: recall_at_1000
value: 100
- type: recall_at_3
value: 77.60000000000001
- type: recall_at_5
value: 83.7
- type: main_score
value: 75.624
task:
type: Retrieval
- dataset:
config: default
name: MTEB T2Retrieval (default)
revision: 8731a845f1bf500a4f111cf1070785c793d10e64
split: dev
type: C-MTEB/T2Retrieval
metrics:
- type: map_at_1
value: 27.173000000000002
- type: map_at_10
value: 76.454
- type: map_at_100
value: 80.021
- type: map_at_1000
value: 80.092
- type: map_at_3
value: 53.876999999999995
- type: map_at_5
value: 66.122
- type: mrr_at_1
value: 89.519
- type: mrr_at_10
value: 92.091
- type: mrr_at_100
value: 92.179
- type: mrr_at_1000
value: 92.183
- type: mrr_at_3
value: 91.655
- type: mrr_at_5
value: 91.94
- type: ndcg_at_1
value: 89.519
- type: ndcg_at_10
value: 84.043
- type: ndcg_at_100
value: 87.60900000000001
- type: ndcg_at_1000
value: 88.32799999999999
- type: ndcg_at_3
value: 85.623
- type: ndcg_at_5
value: 84.111
- type: precision_at_1
value: 89.519
- type: precision_at_10
value: 41.760000000000005
- type: precision_at_100
value: 4.982
- type: precision_at_1000
value: 0.515
- type: precision_at_3
value: 74.944
- type: precision_at_5
value: 62.705999999999996
- type: recall_at_1
value: 27.173000000000002
- type: recall_at_10
value: 82.878
- type: recall_at_100
value: 94.527
- type: recall_at_1000
value: 98.24199999999999
- type: recall_at_3
value: 55.589
- type: recall_at_5
value: 69.476
- type: main_score
value: 84.043
task:
type: Retrieval
- dataset:
config: default
name: MTEB VideoRetrieval (default)
revision: 58c2597a5943a2ba48f4668c3b90d796283c5639
split: dev
type: C-MTEB/VideoRetrieval
metrics:
- type: map_at_1
value: 70.1
- type: map_at_10
value: 79.62
- type: map_at_100
value: 79.804
- type: map_at_1000
value: 79.804
- type: map_at_3
value: 77.81700000000001
- type: map_at_5
value: 79.037
- type: mrr_at_1
value: 70.1
- type: mrr_at_10
value: 79.62
- type: mrr_at_100
value: 79.804
- type: mrr_at_1000
value: 79.804
- type: mrr_at_3
value: 77.81700000000001
- type: mrr_at_5
value: 79.037
- type: ndcg_at_1
value: 70.1
- type: ndcg_at_10
value: 83.83500000000001
- type: ndcg_at_100
value: 84.584
- type: ndcg_at_1000
value: 84.584
- type: ndcg_at_3
value: 80.282
- type: ndcg_at_5
value: 82.472
- type: precision_at_1
value: 70.1
- type: precision_at_10
value: 9.68
- type: precision_at_100
value: 1
- type: precision_at_1000
value: 0.1
- type: precision_at_3
value: 29.133
- type: precision_at_5
value: 18.54
- type: recall_at_1
value: 70.1
- type: recall_at_10
value: 96.8
- type: recall_at_100
value: 100
- type: recall_at_1000
value: 100
- type: recall_at_3
value: 87.4
- type: recall_at_5
value: 92.7
- type: main_score
value: 83.83500000000001
task:
type: Retrieval
tags:
- mteb
Chuxin-Embedding
Chuxin-Embedding 是专为增强中文文本检索能力而设计的嵌入模型。它基于bge-m3-retromae[1]，实现了预训练、微调、精调全流程。该模型在来自各个领域的大量语料库上进行训练，语料库的批量非常大。截至 2024 年 9 月 14 日, Chuxin-Embedding 在检索任务中表现出色，在 C-MTEB 中文检索排行榜上排名第一，领先的性能得分为 77.88，在AIR-Bench中文检索+重排序公开排行榜上排名第一，领先的性能得分为 64.78。
Chuxin-Embedding is a specially designed embedding model aimed at enhancing the capability of Chinese text retrieval. It is based on bge-m3-retromae[1] and implements the entire process of pre-training, fine-tuning, and refining. This model has been trained on a vast amount of corpora from various fields. As of September 14, 2024, Chuxin-Embedding has shown outstanding performance in retrieval tasks. It ranks first on the C-MTEB Chinese Retrieval Leaderboard with a leading performance score of 77.88 and also ranks first on the AIR-Bench Chinese Retrieval + Re-ranking Public Leaderboard with a leading performance score of 64.78.
News
2024/9/14: LLM生成及数据清洗 Code coming soon 。
2024/9/14: 团队的RAG框架欢迎试用 ragnify 。
2024/9/14: LLM generation and data clean Code coming soon.
2024/9/14: The team's RAG framework is available for trial ragnify .
Training Details
基于bge-m3-retromae[1]，主要改动如下：
- 基于bge-m3-retromae[1]在亿级数据上预训练。
- 使用BGE pretrain Code 完成预训练。
- 在收集的公开亿级检索数据集上实现了微调。
- 使用BGE finetune Code 完成微调。
- 在收集的公开百万级检索数据集和百万级LLM合成数据集上实现了精调。
Based on bge-m3-retromae[1], the main modifications are as follows:
- Pre-trained on a billion-level dataset based on bge-m3-retromae[1].
- Pre-training is completed using BGE pretrain Code .
- Fine-tuned on a publicly collected billion-level retrieval dataset.
- Fine-tuning is completed using BGE finetune Code.
- Refined on a publicly collected million-level retrieval dataset and a million-level LLM synthetic dataset.
- Refining is completed using BGE finetune Code and BGE unified_finetune Code.
- Data generation is performed through LLM (QWEN-72B), using LLM to generate new query for messages.
- Data cleaning:
- Simple rule-based cleaning
- LLM to determine whether a query can be used as a search engine query
- The rerank model scores (query, message) pairs, discarding negative examples in the positive set and positive examples in the negative set.
Collect more data for retrieval-type tasks
- 预训练数据
- ChineseWebText、 oasis、 oscar、 SkyPile、 wudao
- 微调数据
- MTP 、webqa、nlpcc、csl、bq、atec、ccks
- 精调数据
- BGE-M3 、Huatuo26M-Lite 、covid ...
- LLM 合成（BGE-M3 、Huatuo26M-Lite 、covid、wudao、wanjuan_news、mnbvc_news_wiki、mldr、medical QA...）
Performance
C_MTEB RETRIEVAL
| Model | Average | CmedqaRetrieval | CovidRetrieval | DuRetrieval | EcomRetrieval | MedicalRetrieval | MMarcoRetrieval | T2Retrieval | VideoRetrieval |
|---|---|---|---|---|---|---|---|---|---|
| Zhihui_LLM_Embedding | 76.74 | 48.69 | 84.39 | 91.34 | 71.96 | 65.19 | 84.77 | 88.3 | 79.31 |
| zpoint_large_embedding_zh | 76.36 | 47.16 | 89.14 | 89.23 | 70.74 | 68.14 | 82.38 | 83.81 | 80.26 |
| Chuxin-Embedding | 77.88 | 56.58 | 84.28 | 85.65 | 74.01 | 75.62 | 79.06 | 84.04 | 83.84 |
AIR-Bench
| Retrieval Method | Reranking Model | Average | wiki_zh | web_zh | news_zh | healthcare_zh | finance_zh |
|---|---|---|---|---|---|---|---|
| bge-m3 | bge-reranker-large | 64.53 | 76.11 | 67.8 | 63.25 | 62.9 | 52.61 |
| gte-Qwen2-7B-instruct | bge-reranker-large | 63.39 | 78.09 | 67.56 | 63.14 | 61.12 | 47.02 |
| Chuxin-Embedding | bge-reranker-large | 64.78 | 76.23 | 68.44 | 64.2 | 62.93 | 52.11 |
Generate Embedding for text
#pip install -U FlagEmbedding
from FlagEmbedding import FlagModel
model = FlagModel('chuxin-llm/Chuxin-Embedding',
query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章：",
use_fp16=True)
sentences_1 = ["样例数据-1", "样例数据-2"]
sentences_2 = ["样例数据-3", "样例数据-1"]
embeddings_1 = model.encode(sentences_1)
embeddings_2 = model.encode(sentences_2)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)