Upload config.json with huggingface_hub
Browse files- config.json +359 -37
config.json
CHANGED
@@ -1,38 +1,360 @@
|
|
1 |
-
|
2 |
-
|
3 |
-
|
4 |
-
|
5 |
-
|
6 |
-
|
7 |
-
|
8 |
-
|
9 |
-
|
10 |
-
|
11 |
-
|
12 |
-
|
13 |
-
|
14 |
-
|
15 |
-
|
16 |
-
|
17 |
-
|
18 |
-
|
19 |
-
|
20 |
-
|
21 |
-
|
22 |
-
|
23 |
-
|
24 |
-
|
25 |
-
|
26 |
-
|
27 |
-
|
28 |
-
|
29 |
-
|
30 |
-
|
31 |
-
|
32 |
-
|
33 |
-
|
34 |
-
|
35 |
-
|
36 |
-
|
37 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
38 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# Reranker
|
2 |
+
|
3 |
+
**More details please refer to our Github: [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/tree/master).**
|
4 |
+
|
5 |
+
- [Model List](#model-list)
|
6 |
+
- [Usage](#usage)
|
7 |
+
- [Fine-tuning](#fine-tune)
|
8 |
+
- [Evaluation](#evaluation)
|
9 |
+
- [Citation](#citation)
|
10 |
+
|
11 |
+
Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding.
|
12 |
+
You can get a relevance score by inputting query and passage to the reranker.
|
13 |
+
And the score can be mapped to a float value in [0,1] by sigmoid function.
|
14 |
+
|
15 |
+
|
16 |
+
## Model List
|
17 |
+
|
18 |
+
| Model | Base model | Language | layerwise | feature |
|
19 |
+
|:--------------------------------------------------------------------------|:--------:|:-----------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------:|
|
20 |
+
| [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) | [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) | Chinese and English | - | Lightweight reranker model, easy to deploy, with fast inference. |
|
21 |
+
| [BAAI/bge-reranker-large](https://huggingface.co/BAAI/bge-reranker-large) | [xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) | Chinese and English | - | Lightweight reranker model, easy to deploy, with fast inference. |
|
22 |
+
| [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) | [bge-m3](https://huggingface.co/BAAI/bge-m3) | Multilingual | - | Lightweight reranker model, possesses strong multilingual capabilities, easy to deploy, with fast inference. |
|
23 |
+
| [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma) | [google/gemma-2b](https://huggingface.co/google/gemma-2b) | Multilingual | - | Suitable for multilingual contexts, performs well in both English proficiency and multilingual capabilities. |
|
24 |
+
| [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise) | [openbmb/MiniCPM-2B-dpo-fp16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp16/tree/main) | Multilingual | 8-40 | Suitable for multilingual contexts, performs well in both English and Chinese proficiency, allows freedom to select layers for output, facilitating accelerated inference. |
|
25 |
+
|
26 |
+
|
27 |
+
You can select the model according your senario and resource.
|
28 |
+
- For **multilingual**, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma)
|
29 |
+
|
30 |
+
- For **Chinese or English**, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise).
|
31 |
+
|
32 |
+
- For **efficiency**, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and the low layer of [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise).
|
33 |
+
|
34 |
+
- For better performance, recommand [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise) and [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma)
|
35 |
+
|
36 |
+
## Usage
|
37 |
+
### Using FlagEmbedding
|
38 |
+
|
39 |
+
```
|
40 |
+
pip install -U FlagEmbedding
|
41 |
+
```
|
42 |
+
|
43 |
+
#### For normal reranker (bge-reranker-base / bge-reranker-large / bge-reranker-v2-m3 )
|
44 |
+
|
45 |
+
Get relevance scores (higher scores indicate more relevance):
|
46 |
+
|
47 |
+
```python
|
48 |
+
from FlagEmbedding import FlagReranker
|
49 |
+
reranker = FlagReranker('BAAI/bge-reranker-v2-m3', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
|
50 |
+
|
51 |
+
score = reranker.compute_score(['query', 'passage'])
|
52 |
+
print(score) # -5.65234375
|
53 |
+
|
54 |
+
# You can map the scores into 0-1 by set "normalize=True", which will apply sigmoid function to the score
|
55 |
+
score = reranker.compute_score(['query', 'passage'], normalize=True)
|
56 |
+
print(score) # 0.003497010252573502
|
57 |
+
|
58 |
+
scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']])
|
59 |
+
print(scores) # [-8.1875, 5.26171875]
|
60 |
+
|
61 |
+
# You can map the scores into 0-1 by set "normalize=True", which will apply sigmoid function to the score
|
62 |
+
scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']], normalize=True)
|
63 |
+
print(scores) # [0.00027803096387751553, 0.9948403768236574]
|
64 |
+
```
|
65 |
+
|
66 |
+
#### For LLM-based reranker
|
67 |
+
|
68 |
+
```python
|
69 |
+
from FlagEmbedding import FlagLLMReranker
|
70 |
+
reranker = FlagLLMReranker('BAAI/bge-reranker-v2-gemma', use_bf16=True) # Setting use_bf16 to True speeds up computation with a slight performance degradation
|
71 |
+
|
72 |
+
score = reranker.compute_score(['query', 'passage'])
|
73 |
+
print(score) # 2.15625
|
74 |
+
|
75 |
+
scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']])
|
76 |
+
print(scores) # [-0.84765625, 10.625]
|
77 |
+
```
|
78 |
+
|
79 |
+
#### For LLM-based layerwise reranker
|
80 |
+
|
81 |
+
```python
|
82 |
+
from FlagEmbedding import LayerWiseFlagLLMReranker
|
83 |
+
reranker = LayerWiseFlagLLMReranker('BAAI/bge-reranker-v2-minicpm-layerwise', use_bf16=True) # Setting use_bf16 to True speeds up computation with a slight performance degradation
|
84 |
+
|
85 |
+
score = reranker.compute_score(['query', 'passage'], cutoff_layers=[28]) # Adjusting 'cutoff_layers' to pick which layers are used for computing the score.
|
86 |
+
print(score) # -7.03125
|
87 |
+
|
88 |
+
scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']], cutoff_layers=[28])
|
89 |
+
print(scores) # [-10.0, 1.8203125]
|
90 |
+
```
|
91 |
+
|
92 |
+
### Using Huggingface transformers
|
93 |
+
|
94 |
+
#### For normal reranker (bge-reranker-base / bge-reranker-large / bge-reranker-v2-m3 )
|
95 |
+
|
96 |
+
Get relevance scores (higher scores indicate more relevance):
|
97 |
+
|
98 |
+
```python
|
99 |
+
import torch
|
100 |
+
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
101 |
+
|
102 |
+
tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-m3')
|
103 |
+
model = AutoModelForSequenceClassification.from_pretrained('BAAI/bge-reranker-v2-m3')
|
104 |
+
model.eval()
|
105 |
+
|
106 |
+
pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
|
107 |
+
with torch.no_grad():
|
108 |
+
inputs = tokenizer(pairs, padding=True, truncation=True, return_tensors='pt', max_length=512)
|
109 |
+
scores = model(**inputs, return_dict=True).logits.view(-1, ).float()
|
110 |
+
print(scores)
|
111 |
+
```
|
112 |
+
|
113 |
+
#### For LLM-based reranker
|
114 |
+
|
115 |
+
```python
|
116 |
+
import torch
|
117 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
118 |
+
|
119 |
+
def get_inputs(pairs, tokenizer, prompt=None, max_length=1024):
|
120 |
+
if prompt is None:
|
121 |
+
prompt = "Given a query A and a passage B, determine whether the passage contains an answer to the query by providing a prediction of either 'Yes' or 'No'."
|
122 |
+
sep = "\n"
|
123 |
+
prompt_inputs = tokenizer(prompt,
|
124 |
+
return_tensors=None,
|
125 |
+
add_special_tokens=False)['input_ids']
|
126 |
+
sep_inputs = tokenizer(sep,
|
127 |
+
return_tensors=None,
|
128 |
+
add_special_tokens=False)['input_ids']
|
129 |
+
inputs = []
|
130 |
+
for query, passage in pairs:
|
131 |
+
query_inputs = tokenizer(f'A: {query}',
|
132 |
+
return_tensors=None,
|
133 |
+
add_special_tokens=False,
|
134 |
+
max_length=max_length * 3 // 4,
|
135 |
+
truncation=True)
|
136 |
+
passage_inputs = tokenizer(f'B: {passage}',
|
137 |
+
return_tensors=None,
|
138 |
+
add_special_tokens=False,
|
139 |
+
max_length=max_length,
|
140 |
+
truncation=True)
|
141 |
+
item = tokenizer.prepare_for_model(
|
142 |
+
[tokenizer.bos_token_id] + query_inputs['input_ids'],
|
143 |
+
sep_inputs + passage_inputs['input_ids'],
|
144 |
+
truncation='only_second',
|
145 |
+
max_length=max_length,
|
146 |
+
padding=False,
|
147 |
+
return_attention_mask=False,
|
148 |
+
return_token_type_ids=False,
|
149 |
+
add_special_tokens=False
|
150 |
+
)
|
151 |
+
item['input_ids'] = item['input_ids'] + sep_inputs + prompt_inputs
|
152 |
+
item['attention_mask'] = [1] * len(item['input_ids'])
|
153 |
+
inputs.append(item)
|
154 |
+
return tokenizer.pad(
|
155 |
+
inputs,
|
156 |
+
padding=True,
|
157 |
+
max_length=max_length + len(sep_inputs) + len(prompt_inputs),
|
158 |
+
pad_to_multiple_of=8,
|
159 |
+
return_tensors='pt',
|
160 |
+
)
|
161 |
+
|
162 |
+
tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-gemma')
|
163 |
+
model = AutoModelForCausalLM.from_pretrained('BAAI/bge-reranker-v2-gemma')
|
164 |
+
yes_loc = tokenizer('Yes', add_special_tokens=False)['input_ids'][0]
|
165 |
+
model.eval()
|
166 |
+
|
167 |
+
pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
|
168 |
+
with torch.no_grad():
|
169 |
+
inputs = get_inputs(pairs, tokenizer)
|
170 |
+
scores = model(**inputs, return_dict=True).logits[:, -1, yes_loc].view(-1, ).float()
|
171 |
+
print(scores)
|
172 |
+
```
|
173 |
+
|
174 |
+
#### For LLM-based layerwise reranker
|
175 |
+
|
176 |
+
```python
|
177 |
+
import torch
|
178 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
179 |
+
|
180 |
+
def get_inputs(pairs, tokenizer, prompt=None, max_length=1024):
|
181 |
+
if prompt is None:
|
182 |
+
prompt = "Given a query A and a passage B, determine whether the passage contains an answer to the query by providing a prediction of either 'Yes' or 'No'."
|
183 |
+
sep = "\n"
|
184 |
+
prompt_inputs = tokenizer(prompt,
|
185 |
+
return_tensors=None,
|
186 |
+
add_special_tokens=False)['input_ids']
|
187 |
+
sep_inputs = tokenizer(sep,
|
188 |
+
return_tensors=None,
|
189 |
+
add_special_tokens=False)['input_ids']
|
190 |
+
inputs = []
|
191 |
+
for query, passage in pairs:
|
192 |
+
query_inputs = tokenizer(f'A: {query}',
|
193 |
+
return_tensors=None,
|
194 |
+
add_special_tokens=False,
|
195 |
+
max_length=max_length * 3 // 4,
|
196 |
+
truncation=True)
|
197 |
+
passage_inputs = tokenizer(f'B: {passage}',
|
198 |
+
return_tensors=None,
|
199 |
+
add_special_tokens=False,
|
200 |
+
max_length=max_length,
|
201 |
+
truncation=True)
|
202 |
+
item = tokenizer.prepare_for_model(
|
203 |
+
[tokenizer.bos_token_id] + query_inputs['input_ids'],
|
204 |
+
sep_inputs + passage_inputs['input_ids'],
|
205 |
+
truncation='only_second',
|
206 |
+
max_length=max_length,
|
207 |
+
padding=False,
|
208 |
+
return_attention_mask=False,
|
209 |
+
return_token_type_ids=False,
|
210 |
+
add_special_tokens=False
|
211 |
+
)
|
212 |
+
item['input_ids'] = item['input_ids'] + sep_inputs + prompt_inputs
|
213 |
+
item['attention_mask'] = [1] * len(item['input_ids'])
|
214 |
+
inputs.append(item)
|
215 |
+
return tokenizer.pad(
|
216 |
+
inputs,
|
217 |
+
padding=True,
|
218 |
+
max_length=max_length + len(sep_inputs) + len(prompt_inputs),
|
219 |
+
pad_to_multiple_of=8,
|
220 |
+
return_tensors='pt',
|
221 |
+
)
|
222 |
+
|
223 |
+
tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-minicpm-layerwise', trust_remote_code=True, torch_dtype=torch.bfloat16)
|
224 |
+
model = AutoModelForCausalLM.from_pretrained('BAAI/bge-reranker-v2-minicpm-layerwise', trust_remote_code=True, torch_dtype=torch.bfloat16)
|
225 |
+
model = model.to('cuda')
|
226 |
+
model.eval()
|
227 |
+
|
228 |
+
pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
|
229 |
+
with torch.no_grad():
|
230 |
+
inputs = get_inputs(pairs, tokenizer).to(model.device)
|
231 |
+
all_scores = model(**inputs, return_dict=True, cutoff_layers=[28])
|
232 |
+
all_scores = [scores[:, -1].view(-1, ).float() for scores in all_scores[0]]
|
233 |
+
print(all_scores)
|
234 |
+
```
|
235 |
+
|
236 |
+
## Fine-tune
|
237 |
+
|
238 |
+
You can fine-tune the reranker with the following code:
|
239 |
+
|
240 |
+
**For llm-based reranker**
|
241 |
+
|
242 |
+
```shell
|
243 |
+
torchrun --nproc_per_node {number of gpus} \
|
244 |
+
-m FlagEmbedding.llm_reranker.finetune_for_instruction.run \
|
245 |
+
--output_dir {path to save model} \
|
246 |
+
--model_name_or_path BAAI/bge-reranker-v2-gemma \
|
247 |
+
--train_data ./toy_finetune_data.jsonl \
|
248 |
+
--learning_rate 2e-4 \
|
249 |
+
--num_train_epochs 1 \
|
250 |
+
--per_device_train_batch_size 1 \
|
251 |
+
--gradient_accumulation_steps 16 \
|
252 |
+
--dataloader_drop_last True \
|
253 |
+
--query_max_len 512 \
|
254 |
+
--passage_max_len 512 \
|
255 |
+
--train_group_size 16 \
|
256 |
+
--logging_steps 1 \
|
257 |
+
--save_steps 2000 \
|
258 |
+
--save_total_limit 50 \
|
259 |
+
--ddp_find_unused_parameters False \
|
260 |
+
--gradient_checkpointing \
|
261 |
+
--deepspeed stage1.json \
|
262 |
+
--warmup_ratio 0.1 \
|
263 |
+
--bf16 \
|
264 |
+
--use_lora True \
|
265 |
+
--lora_rank 32 \
|
266 |
+
--lora_alpha 64 \
|
267 |
+
--use_flash_attn True \
|
268 |
+
--target_modules q_proj k_proj v_proj o_proj
|
269 |
+
```
|
270 |
+
|
271 |
+
**For llm-based layerwise reranker**
|
272 |
+
|
273 |
+
```shell
|
274 |
+
torchrun --nproc_per_node {number of gpus} \
|
275 |
+
-m FlagEmbedding.llm_reranker.finetune_for_layerwise.run \
|
276 |
+
--output_dir {path to save model} \
|
277 |
+
--model_name_or_path BAAI/bge-reranker-v2-minicpm-layerwise \
|
278 |
+
--train_data ./toy_finetune_data.jsonl \
|
279 |
+
--learning_rate 2e-4 \
|
280 |
+
--num_train_epochs 1 \
|
281 |
+
--per_device_train_batch_size 1 \
|
282 |
+
--gradient_accumulation_steps 16 \
|
283 |
+
--dataloader_drop_last True \
|
284 |
+
--query_max_len 512 \
|
285 |
+
--passage_max_len 512 \
|
286 |
+
--train_group_size 16 \
|
287 |
+
--logging_steps 1 \
|
288 |
+
--save_steps 2000 \
|
289 |
+
--save_total_limit 50 \
|
290 |
+
--ddp_find_unused_parameters False \
|
291 |
+
--gradient_checkpointing \
|
292 |
+
--deepspeed stage1.json \
|
293 |
+
--warmup_ratio 0.1 \
|
294 |
+
--bf16 \
|
295 |
+
--use_lora True \
|
296 |
+
--lora_rank 32 \
|
297 |
+
--lora_alpha 64 \
|
298 |
+
--use_flash_attn True \
|
299 |
+
--target_modules q_proj k_proj v_proj o_proj \
|
300 |
+
--start_layer 8 \
|
301 |
+
--head_multi True \
|
302 |
+
--head_type simple
|
303 |
+
```
|
304 |
+
|
305 |
+
Our rerankers are initialized from [google/gemma-2b](https://huggingface.co/google/gemma-2b) (for llm-based reranker) and [openbmb/MiniCPM-2B-dpo-fp16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp16/tree/main) (for llm-based layerwise reranker), and we train it on a mixture of multilingual datasets:
|
306 |
+
|
307 |
+
- [bge-m3-data](https://huggingface.co/datasets/Shitao/bge-m3-data)
|
308 |
+
- [quora train data](https://huggingface.co/datasets/quora)
|
309 |
+
- [fever train data](https://fever.ai/dataset/fever.html)
|
310 |
+
|
311 |
+
## Evaluation
|
312 |
+
|
313 |
+
- llama-index.
|
314 |
+
|
315 |
+
![image-20240317193909373](./evaluation/llama-index.png)
|
316 |
+
|
317 |
+
|
318 |
+
- BEIR.
|
319 |
+
|
320 |
+
rereank the top 100 results from bge-en-v1.5 large.
|
321 |
+
|
322 |
+
![image-20240317174633333](./evaluation/BEIR-bge-en-v1.5.png)
|
323 |
+
|
324 |
+
rereank the top 100 results from e5 mistral 7b instruct.
|
325 |
+
|
326 |
+
![image-20240317172949713](./evaluation/BEIR-e5-mistral.png)
|
327 |
+
|
328 |
+
- CMTEB-retrieval.
|
329 |
+
It rereank the top 100 results from bge-zh-v1.5 large.
|
330 |
+
|
331 |
+
![image-20240317173026235](./evaluation/CMTEB-retrieval-bge-zh-v1.5.png)
|
332 |
+
|
333 |
+
- miracl (multi-language).
|
334 |
+
It rereank the top 100 results from bge-m3.
|
335 |
+
|
336 |
+
![image-20240317173117639](./evaluation/miracl-bge-m3.png)
|
337 |
+
|
338 |
+
|
339 |
+
|
340 |
+
## Citation
|
341 |
+
|
342 |
+
If you find this repository useful, please consider giving a star :star: and citation
|
343 |
+
|
344 |
+
```
|
345 |
+
@misc{li2023making,
|
346 |
+
title={Making Large Language Models A Better Foundation For Dense Retrieval},
|
347 |
+
author={Chaofan Li and Zheng Liu and Shitao Xiao and Yingxia Shao},
|
348 |
+
year={2023},
|
349 |
+
eprint={2312.15503},
|
350 |
+
archivePrefix={arXiv},
|
351 |
+
primaryClass={cs.CL}
|
352 |
}
|
353 |
+
@misc{chen2024bge,
|
354 |
+
title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
|
355 |
+
author={Jianlv Chen and Shitao Xiao and Peitian Zhang and Kun Luo and Defu Lian and Zheng Liu},
|
356 |
+
year={2024},
|
357 |
+
eprint={2402.03216},
|
358 |
+
archivePrefix={arXiv},
|
359 |
+
primaryClass={cs.CL}
|
360 |
+
}
|