Shitao commited on
Commit
2b0acf7
1 Parent(s): 322738c

Upload config.json with huggingface_hub

Browse files
Files changed (1) hide show
  1. config.json +359 -37
config.json CHANGED
@@ -1,38 +1,360 @@
1
- {
2
- "_name_or_path": "BAAI/bge-reranker-v2-minicpm-layerwise",
3
- "architectures": [
4
- "LayerWiseMiniCPMForCausalLM"
5
- ],
6
- "attention_bias": false,
7
- "attention_dropout": 0.0,
8
- "auto_map": {
9
- "AutoConfig": "BAAI/bge-reranker-v2-minicpm-layerwise--configuration_minicpm_reranker.LayerWiseMiniCPMConfig",
10
- "AutoModel": "BAAI/bge-reranker-v2-minicpm-layerwise--modeling_minicpm_reranker.LayerWiseMiniCPMModel",
11
- "AutoModelForCausalLM": "BAAI/bge-reranker-v2-minicpm-layerwise--modeling_minicpm_reranker.LayerWiseMiniCPMForCausalLM"
12
- },
13
- "bos_token_id": 1,
14
- "dim_model_base": 256,
15
- "eos_token_id": 2,
16
- "head_multi": true,
17
- "head_type": "simple",
18
- "hidden_act": "silu",
19
- "hidden_size": 2304,
20
- "initializer_range": 0.1,
21
- "intermediate_size": 5760,
22
- "max_position_embeddings": 2048,
23
- "model_type": "minicpm",
24
- "num_attention_heads": 36,
25
- "num_hidden_layers": 40,
26
- "num_key_value_heads": 36,
27
- "pretraining_tp": 1,
28
- "rms_norm_eps": 1e-05,
29
- "rope_scaling": null,
30
- "rope_theta": 10000.0,
31
- "scale_depth": 1.4,
32
- "scale_emb": 12,
33
- "start_layer": 8,
34
- "torch_dtype": "bfloat16",
35
- "transformers_version": "4.38.1",
36
- "use_cache": false,
37
- "vocab_size": 122753
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
  }
 
 
 
 
 
 
 
 
 
1
+ # Reranker
2
+
3
+ **More details please refer to our Github: [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/tree/master).**
4
+
5
+ - [Model List](#model-list)
6
+ - [Usage](#usage)
7
+ - [Fine-tuning](#fine-tune)
8
+ - [Evaluation](#evaluation)
9
+ - [Citation](#citation)
10
+
11
+ Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding.
12
+ You can get a relevance score by inputting query and passage to the reranker.
13
+ And the score can be mapped to a float value in [0,1] by sigmoid function.
14
+
15
+
16
+ ## Model List
17
+
18
+ | Model | Base model | Language | layerwise | feature |
19
+ |:--------------------------------------------------------------------------|:--------:|:-----------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------:|
20
+ | [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) | [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) | Chinese and English | - | Lightweight reranker model, easy to deploy, with fast inference. |
21
+ | [BAAI/bge-reranker-large](https://huggingface.co/BAAI/bge-reranker-large) | [xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) | Chinese and English | - | Lightweight reranker model, easy to deploy, with fast inference. |
22
+ | [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) | [bge-m3](https://huggingface.co/BAAI/bge-m3) | Multilingual | - | Lightweight reranker model, possesses strong multilingual capabilities, easy to deploy, with fast inference. |
23
+ | [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma) | [google/gemma-2b](https://huggingface.co/google/gemma-2b) | Multilingual | - | Suitable for multilingual contexts, performs well in both English proficiency and multilingual capabilities. |
24
+ | [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise) | [openbmb/MiniCPM-2B-dpo-fp16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp16/tree/main) | Multilingual | 8-40 | Suitable for multilingual contexts, performs well in both English and Chinese proficiency, allows freedom to select layers for output, facilitating accelerated inference. |
25
+
26
+
27
+ You can select the model according your senario and resource.
28
+ - For **multilingual**, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma)
29
+
30
+ - For **Chinese or English**, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise).
31
+
32
+ - For **efficiency**, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and the low layer of [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise).
33
+
34
+ - For better performance, recommand [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise) and [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma)
35
+
36
+ ## Usage
37
+ ### Using FlagEmbedding
38
+
39
+ ```
40
+ pip install -U FlagEmbedding
41
+ ```
42
+
43
+ #### For normal reranker (bge-reranker-base / bge-reranker-large / bge-reranker-v2-m3 )
44
+
45
+ Get relevance scores (higher scores indicate more relevance):
46
+
47
+ ```python
48
+ from FlagEmbedding import FlagReranker
49
+ reranker = FlagReranker('BAAI/bge-reranker-v2-m3', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
50
+
51
+ score = reranker.compute_score(['query', 'passage'])
52
+ print(score) # -5.65234375
53
+
54
+ # You can map the scores into 0-1 by set "normalize=True", which will apply sigmoid function to the score
55
+ score = reranker.compute_score(['query', 'passage'], normalize=True)
56
+ print(score) # 0.003497010252573502
57
+
58
+ scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']])
59
+ print(scores) # [-8.1875, 5.26171875]
60
+
61
+ # You can map the scores into 0-1 by set "normalize=True", which will apply sigmoid function to the score
62
+ scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']], normalize=True)
63
+ print(scores) # [0.00027803096387751553, 0.9948403768236574]
64
+ ```
65
+
66
+ #### For LLM-based reranker
67
+
68
+ ```python
69
+ from FlagEmbedding import FlagLLMReranker
70
+ reranker = FlagLLMReranker('BAAI/bge-reranker-v2-gemma', use_bf16=True) # Setting use_bf16 to True speeds up computation with a slight performance degradation
71
+
72
+ score = reranker.compute_score(['query', 'passage'])
73
+ print(score) # 2.15625
74
+
75
+ scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']])
76
+ print(scores) # [-0.84765625, 10.625]
77
+ ```
78
+
79
+ #### For LLM-based layerwise reranker
80
+
81
+ ```python
82
+ from FlagEmbedding import LayerWiseFlagLLMReranker
83
+ reranker = LayerWiseFlagLLMReranker('BAAI/bge-reranker-v2-minicpm-layerwise', use_bf16=True) # Setting use_bf16 to True speeds up computation with a slight performance degradation
84
+
85
+ score = reranker.compute_score(['query', 'passage'], cutoff_layers=[28]) # Adjusting 'cutoff_layers' to pick which layers are used for computing the score.
86
+ print(score) # -7.03125
87
+
88
+ scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']], cutoff_layers=[28])
89
+ print(scores) # [-10.0, 1.8203125]
90
+ ```
91
+
92
+ ### Using Huggingface transformers
93
+
94
+ #### For normal reranker (bge-reranker-base / bge-reranker-large / bge-reranker-v2-m3 )
95
+
96
+ Get relevance scores (higher scores indicate more relevance):
97
+
98
+ ```python
99
+ import torch
100
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
101
+
102
+ tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-m3')
103
+ model = AutoModelForSequenceClassification.from_pretrained('BAAI/bge-reranker-v2-m3')
104
+ model.eval()
105
+
106
+ pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
107
+ with torch.no_grad():
108
+ inputs = tokenizer(pairs, padding=True, truncation=True, return_tensors='pt', max_length=512)
109
+ scores = model(**inputs, return_dict=True).logits.view(-1, ).float()
110
+ print(scores)
111
+ ```
112
+
113
+ #### For LLM-based reranker
114
+
115
+ ```python
116
+ import torch
117
+ from transformers import AutoModelForCausalLM, AutoTokenizer
118
+
119
+ def get_inputs(pairs, tokenizer, prompt=None, max_length=1024):
120
+ if prompt is None:
121
+ prompt = "Given a query A and a passage B, determine whether the passage contains an answer to the query by providing a prediction of either 'Yes' or 'No'."
122
+ sep = "\n"
123
+ prompt_inputs = tokenizer(prompt,
124
+ return_tensors=None,
125
+ add_special_tokens=False)['input_ids']
126
+ sep_inputs = tokenizer(sep,
127
+ return_tensors=None,
128
+ add_special_tokens=False)['input_ids']
129
+ inputs = []
130
+ for query, passage in pairs:
131
+ query_inputs = tokenizer(f'A: {query}',
132
+ return_tensors=None,
133
+ add_special_tokens=False,
134
+ max_length=max_length * 3 // 4,
135
+ truncation=True)
136
+ passage_inputs = tokenizer(f'B: {passage}',
137
+ return_tensors=None,
138
+ add_special_tokens=False,
139
+ max_length=max_length,
140
+ truncation=True)
141
+ item = tokenizer.prepare_for_model(
142
+ [tokenizer.bos_token_id] + query_inputs['input_ids'],
143
+ sep_inputs + passage_inputs['input_ids'],
144
+ truncation='only_second',
145
+ max_length=max_length,
146
+ padding=False,
147
+ return_attention_mask=False,
148
+ return_token_type_ids=False,
149
+ add_special_tokens=False
150
+ )
151
+ item['input_ids'] = item['input_ids'] + sep_inputs + prompt_inputs
152
+ item['attention_mask'] = [1] * len(item['input_ids'])
153
+ inputs.append(item)
154
+ return tokenizer.pad(
155
+ inputs,
156
+ padding=True,
157
+ max_length=max_length + len(sep_inputs) + len(prompt_inputs),
158
+ pad_to_multiple_of=8,
159
+ return_tensors='pt',
160
+ )
161
+
162
+ tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-gemma')
163
+ model = AutoModelForCausalLM.from_pretrained('BAAI/bge-reranker-v2-gemma')
164
+ yes_loc = tokenizer('Yes', add_special_tokens=False)['input_ids'][0]
165
+ model.eval()
166
+
167
+ pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
168
+ with torch.no_grad():
169
+ inputs = get_inputs(pairs, tokenizer)
170
+ scores = model(**inputs, return_dict=True).logits[:, -1, yes_loc].view(-1, ).float()
171
+ print(scores)
172
+ ```
173
+
174
+ #### For LLM-based layerwise reranker
175
+
176
+ ```python
177
+ import torch
178
+ from transformers import AutoModelForCausalLM, AutoTokenizer
179
+
180
+ def get_inputs(pairs, tokenizer, prompt=None, max_length=1024):
181
+ if prompt is None:
182
+ prompt = "Given a query A and a passage B, determine whether the passage contains an answer to the query by providing a prediction of either 'Yes' or 'No'."
183
+ sep = "\n"
184
+ prompt_inputs = tokenizer(prompt,
185
+ return_tensors=None,
186
+ add_special_tokens=False)['input_ids']
187
+ sep_inputs = tokenizer(sep,
188
+ return_tensors=None,
189
+ add_special_tokens=False)['input_ids']
190
+ inputs = []
191
+ for query, passage in pairs:
192
+ query_inputs = tokenizer(f'A: {query}',
193
+ return_tensors=None,
194
+ add_special_tokens=False,
195
+ max_length=max_length * 3 // 4,
196
+ truncation=True)
197
+ passage_inputs = tokenizer(f'B: {passage}',
198
+ return_tensors=None,
199
+ add_special_tokens=False,
200
+ max_length=max_length,
201
+ truncation=True)
202
+ item = tokenizer.prepare_for_model(
203
+ [tokenizer.bos_token_id] + query_inputs['input_ids'],
204
+ sep_inputs + passage_inputs['input_ids'],
205
+ truncation='only_second',
206
+ max_length=max_length,
207
+ padding=False,
208
+ return_attention_mask=False,
209
+ return_token_type_ids=False,
210
+ add_special_tokens=False
211
+ )
212
+ item['input_ids'] = item['input_ids'] + sep_inputs + prompt_inputs
213
+ item['attention_mask'] = [1] * len(item['input_ids'])
214
+ inputs.append(item)
215
+ return tokenizer.pad(
216
+ inputs,
217
+ padding=True,
218
+ max_length=max_length + len(sep_inputs) + len(prompt_inputs),
219
+ pad_to_multiple_of=8,
220
+ return_tensors='pt',
221
+ )
222
+
223
+ tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-minicpm-layerwise', trust_remote_code=True, torch_dtype=torch.bfloat16)
224
+ model = AutoModelForCausalLM.from_pretrained('BAAI/bge-reranker-v2-minicpm-layerwise', trust_remote_code=True, torch_dtype=torch.bfloat16)
225
+ model = model.to('cuda')
226
+ model.eval()
227
+
228
+ pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
229
+ with torch.no_grad():
230
+ inputs = get_inputs(pairs, tokenizer).to(model.device)
231
+ all_scores = model(**inputs, return_dict=True, cutoff_layers=[28])
232
+ all_scores = [scores[:, -1].view(-1, ).float() for scores in all_scores[0]]
233
+ print(all_scores)
234
+ ```
235
+
236
+ ## Fine-tune
237
+
238
+ You can fine-tune the reranker with the following code:
239
+
240
+ **For llm-based reranker**
241
+
242
+ ```shell
243
+ torchrun --nproc_per_node {number of gpus} \
244
+ -m FlagEmbedding.llm_reranker.finetune_for_instruction.run \
245
+ --output_dir {path to save model} \
246
+ --model_name_or_path BAAI/bge-reranker-v2-gemma \
247
+ --train_data ./toy_finetune_data.jsonl \
248
+ --learning_rate 2e-4 \
249
+ --num_train_epochs 1 \
250
+ --per_device_train_batch_size 1 \
251
+ --gradient_accumulation_steps 16 \
252
+ --dataloader_drop_last True \
253
+ --query_max_len 512 \
254
+ --passage_max_len 512 \
255
+ --train_group_size 16 \
256
+ --logging_steps 1 \
257
+ --save_steps 2000 \
258
+ --save_total_limit 50 \
259
+ --ddp_find_unused_parameters False \
260
+ --gradient_checkpointing \
261
+ --deepspeed stage1.json \
262
+ --warmup_ratio 0.1 \
263
+ --bf16 \
264
+ --use_lora True \
265
+ --lora_rank 32 \
266
+ --lora_alpha 64 \
267
+ --use_flash_attn True \
268
+ --target_modules q_proj k_proj v_proj o_proj
269
+ ```
270
+
271
+ **For llm-based layerwise reranker**
272
+
273
+ ```shell
274
+ torchrun --nproc_per_node {number of gpus} \
275
+ -m FlagEmbedding.llm_reranker.finetune_for_layerwise.run \
276
+ --output_dir {path to save model} \
277
+ --model_name_or_path BAAI/bge-reranker-v2-minicpm-layerwise \
278
+ --train_data ./toy_finetune_data.jsonl \
279
+ --learning_rate 2e-4 \
280
+ --num_train_epochs 1 \
281
+ --per_device_train_batch_size 1 \
282
+ --gradient_accumulation_steps 16 \
283
+ --dataloader_drop_last True \
284
+ --query_max_len 512 \
285
+ --passage_max_len 512 \
286
+ --train_group_size 16 \
287
+ --logging_steps 1 \
288
+ --save_steps 2000 \
289
+ --save_total_limit 50 \
290
+ --ddp_find_unused_parameters False \
291
+ --gradient_checkpointing \
292
+ --deepspeed stage1.json \
293
+ --warmup_ratio 0.1 \
294
+ --bf16 \
295
+ --use_lora True \
296
+ --lora_rank 32 \
297
+ --lora_alpha 64 \
298
+ --use_flash_attn True \
299
+ --target_modules q_proj k_proj v_proj o_proj \
300
+ --start_layer 8 \
301
+ --head_multi True \
302
+ --head_type simple
303
+ ```
304
+
305
+ Our rerankers are initialized from [google/gemma-2b](https://huggingface.co/google/gemma-2b) (for llm-based reranker) and [openbmb/MiniCPM-2B-dpo-fp16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp16/tree/main) (for llm-based layerwise reranker), and we train it on a mixture of multilingual datasets:
306
+
307
+ - [bge-m3-data](https://huggingface.co/datasets/Shitao/bge-m3-data)
308
+ - [quora train data](https://huggingface.co/datasets/quora)
309
+ - [fever train data](https://fever.ai/dataset/fever.html)
310
+
311
+ ## Evaluation
312
+
313
+ - llama-index.
314
+
315
+ ![image-20240317193909373](./evaluation/llama-index.png)
316
+
317
+
318
+ - BEIR.
319
+
320
+ rereank the top 100 results from bge-en-v1.5 large.
321
+
322
+ ![image-20240317174633333](./evaluation/BEIR-bge-en-v1.5.png)
323
+
324
+ rereank the top 100 results from e5 mistral 7b instruct.
325
+
326
+ ![image-20240317172949713](./evaluation/BEIR-e5-mistral.png)
327
+
328
+ - CMTEB-retrieval.
329
+ It rereank the top 100 results from bge-zh-v1.5 large.
330
+
331
+ ![image-20240317173026235](./evaluation/CMTEB-retrieval-bge-zh-v1.5.png)
332
+
333
+ - miracl (multi-language).
334
+ It rereank the top 100 results from bge-m3.
335
+
336
+ ![image-20240317173117639](./evaluation/miracl-bge-m3.png)
337
+
338
+
339
+
340
+ ## Citation
341
+
342
+ If you find this repository useful, please consider giving a star :star: and citation
343
+
344
+ ```
345
+ @misc{li2023making,
346
+ title={Making Large Language Models A Better Foundation For Dense Retrieval},
347
+ author={Chaofan Li and Zheng Liu and Shitao Xiao and Yingxia Shao},
348
+ year={2023},
349
+ eprint={2312.15503},
350
+ archivePrefix={arXiv},
351
+ primaryClass={cs.CL}
352
  }
353
+ @misc{chen2024bge,
354
+ title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
355
+ author={Jianlv Chen and Shitao Xiao and Peitian Zhang and Kun Luo and Defu Lian and Zheng Liu},
356
+ year={2024},
357
+ eprint={2402.03216},
358
+ archivePrefix={arXiv},
359
+ primaryClass={cs.CL}
360
+ }