File size: 12,774 Bytes
1912f64 cb087a3 1912f64 37ca58e 1912f64 cb087a3 1912f64 95fd58b 1912f64 37ca58e 1912f64 37ca58e 1912f64 37ca58e 1912f64 a7ef97d 1912f64 95fd58b 1912f64 cb087a3 1912f64 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 |
---
tags:
- mteb
- qihoo360
- 奇虎360
- RAG-reranking
model-index:
- name: 360Zhinao-1.8B-reranking
results:
- task:
type: Reranking
dataset:
type: None
name: MTEB CMedQAv1
config: default
split: test
revision: None
metrics:
- type: map
value: 86.75017961853382
- type: mrr
value: 89.15436507936508
- task:
type: Reranking
dataset:
type: None
name: MTEB CMedQAv2
config: default
split: test
revision: None
metrics:
- type: map
value: 87.91572151930174
- type: mrr
value: 89.98869047619048
- task:
type: Reranking
dataset:
type: None
name: MTEB MMarcoReranking
config: default
split: dev
revision: None
metrics:
- type: map
value: 37.28779203409935
- type: mrr
value: 36.23730158730159
- task:
type: Reranking
dataset:
type: None
name: MTEB T2Reranking
config: default
split: dev
revision: None
metrics:
- type: map
value: 68.55153559405632
- type: mrr
value: 79.62773774596725
license: apache-2.0
library_name: transformers
---
<br>
# MTEB Leaderboard Chinese Reranking Results
We have validated the performance of our model on the [mteb-chinese-reranking leaderboard](https://huggingface.co/spaces/mteb/leaderboard). Currently, the open-source models on this leaderboard are primarily bidirectional discriminative models (BERT-like models). The only unidirectional generative model (GPT-like model) is gte-Qwen1.5-7B-instruct, which has an average score of 66.38, ranking 25th, with less than ideal results. Our self-developed unidirectional generative model, 360Zhinao-1.8B-reranking, achieved an average score of 70.13, currently ranking first overall and first among open-source models, opening up new possibilities for generative models to undertake discriminative tasks. For more detail, please check [360zhinao](https://github.com/Qihoo360/360zhinao/tree/main).
| Model | T2Reranking | MMarcoReranking | CMedQAv1 | CMedQAv2 | Avg |
|:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|
| **360Zhinao-1.8B-Reranking** | **68.55** | **37.29** | **86.75** | **87.92** | **70.13** |
| piccolo-large-zh-v2 | 67.15 | 33.39 | 90.14 | 89.31 | 70 |
| Baichuan-text-embedding | 67.85 | 34.3 | 88.46 | 88.06 | 69.67 |
| stella-mrl-large-zh-v3.5-1792d | 66.43 | 28.85 | 89.18 | 89.33 | 68.45 |
| PEG | 69.43 | 33.55 | 86.56 | 84.09 | 68.41 |
| bge-reranker-base | 67.28 | 35.46 | 81.27 | 84.1 | 67.03 |
| bge-reranker-large | 67.6 | 37.17 | 82.14 | 84.19 | 67.78 |
# Requirements
```bash
pip install -r requirements.txt
```
If your GPU supports fp16 or bf16 precision, we also recommend installing [flash-attention](https://github.com/Dao-AILab/flash-attention) (**now with support for flash attention 2**) to improve your runtime efficiency and reduce memory usage. (**flash-attention is optional and not required for running this project**)
```bash
git clone https://github.com/Dao-AILab/flash-attention
cd flash-attention && pip install .
# The installation below is optional and might be slow.
# pip install csrc/layer_norm
# No need to install the following if the flash-attn version is above 2.1.1.
# pip install csrc/rotary
```
You can also use the following command to install flash-attention.
```bash
FLASH_ATTENTION_FORCE_BUILD=TRUE ./miniconda3/bin/python -m pip install flash-attn==2.3.6
```
# Model Introduction
The **360Zhinao-1.8B-Reranking** model utilizes the self-developed 360Zhinao-1.8b-Base model as its foundation. Through iterative discovery and resolution of the following technical issues, it continuously stimulates the world knowledge inherent in the large model during the pre-training phase, better bridging the gap between generative models and discriminative tasks.
## Data Processing
The model training did not utilize world knowledge, meaning it neither continue pre-training with domain-specific data nor fine-tuned datasets outside of the four datasets on the leaderboard. It only used the four datasets within the leaderboard, carefully iterating through data perception, data cleaning and data mining to ensure that the ranking in individual tasks could reach the top three level.
## Resolving Task Conflicts and Catastrophic Forgetting
When merging four tasks, due to different data domain distributions, answer patterns, training data volumes, convergence steps, and even sequence lengths, conflicts exist between different tasks. Deeply resolving these conflict issues is crucial to obtaining a universal model with the best comprehensive indicators across different tasks.
## Resolving Training Instability
Unlike generative tasks that produce multiple characters, using generative models for discriminative tasks requires the model to output a continuous value. Therefore, there is an oscillation problem during the training process. Deeply analyzing and resolving training instability can result in a model with better generalization and robustness.
# Inference Script
You can copy the following scripts to [mteb-eval-scripts](https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB), then replace FlagReranker with FlagRerankerCustom in [eval_cross_encoder](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/eval_cross_encoder.py) scripts, then run [eval_cross_encoder](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/eval_cross_encoder.py) to reproduce our complete performance on the [mteb-chinese-reranking leaderboard](https://huggingface.co/spaces/mteb/leaderboard).
```python
from typing import cast, List, Union, Tuple, Dict, Optional
import numpy as np
import torch
from tqdm import tqdm
from transformers import AutoModel, AutoTokenizer, AutoModelForSequenceClassification
import transformers
from transformers.trainer_pt_utils import LabelSmoother
IGNORE_TOKEN_ID = LabelSmoother.ignore_index
def preprocess(
sources,
tokenizer: transformers.PreTrainedTokenizer,
max_len: int = 1024,
system_message: str = "",
device = None,
) -> Dict:
roles = {"user": "<|im_start|>user", "assistant": "<|im_start|>assistant"}
answer_len = 64
im_start = tokenizer.im_start_id
im_end = tokenizer.im_end_id
nl_tokens = tokenizer('\n').input_ids
_system = tokenizer('system').input_ids + nl_tokens
_user = tokenizer('user').input_ids + nl_tokens
_assistant = tokenizer('assistant').input_ids + nl_tokens
# Apply prompt templates
input_ids, targets = [], []
for i, source in enumerate(sources):
## system_message
input_id, target = [], []
system = [im_start] + _system + tokenizer(system_message, max_length=max_len-answer_len, truncation=True).input_ids + [im_end] + nl_tokens
input_id += system
target += [im_start] + [IGNORE_TOKEN_ID] * (len(system)-3) + [im_end] + nl_tokens
assert len(input_id) == len(target)
## query ans
source = "\n\n".join(source)
role = "<|im_start|>user"
_input_id = tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids + nl_tokens + \
tokenizer(source, max_length=max_len-answer_len, truncation=True).input_ids + [im_end] + nl_tokens
input_id += _input_id
if role == '<|im_start|>user':
_target = [im_start] + [IGNORE_TOKEN_ID] * (len(_input_id)-3) + [im_end] + nl_tokens
elif role == '<|im_start|>assistant':
_target = [im_start] + [IGNORE_TOKEN_ID] * len(tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids) + \
_input_id[len(tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids)+1:-2] + [im_end] + nl_tokens
else:
raise NotImplementedError
target += _target
## label use placeholder 0; It will be masked later in the modeling_zhinao.py
role = "<|im_start|>assistant"
_input_id = tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids + nl_tokens + \
tokenizer("0", max_length=max_len-answer_len, truncation=True).input_ids + [im_end] + nl_tokens
input_id += _input_id
if role == '<|im_start|>user':
_target = [im_start] + [IGNORE_TOKEN_ID] * (len(_input_id)-3) + [im_end] + nl_tokens
elif role == '<|im_start|>assistant':
_target = [im_start] + [IGNORE_TOKEN_ID] * len(tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids) + \
_input_id[len(tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids)+1:-2] + [im_end] + nl_tokens
else:
raise NotImplementedError
target += _target
assert len(input_id) == len(target)
input_id += [tokenizer.pad_token_id] * (max_len - len(input_id))
target += [IGNORE_TOKEN_ID] * (max_len - len(target))
if len(input_id) > max_len:
print("max_len_error")
print(tokenizer.decode(input_id))
input_ids.append(input_id[:max_len])
targets.append(target[:max_len])
input_ids = torch.tensor(input_ids, dtype=torch.int)
targets = torch.tensor(targets, dtype=torch.int)
#print(f"input_ids {input_ids.shape}")
#print(f"targets {targets.shape}")
return dict(
input_ids=input_ids.to(device),
labels=targets.to(device),
attention_mask=input_ids.ne(tokenizer.pad_token_id).to(device),
)
class FlagRerankerCustom:
def __init__(
self,
model_name_or_path: str = None,
use_fp16: bool = False
) -> None:
self.tokenizer = transformers.AutoTokenizer.from_pretrained(
model_name_or_path,
model_max_length=1024,
padding_side="right",
use_fast=False,
trust_remote_code=True
)
self.tokenizer.pad_token_id = self.tokenizer.eod_id
config = transformers.AutoConfig.from_pretrained(
model_name_or_path,
trust_remote_code=True,
bf16=True,
)
config.use_cache = False
self.model = transformers.AutoModelForCausalLM.from_pretrained(
model_name_or_path,
config=config,
trust_remote_code=True,
)
self.model.linear.bfloat16()
if torch.cuda.is_available():
self.device = torch.device('cuda')
elif torch.backends.mps.is_available():
self.device = torch.device('mps')
else:
self.device = torch.device('cpu')
use_fp16 = False
if use_fp16:
self.model.half()
self.model = self.model.to(self.device)
self.model.eval()
self.num_gpus = torch.cuda.device_count()
if self.num_gpus > 1:
print(f"----------using {self.num_gpus}*GPUs----------")
self.model = torch.nn.DataParallel(self.model)
@torch.no_grad()
def compute_score(self, sentence_pairs: Union[List[Tuple[str, str]], Tuple[str, str]], batch_size: int =128,
max_length: int = 1024) -> List[float]:
if self.num_gpus > 0:
batch_size = batch_size * self.num_gpus
assert isinstance(sentence_pairs, list)
if isinstance(sentence_pairs[0], str):
sentence_pairs = [sentence_pairs]
all_scores = []
for start_index in tqdm(range(0, len(sentence_pairs), batch_size), desc="Compute Scores",
disable=False):
sentences_batch = sentence_pairs[start_index:start_index + batch_size] # [[q,ans],[q, ans]...]
inputs = preprocess(sources=sentences_batch, tokenizer=self.tokenizer,max_len=1024,device=self.device)
scores = self.model(**inputs, return_dict=True).logits.view(-1, ).float()
all_scores.extend(scores.cpu().numpy().tolist())
if len(all_scores) == 1:
return all_scores[0]
return all_scores
if __name__ == "__main__":
model_name_or_path = "360Zhinao-1.8B-Reranking"
model = FlagRerankerCustom(model_name_or_path, use_fp16=False)
inputs=[["What Color Is the Sky","Blue"], ["What Color Is the Sky","Pink"],]
ret = model.compute_score(inputs)
print(ret)
```
## License
The source code of this repository follows the open-source license Apache 2.0.
360Zhinao open-source models support commercial use. If you wish to use these models or continue training them for commercial purposes, please contact us via email ([email protected]) to apply. For the specific license agreement, please see <<360 Zhinao Open-Source Model License>>.
|