Update README.md

README.md

@@ -1,3 +1,81 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+---
+
+**English** | [中文](./README_zh.md)
+
+<!-- [Arxiv PDF](https://arxiv.org/pdf/2407.19669), [HF paper page](https://huggingface.co/papers/2407.19669)
+ -->
+
+## Code implementation of Qwen2 based embeddings
+
+This model code is for Qwen2 based embedding models.
+
+We enable the bidirectional attention by default.
+
+### Usage
+
+1. Download the `configuration.py` and `modeling.py` to your saved `gte-Qwen2` model directory.
+2. Replace the `modeling_qwen.` with `modeling.` in `auto_map` field of `config.json`.
+
+
+### Recommendation: Enable Unpadding and Acceleration with `xformers`
+
+This code supports the acceleration of attention computations using `xformers`,
+which can automatically choose the optimal implementation based on the type of device, such as `flash_attn`.
+Therefore, we can also achieve significant acceleration on old devices like the V100.
+
+
+Firstly, install `xformers` (with `pytorch` pre-installed):
+```
+if pytorch is installed using conda:
+    conda install xformers -c xformers
+elif pytorch is installed using pip:
+    # cuda 11.8 version
+    pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118
+    # cuda 12.1 version
+    pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu121
+```
+For more information, refer to [Installing xformers](https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers).
+
+Then, when loading the model, set `unpad_inputs` and `use_memory_efficient_attention` to `true`,
+and set `torch_dtype` to `torch.float16` (or `torch.bfloat16`) to achieve the acceleration.
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+path = 'Alibaba-NLP/gte-Qwen2-1.5B-instruct'
+device = torch.device('cuda')
+tokenzier = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
+model = AutoModel.from_pretrained(
+    path,
+    trust_remote_code=True,
+    unpad_inputs=True,
+    use_memory_efficient_attention=True,
+    torch_dtype=torch.float16
+).to(device)
+
+inputs = tokenzier(['test input'], truncation=True, max_length=8192, padding=True, return_tensors='pt')
+
+with torch.inference_mode():
+    outputs = model(**inputs.to(device))
+
+```
+
+Alternatively, you can directly modify the `unpad_inputs` and `use_memory_efficient_attention` settings to `true` in the model's `config.json`,
+eliminating the need to set them in the code.
+
+
+## Citation
+```
+@misc{zhang2024mgte,
+  title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval}, 
+  author={Xin Zhang and Yanzhao Zhang and Dingkun Long and Wen Xie and Ziqi Dai and Jialong Tang and Huan Lin and Baosong Yang and Pengjun Xie and Fei Huang and Meishan Zhang and Wenjie Li and Min Zhang},
+  year={2024},
+  eprint={2407.19669},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL},
+  url={https://arxiv.org/abs/2407.19669}, 
+}
+```

README_zh.md

@@ -0,0 +1,76 @@
+---
+license: apache-2.0
+---
+
+[English](./README.md) | **中文**
+
+<!-- [Arxiv PDF](https://arxiv.org/pdf/2407.19669), [HF paper page](https://huggingface.co/papers/2407.19669) -->
+
+## Qwen2 模型代码实现
+
+此模型代码适用于基于`Qwen2` 的文本表示模型。
+
+默认启用双向注意力机制。
+
+
+### 使用方法
+
+1. 下载此仓库中的 `configuration.py` 和 `modeling.py` 到你本地保存的 `gte-Qwen2` 模型目录
+2. 将 `config.json` 的 `auto_map` 下所有的 `modeling_qwen.` 替换为 `modeling.`
+
+### 推荐：启用 Unpadding 和 xformers 加速
+
+此代码支持使用 `xformers` 加速 attention 计算，可以根据设备类型自动选择优化实现，比如 `flash_attn`。
+通过 `xformers`，在不能支持 `flash_attn` 的旧设备比如`V100`上也可以获得极大的加速。
+
+首先，安装 `xformers`（需要预先安装`pytorch`）：
+```
+if pytorch 使用 conda 安装 :
+    conda install xformers -c xformers
+
+elif pytorch 使用 pip 安装 :
+    # cuda 11.8 version
+    pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118
+    # cuda 12.1 version
+    pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu121
+```
+更多信息可参考 [installing-xformers](https://github.com/facebookresearch/xformers?tab=readme-ov-file#installing-xformers)。
+
+然后，加载模型时设置 `unpad_inputs` 和 `use_memory_efficient_attention` 为 `true`，并设置 `torch_dtype` 为 `torch.float16` (or `torch.bfloat16`)，即可获得加速。
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+path = 'Alibaba-NLP/gte-Qwen2-1.5B-instruct'
+device = torch.device('cuda')
+tokenzier = AutoTokenizer.from_pretrained(path)
+model = AutoModel.from_pretrained(
+    path,
+    trust_remote_code=True,
+    unpad_inputs=True,
+    use_memory_efficient_attention=True,
+    torch_dtype=torch.float16
+).to(device)
+
+inputs = tokenzier(['test input'], truncation=True, max_length=8192, padding=True, return_tensors='pt')
+
+with torch.inference_mode():
+    outputs = model(**inputs.to(device))
+
+```
+也可以直接修改模型的 `config.json` 中 `unpad_inputs` 和 `use_memory_efficient_attention` 为 `true`，省去代码中的设置。
+
+
+## Citation
+```
+@misc{zhang2024mgte,
+  title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval}, 
+  author={Xin Zhang and Yanzhao Zhang and Dingkun Long and Wen Xie and Ziqi Dai and Jialong Tang and Huan Lin and Baosong Yang and Pengjun Xie and Fei Huang and Meishan Zhang and Wenjie Li and Min Zhang},
+  year={2024},
+  eprint={2407.19669},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL},
+  url={https://arxiv.org/abs/2407.19669}, 
+}
+```