xun commited on
Commit
8bf83e9
1 Parent(s): 91ea42d
LICENSE ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Tongyi Qianwen LICENSE AGREEMENT
2
+
3
+ Tongyi Qianwen Release Date: August 3, 2023
4
+
5
+ By clicking to agree or by using or distributing any portion or element of the Tongyi Qianwen Materials, you will be deemed to have recognized and accepted the content of this Agreement, which is effective immediately.
6
+
7
+ 1. Definitions
8
+ a. This Tongyi Qianwen LICENSE AGREEMENT (this "Agreement") shall mean the terms and conditions for use, reproduction, distribution and modification of the Materials as defined by this Agreement.
9
+ b. "We"(or "Us") shall mean Alibaba Cloud.
10
+ c. "You" (or "Your") shall mean a natural person or legal entity exercising the rights granted by this Agreement and/or using the Materials for any purpose and in any field of use.
11
+ d. "Third Parties" shall mean individuals or legal entities that are not under common control with Us or You.
12
+ e. "Tongyi Qianwen" shall mean the large language models (including Qwen model and Qwen-Chat model), and software and algorithms, consisting of trained model weights, parameters (including optimizer states), machine-learning model code, inference-enabling code, training-enabling code, fine-tuning enabling code and other elements of the foregoing distributed by Us.
13
+ f. "Materials" shall mean, collectively, Alibaba Cloud's proprietary Tongyi Qianwen and Documentation (and any portion thereof) made available under this Agreement.
14
+ g. "Source" form shall mean the preferred form for making modifications, including but not limited to model source code, documentation source, and configuration files.
15
+ h. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation,
16
+ and conversions to other media types.
17
+
18
+ 2. Grant of Rights
19
+ You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Alibaba Cloud's intellectual property or other rights owned by Us embodied in the Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Materials.
20
+
21
+ 3. Redistribution
22
+ You may reproduce and distribute copies of the Materials or derivative works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
23
+ a. You shall give any other recipients of the Materials or derivative works a copy of this Agreement;
24
+ b. You shall cause any modified files to carry prominent notices stating that You changed the files;
25
+ c. You shall retain in all copies of the Materials that You distribute the following attribution notices within a "Notice" text file distributed as a part of such copies: "Tongyi Qianwen is licensed under the Tongyi Qianwen LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved."; and
26
+ d. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such derivative works as a whole, provided Your use, reproduction, and distribution of the work otherwise complies with the terms and conditions of this Agreement.
27
+
28
+ 4. Restrictions
29
+ If you are commercially using the Materials, and your product or service has more than 100 million monthly active users, You shall request a license from Us. You cannot exercise your rights under this Agreement without our express authorization.
30
+
31
+ 5. Rules of use
32
+ a. The Materials may be subject to export controls or restrictions in China, the United States or other countries or regions. You shall comply with applicable laws and regulations in your use of the Materials.
33
+ b. You can not use the Materials or any output therefrom to improve any other large language model (excluding Tongyi Qianwen or derivative works thereof).
34
+
35
+ 6. Intellectual Property
36
+ a. We retain ownership of all intellectual property rights in and to the Materials and derivatives made by or for Us. Conditioned upon compliance with the terms and conditions of this Agreement, with respect to any derivative works and modifications of the Materials that are made by you, you are and will be the owner of such derivative works and modifications.
37
+ b. No trademark license is granted to use the trade names, trademarks, service marks, or product names of Us, except as required to fulfill notice requirements under this Agreement or as required for reasonable and customary use in describing and redistributing the Materials.
38
+ c. If you commence a lawsuit or other proceedings (including a cross-claim or counterclaim in a lawsuit) against Us or any entity alleging that the Materials or any output therefrom, or any part of the foregoing, infringe any intellectual property or other right owned or licensable by you, then all licences granted to you under this Agreement shall terminate as of the date such lawsuit or other proceeding is commenced or brought.
39
+
40
+ 7. Disclaimer of Warranty and Limitation of Liability
41
+
42
+ a. We are not obligated to support, update, provide training for, or develop any further version of the Tongyi Qianwen Materials or to grant any license thereto.
43
+ b. THE MATERIALS ARE PROVIDED "AS IS" WITHOUT ANY EXPRESS OR IMPLIED WARRANTY OF ANY KIND INCLUDING WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. WE MAKE NO WARRANTY AND ASSUME NO RESPONSIBILITY FOR THE SAFETY OR STABILITY OF THE MATERIALS AND ANY OUTPUT THEREFROM.
44
+ c. IN NO EVENT SHALL WE BE LIABLE TO YOU FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO ANY DIRECT, OR INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES ARISING FROM YOUR USE OR INABILITY TO USE THE MATERIALS OR ANY OUTPUT OF IT, NO MATTER HOW IT’S CAUSED.
45
+ d. You will defend, indemnify and hold harmless Us from and against any claim by any third party arising out of or related to your use or distribution of the Materials.
46
+
47
+ 8. Survival and Termination.
48
+ a. The term of this Agreement shall commence upon your acceptance of this Agreement or access to the Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein.
49
+ b. We may terminate this Agreement if you breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, you must delete and cease use of the Materials. Sections 7 and 9 shall survive the termination of this Agreement.
50
+
51
+ 9. Governing Law and Jurisdiction.
52
+ a. This Agreement and any dispute arising out of or relating to it will be governed by the laws of China, without regard to conflict of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement.
53
+ b. The People's Courts in Hangzhou City shall have exclusive jurisdiction over any dispute arising out of this Agreement.
README.md ADDED
@@ -0,0 +1,136 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - zh
4
+ - en
5
+ tags:
6
+ - qwen
7
+ - multimodal
8
+ - 音频理解
9
+ license: other
10
+ pipeline_tag: text-generation
11
+ inference: false
12
+ ---
13
+
14
+ # Qwen-Audio-Chat
15
+
16
+ <br>
17
+
18
+ <p align="center">
19
+ <img src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Audio/audio_logo.jpg" width="400"/>
20
+ <p>
21
+ <br>
22
+
23
+ <p align="center">
24
+ Qwen-Audio <a href="https://www.modelscope.cn/models/qwen/QWen-Audio/summary">🤖 <a> | <a href="https://huggingface.co/Qwen/Qwen-Audio">🤗</a>&nbsp | Qwen-Audio-Chat <a href="https://www.modelscope.cn/models/qwen/QWen-Audio-Chat/summary">🤖 <a>| <a href="https://huggingface.co/Qwen/Qwen-Audio-Chat">🤗</a>&nbsp
25
+ <br>
26
+ &nbsp&nbsp<a href="https://modelscope.cn/studios/qwen/Qwen-Audio-Chat-Demo/summary">Demo</a>&nbsp | &nbsp&nbsp<a href="https://qwen-audio.github.io/Qwen-Audio/">Homepage</a>&nbsp | &nbsp<a href="http://arxiv.org/abs/2311.07919">Paper</a>
27
+ </p>
28
+ <br><br>
29
+
30
+ **Qwen-Audio** (Qwen Large Audio Language Model) is the multimodal version of the large model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-Audio accepts diverse audio (human speech, natural sound, music and song) and text as inputs, outputs text. The contribution of Qwen-Audio include:
31
+
32
+ - **Fundamental audio models**: Qwen-Audio is a fundamental multi-task audio-language model that supports various tasks, languages, and audio types, serving as a universal audio understanding model. Building upon Qwen-Audio, we develop Qwen-Audio-Chat through instruction fine-tuning, enabling multi-turn dialogues and supporting diverse audio-oriented scenarios.
33
+ - **Multi-task learning framework for all types of audios**: To scale up audio-language pre-training, we address the challenge of variation in textual labels associated with different datasets by proposing a multi-task training framework, enabling knowledge sharing and avoiding one-to-many interference. Our model incorporates more than 30 tasks and extensive experiments show the model achieves strong performance.
34
+ - **Strong Performance**: Experimental results show that Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning, surpassing its counterparts. Specifically, Qwen-Audio achieves state-of-the-art results on the test set of Aishell1, cochlscene, ClothoAQA, and VocalSound.
35
+ - **Flexible multi-run chat from audio and text input**: Qwen-Audio supports multiple-audio analysis, sound understading and reasoning, music appreciation, and tool usage for speech editing.
36
+
37
+ **Qwen-Audio** 是阿里云研发的大规模音频语言模型(Large Audio Language Model)。Qwen-Audio 可以以多种音频 (包括说话人语音、自然音、音乐、歌声)和文本作为输入,并以文本作为输出。Qwen-Audio 系列模型的特点包括:
38
+
39
+ - **音频基石模型**:Qwen-Audio是一个性能卓越的通用的音频理解模型,支持各种任务、语言和音频类型。在Qwen-Audio的基础上,我们通过指令微调开发了Qwen-Audio-Chat,支持多轮、多语言、多语言对话。Qwen-Audio和Qwen-Audio-Chat模型均已开源。
40
+ - **兼容多种复杂音频的多任务学习框架**:为了避免由于数据收集来源不同以及任务类型不同,带来的音频到文本的一对多的干扰问题,我们提出了一种多任务训练框架,实现相似任务的知识共享,并尽可能减少不同任务之间的干扰。通过提出的框架,Qwen-Audio可以容纳训练超过30多种不同的音频任务;
41
+ - **出色的性能**:Qwen-Audio在不需要任何任务特定的微调的情况下,在各种基准任务上取得了领先的结果。具体得,Qwen-Audio在Aishell1、cochlscene、ClothoAQA和VocalSound的测试集上都达到了SOTA;
42
+ - **支持多轮音频和文本对话,支持各种语音场景**:Qwen-Audio-Chat支持声音理解和推理、音乐欣赏、多音频分析、多轮音频-文本交错对话以及外部语音工具的使用(如语音编辑)。
43
+
44
+
45
+ We release Qwen-Audio and Qwen-Audio-Chat, which are pretrained model and Chat model respectively. For more details about Qwen-Audio, please refer to our [Github Repo](https://github.com/QwenLM/Qwen-Audio/tree/main). This repo is the one for Qwen-Audio-Chat.
46
+ <br>
47
+
48
+ 目前,我们提供了Qwen-Audio和Qwen-Audio-Chat两个模型,分别为预训练模型和Chat模型。如果想了解更多关于信息,请点击[链接](https://github.com/QwenLM/Qwen-Audio/tree/main)查看Github仓库。本仓库为Qwen-Audio-Chat仓库。
49
+
50
+
51
+ ## Requirements
52
+ * python 3.8 and above
53
+ * pytorch 1.12 and above, 2.0 and above are recommended
54
+ * CUDA 11.4 and above are recommended (this is for GPU users)
55
+ * FFmpeg
56
+ <br>
57
+
58
+ ## Quickstart
59
+ Below, we provide simple examples to show how to use Qwen-Audio with 🤗 Transformers.
60
+
61
+ Before running the code, make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries.
62
+
63
+ ```bash
64
+ pip install -r requirements.txt
65
+ ```
66
+ Now you can start with Transformers. For more usage, please refer to [tutorial](https://github.com/QwenLM/Qwen-Audio/blob/main/TUTORIAL.md).
67
+
68
+ #### 🤖 ModelScope
69
+
70
+ To use Qwen-Audio for the inference, all you need to do is to input a few lines of codes as demonstrated below. However, **please make sure that you are using the latest code.**
71
+
72
+ ```python
73
+ from modelscope import (
74
+ snapshot_download, AutoModelForCausalLM, AutoTokenizer, GenerationConfig
75
+ )
76
+ import torch
77
+ model_id = 'qwen/Qwen-Audio-Chat'
78
+ revision = 'master'
79
+
80
+ model_dir = snapshot_download(model_id, revision=revision)
81
+ torch.manual_seed(1234)
82
+
83
+ tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
84
+ if not hasattr(tokenizer, 'model_dir'):
85
+ tokenizer.model_dir = model_dir
86
+
87
+ # Note: The default behavior now has injection attack prevention off.
88
+ tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
89
+
90
+ # use bf16
91
+ # model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="auto", trust_remote_code=True, bf16=True).eval()
92
+ # use fp16
93
+ # model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="auto", trust_remote_code=True, fp16=True).eval()
94
+ # use cpu only
95
+ # model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="cpu", trust_remote_code=True).eval()
96
+ # use cuda device
97
+ model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="cuda", trust_remote_code=True).eval()
98
+
99
+
100
+ # 1st dialogue turn
101
+ query = tokenizer.from_list_format([
102
+ {'audio': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Audio/1272-128104-0000.flac'}, # Either a local path or an url
103
+ {'text': 'what does the person say?'},
104
+ ])
105
+ response, history = model.chat(tokenizer, query=query, history=None)
106
+ print(response)
107
+ # The person says: "mister quilter is the apostle of the middle classes and we are glad to welcome his gospel".
108
+
109
+ # 2nd dialogue turn
110
+ response, history = model.chat(tokenizer, 'Find the start time and end time of the word "middle classes"', history=history)
111
+ print(response)
112
+ # The word "middle classes" starts at <|2.33|> seconds and ends at <|3.26|> seconds.
113
+ ```
114
+
115
+
116
+ ## License Agreement
117
+ Researchers and developers are free to use the codes and model weights of Qwen-Audio-Chat. We also allow its commercial use. Check our license at [LICENSE](LICENSE) for more details.
118
+ <br>
119
+
120
+ ## Citation
121
+ If you find our paper and code useful in your research, please consider giving a star and citation
122
+
123
+ ```BibTeX
124
+ @article{Qwen-Audio,
125
+ title={Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models},
126
+ author={Chu, Yunfei and Xu, Jin and Zhou, Xiaohuan and Yang, Qian and Zhang, Shiliang and Yan, Zhijie and Zhou, Chang and Zhou, Jingren},
127
+ journal={arXiv preprint arXiv:2311.07919},
128
+ year={2023}
129
+ }
130
+ ```
131
+ <br>
132
+
133
+ ## Contact Us
134
+
135
+ If you are interested to leave a message to either our research team or product team, feel free to send an email to [email protected].
136
+
audio.py ADDED
@@ -0,0 +1,443 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import base64
2
+ import gzip
3
+ from dataclasses import dataclass
4
+ from typing import Dict, Iterable, Optional, List
5
+
6
+ import numpy as np
7
+ import torch
8
+ import torch.nn.functional as F
9
+ from torch import Tensor, nn
10
+ from subprocess import CalledProcessError, run, Popen, PIPE
11
+
12
+ import os
13
+ from functools import lru_cache
14
+ from typing import Optional, Union
15
+
16
+ def exact_div(x, y):
17
+ assert x % y == 0
18
+ return x // y
19
+
20
+ # hard-coded audio hyperparameters
21
+ SAMPLE_RATE = 16000
22
+ N_FFT = 400
23
+ N_MELS = 80
24
+ HOP_LENGTH = 160
25
+ CHUNK_LENGTH = 30
26
+ N_SAMPLES = CHUNK_LENGTH * SAMPLE_RATE # 480000 samples in a 30-second chunk
27
+ N_FRAMES = exact_div(N_SAMPLES, HOP_LENGTH) # 3000 frames in a mel spectrogram input
28
+
29
+ N_SAMPLES_PER_TOKEN = HOP_LENGTH * 2 # the initial convolutions has stride 2
30
+ FRAMES_PER_SECOND = exact_div(SAMPLE_RATE, HOP_LENGTH) # 10ms per audio frame
31
+ TOKENS_PER_SECOND = exact_div(SAMPLE_RATE, N_SAMPLES_PER_TOKEN) # 20ms per audio token
32
+
33
+
34
+
35
+ def get_T_after_cnn(L_in, dilation=1):
36
+ for (padding, kernel_size, stride) in eval("[(1,3,1)] + [(1,3,2)] "):
37
+ L_out = L_in + 2 * padding - dilation * (kernel_size - 1) - 1
38
+ L_out = 1 + L_out // stride
39
+ L_in = L_out
40
+ return L_out
41
+
42
+ def load_bytesio_audio(content, sr: int = SAMPLE_RATE):
43
+ cmd = [
44
+ "ffmpeg",
45
+ "-nostdin",
46
+ "-threads", "0",
47
+ "-i", "pipe:",
48
+ "-f", "s16le",
49
+ "-ac", "1",
50
+ "-acodec", "pcm_s16le",
51
+ "-ar", str(sr),
52
+ "pipe:"
53
+ ]
54
+ p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, bufsize=-1)
55
+ out, _ = p.communicate(input=content)
56
+ return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0
57
+
58
+ def load_audio(file: str, sr: int = SAMPLE_RATE):
59
+ """
60
+ Open an audio file and read as mono waveform, resampling as necessary
61
+
62
+ Parameters
63
+ ----------
64
+ file: str
65
+ The audio file to open
66
+
67
+ sr: int
68
+ The sample rate to resample the audio if necessary
69
+
70
+ Returns
71
+ -------
72
+ A NumPy array containing the audio waveform, in float32 dtype.
73
+ """
74
+
75
+ # This launches a subprocess to decode audio while down-mixing
76
+ # and resampling as necessary. Requires the ffmpeg CLI in PATH.
77
+ # fmt: off
78
+ cmd = [
79
+ "ffmpeg",
80
+ "-nostdin",
81
+ "-threads", "0",
82
+ "-i", file,
83
+ "-f", "s16le",
84
+ "-ac", "1",
85
+ "-acodec", "pcm_s16le",
86
+ "-ar", str(sr),
87
+ "-"
88
+ ]
89
+ # fmt: on
90
+ try:
91
+ out = run(cmd, capture_output=True, check=True).stdout
92
+ except CalledProcessError as e:
93
+ raise RuntimeError(f"Failed to load audio: {e.stderr.decode()}") from e
94
+
95
+ return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0
96
+
97
+
98
+ def pad_or_trim(array, length: int = N_SAMPLES, *, axis: int = -1):
99
+ """
100
+ Pad or trim the audio array to N_SAMPLES, as expected by the encoder.
101
+ """
102
+ if torch.is_tensor(array):
103
+ if array.shape[axis] > length:
104
+ array = array.index_select(
105
+ dim=axis, index=torch.arange(length, device=array.device)
106
+ )
107
+
108
+ if array.shape[axis] < length:
109
+ pad_widths = [(0, 0)] * array.ndim
110
+ pad_widths[axis] = (0, length - array.shape[axis])
111
+ array = F.pad(array, [pad for sizes in pad_widths[::-1] for pad in sizes])
112
+ else:
113
+ if array.shape[axis] > length:
114
+ array = array.take(indices=range(length), axis=axis)
115
+
116
+ if array.shape[axis] < length:
117
+ pad_widths = [(0, 0)] * array.ndim
118
+ pad_widths[axis] = (0, length - array.shape[axis])
119
+ array = np.pad(array, pad_widths)
120
+
121
+ return array
122
+
123
+ def trim(array, length: int = N_SAMPLES, *, axis: int = -1):
124
+ """
125
+ Pad or trim the audio array to N_SAMPLES, as expected by the encoder.
126
+ """
127
+ if torch.is_tensor(array):
128
+ if array.shape[axis] > length:
129
+ array = array.index_select(
130
+ dim=axis, index=torch.arange(length, device=array.device)
131
+ )
132
+ else:
133
+ if array.shape[axis] > length:
134
+ array = array.take(indices=range(length), axis=axis)
135
+ return array
136
+
137
+
138
+ @lru_cache(maxsize=None)
139
+ def mel_filters(device, n_mels: int = N_MELS) -> torch.Tensor:
140
+ """
141
+ load the mel filterbank matrix for projecting STFT into a Mel spectrogram.
142
+ Allows decoupling librosa dependency; saved using:
143
+
144
+ np.savez_compressed(
145
+ "mel_filters.npz",
146
+ mel_80=librosa.filters.mel(sr=16000, n_fft=400, n_mels=80),
147
+ )
148
+ """
149
+ assert n_mels == 80, f"Unsupported n_mels: {n_mels}"
150
+ with np.load(
151
+ os.path.join(os.path.dirname(__file__), "mel_filters.npz") # todo
152
+ # os.path.join("assets", "mel_filters.npz")
153
+ ) as f:
154
+ return torch.from_numpy(f[f"mel_{n_mels}"]).to(device)
155
+
156
+
157
+ def log_mel_spectrogram(
158
+ audio: Union[str, np.ndarray, torch.Tensor],
159
+ n_mels: int = N_MELS,
160
+ padding: int = 0,
161
+ device: Optional[Union[str, torch.device]] = None,
162
+ ):
163
+ """
164
+ Compute the log-Mel spectrogram of
165
+
166
+ Parameters
167
+ ----------
168
+ audio: Union[str, np.ndarray, torch.Tensor], shape = (*)
169
+ The path to audio or either a NumPy array or Tensor containing the audio waveform in 16 kHz
170
+
171
+ n_mels: int
172
+ The number of Mel-frequency filters, only 80 is supported
173
+
174
+ padding: int
175
+ Number of zero samples to pad to the right
176
+
177
+ device: Optional[Union[str, torch.device]]
178
+ If given, the audio tensor is moved to this device before STFT
179
+
180
+ Returns
181
+ -------
182
+ torch.Tensor, shape = (80, n_frames)
183
+ A Tensor that contains the Mel spectrogram
184
+ """
185
+ if not torch.is_tensor(audio):
186
+ if isinstance(audio, str):
187
+ audio = load_audio(audio)
188
+ audio = torch.from_numpy(audio)
189
+
190
+ if device is not None:
191
+ audio = audio.to(device)
192
+ if padding > 0:
193
+ audio = F.pad(audio, (0, padding))
194
+ window = torch.hann_window(N_FFT).to(audio.device)
195
+ stft = torch.stft(audio, N_FFT, HOP_LENGTH, window=window, return_complex=True)
196
+ magnitudes = stft[..., :-1].abs() ** 2
197
+
198
+ filters = mel_filters(audio.device, n_mels)
199
+ mel_spec = filters @ magnitudes
200
+
201
+ log_spec = torch.clamp(mel_spec, min=1e-10).log10()
202
+ log_spec = torch.maximum(log_spec, log_spec.max() - 8.0)
203
+ log_spec = (log_spec + 4.0) / 4.0
204
+ return log_spec
205
+
206
+
207
+ @dataclass
208
+ class ModelDimensions:
209
+ n_mels: int
210
+ n_audio_ctx: int
211
+ n_audio_state: int
212
+ n_audio_head: int
213
+ n_audio_layer: int
214
+ n_vocab: int
215
+ n_text_ctx: int
216
+ n_text_state: int
217
+ n_text_head: int
218
+ n_text_layer: int
219
+
220
+
221
+ class LayerNorm(nn.LayerNorm):
222
+ def forward(self, x: Tensor) -> Tensor:
223
+ # return super().forward(x.float()).type(x.dtype)
224
+ return super().forward(x).type(x.dtype)
225
+
226
+
227
+
228
+
229
+ class Linear(nn.Linear):
230
+ def forward(self, x: Tensor) -> Tensor:
231
+ return F.linear(
232
+ x,
233
+ self.weight.to(x.dtype),
234
+ None if self.bias is None else self.bias.to(x.dtype),
235
+ )
236
+
237
+
238
+ class Conv1d(nn.Conv1d):
239
+ def _conv_forward(
240
+ self, x: Tensor, weight: Tensor, bias: Optional[Tensor]
241
+ ) -> Tensor:
242
+ return super()._conv_forward(
243
+ x, weight.to(x.dtype), None if bias is None else bias.to(x.dtype)
244
+ )
245
+
246
+
247
+ def sinusoids(length, channels, max_timescale=10000):
248
+ """Returns sinusoids for positional embedding"""
249
+ assert channels % 2 == 0
250
+ log_timescale_increment = np.log(max_timescale) / (channels // 2 - 1)
251
+ inv_timescales = torch.exp(-log_timescale_increment * torch.arange(channels // 2))
252
+ scaled_time = torch.arange(length)[:, np.newaxis] * inv_timescales[np.newaxis, :]
253
+ return torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], dim=1)
254
+
255
+
256
+ class MultiHeadAttention(nn.Module):
257
+ def __init__(self, n_state: int, n_head: int):
258
+ super().__init__()
259
+ self.n_head = n_head
260
+ self.query = Linear(n_state, n_state)
261
+ self.key = Linear(n_state, n_state, bias=False)
262
+ self.value = Linear(n_state, n_state)
263
+ self.out = Linear(n_state, n_state)
264
+
265
+ def forward(
266
+ self,
267
+ x: Tensor,
268
+ xa: Optional[Tensor] = None,
269
+ mask: Optional[Tensor] = None,
270
+ kv_cache: Optional[dict] = None,
271
+ ):
272
+ q = self.query(x)
273
+
274
+ if kv_cache is None or xa is None or self.key not in kv_cache:
275
+ # hooks, if installed (i.e. kv_cache is not None), will prepend the cached kv tensors;
276
+ # otherwise, perform key/value projections for self- or cross-attention as usual.
277
+ k = self.key(x if xa is None else xa)
278
+ v = self.value(x if xa is None else xa)
279
+ else:
280
+ # for cross-attention, calculate keys and values once and reuse in subsequent calls.
281
+ k = kv_cache[self.key]
282
+ v = kv_cache[self.value]
283
+
284
+ wv, qk = self.qkv_attention(q, k, v, mask)
285
+ return self.out(wv), qk
286
+
287
+ def qkv_attention(
288
+ self, q: Tensor, k: Tensor, v: Tensor, mask: Optional[Tensor] = None
289
+ ):
290
+ n_batch, n_ctx, n_state = q.shape
291
+ scale = (n_state // self.n_head) ** -0.25
292
+ q = q.view(*q.shape[:2], self.n_head, -1).permute(0, 2, 1, 3) * scale
293
+ k = k.view(*k.shape[:2], self.n_head, -1).permute(0, 2, 3, 1) * scale
294
+ v = v.view(*v.shape[:2], self.n_head, -1).permute(0, 2, 1, 3)
295
+
296
+ qk = q @ k
297
+ if mask is not None:
298
+ qk += mask
299
+
300
+ w = F.softmax(qk, dim=-1).to(q.dtype)
301
+ return (w @ v).permute(0, 2, 1, 3).flatten(start_dim=2), qk.detach()
302
+
303
+
304
+ class ResidualAttentionBlock(nn.Module):
305
+ def __init__(self, n_state: int, n_head: int, cross_attention: bool = False):
306
+ super().__init__()
307
+
308
+ self.attn = MultiHeadAttention(n_state, n_head)
309
+ self.attn_ln = LayerNorm(n_state)
310
+
311
+ self.cross_attn = (
312
+ MultiHeadAttention(n_state, n_head) if cross_attention else None
313
+ )
314
+ self.cross_attn_ln = LayerNorm(n_state) if cross_attention else None
315
+
316
+ n_mlp = n_state * 4
317
+ self.mlp = nn.Sequential(
318
+ Linear(n_state, n_mlp), nn.GELU(), Linear(n_mlp, n_state)
319
+ )
320
+ self.mlp_ln = LayerNorm(n_state)
321
+
322
+ def forward(
323
+ self,
324
+ x: Tensor,
325
+ xa: Optional[Tensor] = None,
326
+ mask: Optional[Tensor] = None,
327
+ kv_cache: Optional[dict] = None,
328
+ ):
329
+ x = x + self.attn(self.attn_ln(x), mask=mask, kv_cache=kv_cache)[0]
330
+ if self.cross_attn:
331
+ x = x + self.cross_attn(self.cross_attn_ln(x), xa, kv_cache=kv_cache)[0]
332
+ x = x + self.mlp(self.mlp_ln(x))
333
+ return x
334
+
335
+
336
+ class AudioEncoder(nn.Module):
337
+ def __init__(
338
+ self,
339
+ n_mels: int,
340
+ n_ctx: int,
341
+ n_state: int,
342
+ n_head: int,
343
+ n_layer: int,
344
+ output_dim: int = 512,
345
+ avg_pool: bool = True,
346
+ add_audio_bos_eos_token: bool = True,
347
+ **kwargs
348
+ ):
349
+ super().__init__()
350
+ self.conv1 = Conv1d(n_mels, n_state, kernel_size=3, padding=1)
351
+ self.conv2 = Conv1d(n_state, n_state, kernel_size=3, stride=2, padding=1)
352
+ self.register_buffer("positional_embedding", sinusoids(n_ctx, n_state))
353
+
354
+ self.blocks: Iterable[ResidualAttentionBlock] = nn.ModuleList(
355
+ [ResidualAttentionBlock(n_state, n_head) for _ in range(n_layer)]
356
+ )
357
+ self.ln_post = LayerNorm(n_state)
358
+
359
+ if avg_pool:
360
+ self.avg_pooler = nn.AvgPool1d(2, stride=2)
361
+ else:
362
+ self.avg_pooler = None
363
+ self.proj = nn.Linear(n_state, output_dim)
364
+ if add_audio_bos_eos_token:
365
+ self.audio_bos_eos_token = nn.Embedding(2, output_dim)
366
+ else:
367
+ self.audio_bos_eos_token = None
368
+ self.output_dim = output_dim
369
+ self.n_head = n_head
370
+
371
+ def forward(self, x: Tensor, padding_mask: Tensor=None, audio_lengths: Tensor=None):
372
+ """
373
+ x : torch.Tensor, shape = (batch_size, n_mels, n_ctx)
374
+ the mel spectrogram of the audio
375
+ """
376
+ x = x.to(dtype=self.conv1.weight.dtype,
377
+ device=self.conv1.weight.device)
378
+ if audio_lengths is not None:
379
+ input_mel_len = audio_lengths[:,0] * 2
380
+ max_mel_len_in_batch = input_mel_len.max()
381
+ x = x[:, :, :max_mel_len_in_batch]
382
+ x = F.gelu(self.conv1(x))
383
+ x = F.gelu(self.conv2(x))
384
+ x = x.permute(0, 2, 1) # B, L, D
385
+ bsz = x.size(0)
386
+ src_len = x.size(1)
387
+
388
+
389
+ self.input_positional_embedding = self.positional_embedding[:src_len]
390
+ assert x.shape[1:] == self.input_positional_embedding.shape, f"incorrect audio shape: {x.shape[1:], self.input_positional_embedding.shape}"
391
+ x = (x + self.input_positional_embedding).to(x.dtype)
392
+ if padding_mask is not None:
393
+ padding_mask = padding_mask.to(dtype=self.conv1.weight.dtype,
394
+ device=self.conv1.weight.device)
395
+ batch_src_len = padding_mask.size(1)
396
+ x = x[:, :batch_src_len, :]
397
+ padding_mask = padding_mask.view(
398
+ bsz, -1, batch_src_len
399
+ )
400
+ padding_mask_ = padding_mask.all(1)
401
+ x[padding_mask_] = 0
402
+ key_padding_mask = padding_mask_.view(bsz, 1, 1, batch_src_len). \
403
+ expand(-1, self.n_head, -1, -1).reshape(bsz, self.n_head, 1, batch_src_len)
404
+ new_padding_mask = torch.zeros_like(key_padding_mask, dtype=x.dtype)
405
+ padding_mask = new_padding_mask.masked_fill(key_padding_mask, float("-inf"))
406
+
407
+ for block in self.blocks:
408
+ x = block(x, mask=padding_mask)
409
+
410
+
411
+ if self.avg_pooler:
412
+ x = x.permute(0, 2, 1)
413
+ x = self.avg_pooler(x)
414
+ x = x.permute(0, 2, 1)
415
+
416
+
417
+ x = self.ln_post(x)
418
+ x = self.proj(x)
419
+
420
+ if self.audio_bos_eos_token is not None:
421
+ bos = self.audio_bos_eos_token.weight[0][None, :]
422
+ eos = self.audio_bos_eos_token.weight[1][None, :]
423
+ else:
424
+ bos, eos = None, None
425
+ return x, bos, eos
426
+
427
+ def encode(self, input_audios: Tensor, input_audio_lengths: Tensor, audio_span_tokens: List):
428
+ real_input_audio_lens = input_audio_lengths[:, 0].tolist()
429
+ max_len_in_batch = max(real_input_audio_lens)
430
+ padding_mask = torch.ones([input_audios.size(0), max_len_in_batch]).to(dtype=self.conv1.weight.dtype,
431
+ device=self.conv1.weight.device)
432
+ for index in range(len(input_audios)):
433
+ padding_mask[index, :input_audio_lengths[index][0].item()] = 0
434
+ x, bos, eos = self(input_audios, padding_mask,input_audio_lengths)
435
+ output_audios = []
436
+ for i in range(len(audio_span_tokens)):
437
+ audio_span = audio_span_tokens[i]
438
+ audio = x[i][:audio_span-2]
439
+ if bos is not None:
440
+ audio = torch.concat([bos, audio, eos])
441
+ assert len(audio) == audio_span
442
+ output_audios.append(audio)
443
+ return output_audios
config.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "QWenLMHeadModel"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_qwen.QWenConfig",
7
+ "AutoModelForCausalLM": "modeling_qwen.QWenLMHeadModel"
8
+ },
9
+ "audio": {
10
+ "add_audio_bos_eos_token": true,
11
+ "audio_start_id": 155163,
12
+ "avg_pool": true,
13
+ "n_ctx": 1500,
14
+ "n_head": 20,
15
+ "n_layer": 32,
16
+ "n_mels": 80,
17
+ "n_state": 1280,
18
+ "output_dim": 4096
19
+ },
20
+ "attn_dropout_prob": 0.0,
21
+ "bf16": false,
22
+ "emb_dropout_prob": 0.0,
23
+ "fp16": true,
24
+ "fp32": false,
25
+ "hidden_size": 4096,
26
+ "intermediate_size": 22016,
27
+ "initializer_range": 0.02,
28
+ "kv_channels": 128,
29
+ "layer_norm_epsilon": 1e-06,
30
+ "max_position_embeddings": 8192,
31
+ "model_type": "qwen",
32
+ "no_bias": true,
33
+ "num_attention_heads": 32,
34
+ "num_hidden_layers": 32,
35
+ "onnx_safe": null,
36
+ "quantization_config": {
37
+ "bits": 4,
38
+ "group_size": 128,
39
+ "damp_percent": 0.01,
40
+ "desc_act": false,
41
+ "static_groups": false,
42
+ "sym": true,
43
+ "true_sequential": true,
44
+ "model_name_or_path": null,
45
+ "model_file_base_name": "model",
46
+ "quant_method": "gptq"
47
+ },
48
+ "rotary_emb_base": 10000,
49
+ "rotary_pct": 1.0,
50
+ "scale_attn_weights": true,
51
+ "seq_length": 8192,
52
+ "tie_word_embeddings": false,
53
+ "tokenizer_class": "QWenTokenizer",
54
+ "transformers_version": "4.32.0",
55
+ "use_cache": true,
56
+ "use_dynamic_ntk": true,
57
+ "use_flash_attn": "auto",
58
+ "use_logn_attn": true,
59
+ "vocab_size": 155947
60
+ }
configuration.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {"framework":"Pytorch",
2
+ "task":"multimodal-dialogue",
3
+ "allow_remote": true}
configuration_qwen.py ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright (c) Alibaba Cloud.
2
+ #
3
+ # This source code is licensed under the license found in the
4
+ # LICENSE file in the root directory of this source tree.
5
+
6
+ from transformers import PretrainedConfig
7
+
8
+
9
+ class QWenConfig(PretrainedConfig):
10
+ model_type = "qwen"
11
+ keys_to_ignore_at_inference = ["past_key_values"]
12
+
13
+ def __init__(
14
+ self,
15
+ vocab_size=151936,
16
+ hidden_size=4096,
17
+ num_hidden_layers=32,
18
+ num_attention_heads=32,
19
+ emb_dropout_prob=0.0,
20
+ attn_dropout_prob=0.0,
21
+ layer_norm_epsilon=1e-6,
22
+ initializer_range=0.02,
23
+ max_position_embeddings=8192,
24
+ scale_attn_weights=True,
25
+ use_cache=True,
26
+ bf16=False,
27
+ fp16=False,
28
+ fp32=False,
29
+ kv_channels=128,
30
+ rotary_pct=1.0,
31
+ rotary_emb_base=10000,
32
+ use_dynamic_ntk=True,
33
+ use_logn_attn=True,
34
+ use_flash_attn="auto",
35
+ intermediate_size=22016,
36
+ no_bias=True,
37
+ tie_word_embeddings=False,
38
+ use_cache_quantization=False,
39
+ use_cache_kernel=False,
40
+ softmax_in_fp32=False,
41
+ **kwargs,
42
+ ):
43
+ self.vocab_size = vocab_size
44
+ self.hidden_size = hidden_size
45
+ self.intermediate_size = intermediate_size
46
+ self.num_hidden_layers = num_hidden_layers
47
+ self.num_attention_heads = num_attention_heads
48
+ self.emb_dropout_prob = emb_dropout_prob
49
+ self.attn_dropout_prob = attn_dropout_prob
50
+ self.layer_norm_epsilon = layer_norm_epsilon
51
+ self.initializer_range = initializer_range
52
+ self.scale_attn_weights = scale_attn_weights
53
+ self.use_cache = use_cache
54
+ self.max_position_embeddings = max_position_embeddings
55
+ self.bf16 = bf16
56
+ self.fp16 = fp16
57
+ self.fp32 = fp32
58
+ self.kv_channels = kv_channels
59
+ self.rotary_pct = rotary_pct
60
+ self.rotary_emb_base = rotary_emb_base
61
+ self.use_dynamic_ntk = use_dynamic_ntk
62
+ self.use_logn_attn = use_logn_attn
63
+ self.use_flash_attn = use_flash_attn
64
+ self.no_bias = no_bias
65
+ self.use_cache_quantization = use_cache_quantization
66
+ self.use_cache_kernel = use_cache_kernel
67
+ self.softmax_in_fp32 = softmax_in_fp32
68
+ super().__init__(
69
+ tie_word_embeddings=tie_word_embeddings,
70
+ **kwargs
71
+ )
cpp_kernels.py ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from torch.utils import cpp_extension
2
+ import pathlib
3
+ import os
4
+ import subprocess
5
+
6
+ def _get_cuda_bare_metal_version(cuda_dir):
7
+ raw_output = subprocess.check_output([cuda_dir + "/bin/nvcc", "-V"],
8
+ universal_newlines=True)
9
+ output = raw_output.split()
10
+ release_idx = output.index("release") + 1
11
+ release = output[release_idx].split(".")
12
+ bare_metal_major = release[0]
13
+ bare_metal_minor = release[1][0]
14
+
15
+ return raw_output, bare_metal_major, bare_metal_minor
16
+
17
+ def _create_build_dir(buildpath):
18
+ try:
19
+ os.mkdir(buildpath)
20
+ except OSError:
21
+ if not os.path.isdir(buildpath):
22
+ print(f"Creation of the build directory {buildpath} failed")
23
+
24
+ # Check if cuda 11 is installed for compute capability 8.0
25
+ cc_flag = []
26
+ _, bare_metal_major, bare_metal_minor = _get_cuda_bare_metal_version(cpp_extension.CUDA_HOME)
27
+ if int(bare_metal_major) >= 11:
28
+ cc_flag.append('-gencode')
29
+ cc_flag.append('arch=compute_80,code=sm_80')
30
+ if int(bare_metal_minor) >= 7:
31
+ cc_flag.append('-gencode')
32
+ cc_flag.append('arch=compute_90,code=sm_90')
33
+
34
+ # Build path
35
+ srcpath = pathlib.Path(__file__).parent.absolute()
36
+ buildpath = srcpath / 'build'
37
+ _create_build_dir(buildpath)
38
+
39
+ def _cpp_extention_load_helper(name, sources, extra_cuda_flags):
40
+ return cpp_extension.load(
41
+ name=name,
42
+ sources=sources,
43
+ build_directory=buildpath,
44
+ extra_cflags=['-O3', ],
45
+ extra_cuda_cflags=['-O3',
46
+ '-gencode', 'arch=compute_70,code=sm_70',
47
+ '--use_fast_math'] + extra_cuda_flags + cc_flag,
48
+ verbose=1
49
+ )
50
+
51
+ extra_flags = []
52
+
53
+ cache_autogptq_cuda_256_sources = ["./cache_autogptq_cuda_256.cpp",
54
+ "./cache_autogptq_cuda_kernel_256.cu"]
55
+ cache_autogptq_cuda_256 = _cpp_extention_load_helper("cache_autogptq_cuda_256", cache_autogptq_cuda_256_sources, extra_flags)
generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "chat_format": "chatml",
3
+ "eos_token_id": 151643,
4
+ "pad_token_id": 151643,
5
+ "max_window_size": 6144,
6
+ "max_new_tokens": 512,
7
+ "do_sample": true,
8
+ "top_k": 0,
9
+ "top_p": 0.5,
10
+ "transformers_version": "4.31.0"
11
+ }
mel_filters.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dd2cc75e70e36fcbdd8ffbc2499062f30094093e6bf2cbafa9859f59972b420b
3
+ size 2048
modeling_qwen.py ADDED
@@ -0,0 +1,1425 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright (c) Alibaba Cloud.
2
+ #
3
+ # This source code is licensed under the license found in the
4
+ # LICENSE file in the root directory of this source tree.
5
+
6
+ import copy
7
+ import importlib
8
+ import math
9
+ import shutil
10
+ import pathlib
11
+ from typing import TYPE_CHECKING, Optional, Tuple, Union, Callable, List, Any, Generator, Dict
12
+ import os
13
+
14
+ import torch
15
+ import torch.nn.functional as F
16
+ import torch.utils.checkpoint
17
+ import warnings
18
+ from torch.cuda.amp import autocast
19
+
20
+ from torch.nn import CrossEntropyLoss
21
+ from transformers import PreTrainedTokenizer, GenerationConfig, StoppingCriteriaList
22
+ from transformers.generation.logits_process import LogitsProcessorList
23
+
24
+ if TYPE_CHECKING:
25
+ from transformers.generation.streamers import BaseStreamer
26
+ from transformers.generation.utils import GenerateOutput
27
+ from transformers.modeling_outputs import (
28
+ BaseModelOutputWithPast,
29
+ CausalLMOutputWithPast,
30
+ )
31
+ from transformers.modeling_utils import PreTrainedModel
32
+ from transformers.utils import logging
33
+
34
+ try:
35
+ from einops import rearrange
36
+ except ImportError:
37
+ rearrange = None
38
+ from torch import nn
39
+
40
+ SUPPORT_CUDA = torch.cuda.is_available()
41
+ SUPPORT_BF16 = SUPPORT_CUDA and torch.cuda.is_bf16_supported()
42
+ SUPPORT_FP16 = SUPPORT_CUDA and torch.cuda.get_device_capability(0)[0] >= 7
43
+ SUPPORT_TORCH2 = hasattr(torch, '__version__') and int(torch.__version__.split(".")[0]) >= 2
44
+
45
+
46
+ from .configuration_qwen import QWenConfig
47
+ from .qwen_generation_utils import (
48
+ HistoryType,
49
+ make_context,
50
+ decode_tokens,
51
+ get_stop_words_ids,
52
+ StopWordsLogitsProcessor,
53
+ )
54
+ from .audio import AudioEncoder
55
+
56
+ logger = logging.get_logger(__name__)
57
+
58
+ _CHECKPOINT_FOR_DOC = "qwen"
59
+ _CONFIG_FOR_DOC = "QWenConfig"
60
+
61
+ QWen_PRETRAINED_MODEL_ARCHIVE_LIST = ["qwen-7b"]
62
+
63
+ _ERROR_BAD_CHAT_FORMAT = """\
64
+ We detect you are probably using the pretrained model (rather than chat model) for chatting, since the chat_format in generation_config is not "chatml".
65
+ If you are directly using the model downloaded from Huggingface, please make sure you are using our "Qwen/Qwen-7B-Chat" Huggingface model (rather than "Qwen/Qwen-7B") when you call model.chat().
66
+ 我们检测到您可能在使用预训练模型(而非chat模型)进行多轮chat,因为您当前在generation_config指定的chat_format,并未设置为我们在对话中所支持的"chatml"格式。
67
+ 如果您在直接使用我们从Huggingface提供的模型,请确保您在调用model.chat()时,使用的是"Qwen/Qwen-7B-Chat"模型(而非"Qwen/Qwen-7B"预训练模型)。
68
+ """
69
+
70
+ _SENTINEL = object()
71
+ _ERROR_STREAM_IN_CHAT = """\
72
+ Pass argument `stream` to model.chat() is buggy, deprecated, and marked for removal. Please use model.chat_stream(...) instead of model.chat(..., stream=True).
73
+ 向model.chat()传入参数stream的用法可能存在Bug,该用法已被废弃,将在未来被移除。请使用model.chat_stream(...)代替model.chat(..., stream=True)。
74
+ """
75
+
76
+ _ERROR_INPUT_CPU_QUERY_WITH_FLASH_ATTN_ACTIVATED = """\
77
+ We detect you have activated flash attention support, but running model computation on CPU. Please make sure that your input data has been placed on GPU. If you actually want to run CPU computation, please following the readme and set device_map="cpu" to disable flash attention when loading the model (calling AutoModelForCausalLM.from_pretrained).
78
+ 检测到您的模型已激活了flash attention支持,但正在执行CPU运算任务。如使用flash attention,请您确认模型输入已经传到GPU上。如果您确认要执行CPU运算,请您在载入模型(调用AutoModelForCausalLM.from_pretrained)时,按照readme说法,指定device_map="cpu"以禁用flash attention。
79
+ """
80
+
81
+ apply_rotary_emb_func = None
82
+ rms_norm = None
83
+ flash_attn_unpadded_func = None
84
+
85
+ def _import_flash_attn():
86
+ global apply_rotary_emb_func, rms_norm, flash_attn_unpadded_func
87
+ try:
88
+ from flash_attn.layers.rotary import apply_rotary_emb_func as __apply_rotary_emb_func
89
+ apply_rotary_emb_func = __apply_rotary_emb_func
90
+ except ImportError:
91
+ logger.warn(
92
+ "Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency "
93
+ "https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary"
94
+ )
95
+
96
+ try:
97
+ from flash_attn.ops.rms_norm import rms_norm as __rms_norm
98
+ rms_norm = __rms_norm
99
+ except ImportError:
100
+ logger.warn(
101
+ "Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency "
102
+ "https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm"
103
+ )
104
+
105
+ try:
106
+ import flash_attn
107
+ if not hasattr(flash_attn, '__version__'):
108
+ from flash_attn.flash_attn_interface import flash_attn_unpadded_func as __flash_attn_unpadded_func
109
+ else:
110
+ if int(flash_attn.__version__.split(".")[0]) >= 2:
111
+ from flash_attn.flash_attn_interface import flash_attn_varlen_func as __flash_attn_unpadded_func
112
+ else:
113
+ from flash_attn.flash_attn_interface import flash_attn_unpadded_func as __flash_attn_unpadded_func
114
+ flash_attn_unpadded_func = __flash_attn_unpadded_func
115
+ except ImportError:
116
+ logger.warn(
117
+ "Warning: import flash_attn fail, please install FlashAttention to get higher efficiency "
118
+ "https://github.com/Dao-AILab/flash-attention"
119
+ )
120
+
121
+ def quantize_cache_v(fdata, bits, qmax, qmin):
122
+ # b, s, head, h-dim->b, head, s, h-dim
123
+ qtype = torch.uint8
124
+ device = fdata.device
125
+ shape = fdata.shape
126
+
127
+ fdata_cal = torch.flatten(fdata, 2)
128
+ fmax = torch.amax(fdata_cal, dim=-1, keepdim=True)
129
+ fmin = torch.amin(fdata_cal, dim=-1, keepdim=True)
130
+ # Compute params
131
+ if qmax.device != fmax.device:
132
+ qmax = qmax.to(device)
133
+ qmin = qmin.to(device)
134
+ scale = (fmax - fmin) / (qmax - qmin)
135
+ zero = qmin - fmin / scale
136
+ scale = scale.unsqueeze(-1).repeat(1,1,shape[2],1).contiguous()
137
+ zero = zero.unsqueeze(-1).repeat(1,1,shape[2],1).contiguous()
138
+ # Quantize
139
+ res_data = fdata / scale + zero
140
+ qdata = torch.clamp(res_data, qmin, qmax).to(qtype)
141
+ return qdata.contiguous(), scale, zero
142
+
143
+ def dequantize_cache_torch(qdata, scale, zero):
144
+ data = scale * (qdata - zero)
145
+ return data
146
+
147
+ class FlashSelfAttention(torch.nn.Module):
148
+ def __init__(
149
+ self,
150
+ causal=False,
151
+ softmax_scale=None,
152
+ attention_dropout=0.0,
153
+ ):
154
+ super().__init__()
155
+ assert flash_attn_unpadded_func is not None, (
156
+ "Please install FlashAttention first, " "e.g., with pip install flash-attn"
157
+ )
158
+ assert (
159
+ rearrange is not None
160
+ ), "Please install einops first, e.g., with pip install einops"
161
+ self.causal = causal
162
+ self.softmax_scale = softmax_scale
163
+ self.dropout_p = attention_dropout
164
+
165
+ def unpad_input(self, hidden_states, attention_mask):
166
+ valid_mask = attention_mask.squeeze(1).squeeze(1).eq(0)
167
+ seqlens_in_batch = valid_mask.sum(dim=-1, dtype=torch.int32)
168
+ indices = torch.nonzero(valid_mask.flatten(), as_tuple=False).flatten()
169
+ max_seqlen_in_batch = seqlens_in_batch.max().item()
170
+ cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.torch.int32), (1, 0))
171
+ hidden_states = hidden_states[indices]
172
+ return hidden_states, indices, cu_seqlens, max_seqlen_in_batch
173
+
174
+ def pad_input(self, hidden_states, indices, batch, seqlen):
175
+ output = torch.zeros(batch * seqlen, *hidden_states.shape[1:], device=hidden_states.device,
176
+ dtype=hidden_states.dtype)
177
+ output[indices] = hidden_states
178
+ return rearrange(output, '(b s) ... -> b s ...', b=batch)
179
+
180
+ def forward(self, q, k, v, attention_mask=None):
181
+ assert all((i.dtype in [torch.float16, torch.bfloat16] for i in (q, k, v)))
182
+ assert all((i.is_cuda for i in (q, k, v)))
183
+ batch_size, seqlen_q = q.shape[0], q.shape[1]
184
+ seqlen_k = k.shape[1]
185
+ seqlen_out = seqlen_q
186
+
187
+ q, k, v = [rearrange(x, "b s ... -> (b s) ...") for x in [q, k, v]]
188
+ cu_seqlens_q = torch.arange(
189
+ 0,
190
+ (batch_size + 1) * seqlen_q,
191
+ step=seqlen_q,
192
+ dtype=torch.int32,
193
+ device=q.device,
194
+ )
195
+
196
+ if batch_size > 1 and attention_mask is not None:
197
+ k, indices_k, cu_seqlens_k, seqlen_k = self.unpad_input(k, attention_mask)
198
+ if q.size(0) == v.size(0):
199
+ q = q[indices_k]
200
+ cu_seqlens_q = cu_seqlens_k
201
+ seqlen_q = seqlen_k
202
+ v = v[indices_k]
203
+ else:
204
+ cu_seqlens_k = torch.arange(
205
+ 0,
206
+ (batch_size + 1) * seqlen_k,
207
+ step=seqlen_k,
208
+ dtype=torch.int32,
209
+ device=q.device,
210
+ )
211
+
212
+ if self.training:
213
+ assert seqlen_k == seqlen_q
214
+ is_causal = self.causal
215
+ dropout_p = self.dropout_p
216
+ else:
217
+ is_causal = seqlen_q == seqlen_k
218
+ dropout_p = 0
219
+
220
+ output = flash_attn_unpadded_func(
221
+ q,
222
+ k,
223
+ v,
224
+ cu_seqlens_q,
225
+ cu_seqlens_k,
226
+ seqlen_q,
227
+ seqlen_k,
228
+ dropout_p,
229
+ softmax_scale=self.softmax_scale,
230
+ causal=is_causal,
231
+ )
232
+ if batch_size > 1 and attention_mask is not None and seqlen_q == seqlen_k:
233
+ output = self.pad_input(output, indices_k, batch_size, seqlen_out)
234
+ else:
235
+ new_shape = (batch_size, output.shape[0] // batch_size) + output.shape[1:]
236
+ output = output.view(new_shape)
237
+ return output
238
+
239
+
240
+ class QWenAttention(nn.Module):
241
+ def __init__(self, config):
242
+ super().__init__()
243
+
244
+ self.register_buffer("masked_bias", torch.tensor(-1e4), persistent=False)
245
+ self.seq_length = config.seq_length
246
+
247
+ self.hidden_size = config.hidden_size
248
+ self.split_size = config.hidden_size
249
+ self.num_heads = config.num_attention_heads
250
+ self.head_dim = self.hidden_size // self.num_heads
251
+
252
+ self.use_flash_attn = config.use_flash_attn
253
+ self.scale_attn_weights = True
254
+
255
+ self.projection_size = config.kv_channels * config.num_attention_heads
256
+
257
+ assert self.projection_size % config.num_attention_heads == 0
258
+ self.hidden_size_per_attention_head = (
259
+ self.projection_size // config.num_attention_heads
260
+ )
261
+
262
+ self.c_attn = nn.Linear(config.hidden_size, 3 * self.projection_size)
263
+
264
+ self.c_proj = nn.Linear(
265
+ config.hidden_size, self.projection_size, bias=not config.no_bias
266
+ )
267
+
268
+ self.is_fp32 = not (config.bf16 or config.fp16)
269
+ if (
270
+ self.use_flash_attn
271
+ and flash_attn_unpadded_func is not None
272
+ and not self.is_fp32
273
+ ):
274
+ self.core_attention_flash = FlashSelfAttention(
275
+ causal=True, attention_dropout=config.attn_dropout_prob
276
+ )
277
+ self.bf16 = config.bf16
278
+
279
+ self.use_dynamic_ntk = config.use_dynamic_ntk
280
+ self.use_logn_attn = config.use_logn_attn
281
+
282
+ logn_list = [
283
+ math.log(i, self.seq_length) if i > self.seq_length else 1
284
+ for i in range(1, 32768)
285
+ ]
286
+ logn_tensor = torch.tensor(logn_list)[None, :, None, None]
287
+ self.register_buffer("logn_tensor", logn_tensor, persistent=False)
288
+
289
+ self.attn_dropout = nn.Dropout(config.attn_dropout_prob)
290
+ self.softmax_in_fp32 = config.softmax_in_fp32 if hasattr(config, 'softmax_in_fp32') else False
291
+ self.use_cache_quantization = config.use_cache_quantization if hasattr(config, 'use_cache_quantization') else False
292
+ self.use_cache_kernel = config.use_cache_kernel if hasattr(config,'use_cache_kernel') else False
293
+ cache_dtype = torch.float
294
+ if self.bf16:
295
+ cache_dtype=torch.bfloat16
296
+ elif config.fp16:
297
+ cache_dtype = torch.float16
298
+ self.cache_qmax = torch.tensor(torch.iinfo(torch.uint8).max, dtype=cache_dtype)
299
+ self.cache_qmin = torch.tensor(torch.iinfo(torch.uint8).min, dtype=cache_dtype)
300
+
301
+ if config.use_cache_quantization and config.use_cache_kernel:
302
+ # pre check if the support files existing
303
+ module_root = pathlib.Path(__file__).parent
304
+ src_files = ("cache_autogptq_cuda_256.cpp", "cache_autogptq_cuda_kernel_256.cu")
305
+ if any(not (module_root/src).is_file() for src in src_files):
306
+ warnings.warn("KV cache kernel source files (.cpp and .cu) not found.")
307
+ self.cache_kernels = None
308
+ else:
309
+ try:
310
+ from .cpp_kernels import cache_autogptq_cuda_256
311
+ self.cache_kernels = cache_autogptq_cuda_256
312
+ except ImportError:
313
+ warnings.warn("Failed to import KV cache kernels.")
314
+ self.cache_kernels = None
315
+
316
+ def _attn(self, query, key, value, registered_causal_mask, attention_mask=None, head_mask=None):
317
+ device = query.device
318
+ if self.use_cache_quantization:
319
+ qk, qk_scale, qk_zero = key
320
+ if self.use_cache_kernel and self.cache_kernels is not None:
321
+ shape = query.shape[:-1] + (qk.shape[-2],)
322
+ attn_weights = torch.zeros(shape, dtype=torch.float16, device=device)
323
+ self.cache_kernels.vecquant8matmul_batched_faster_old(
324
+ query.contiguous() if query.dtype == torch.float16 else query.to(torch.float16).contiguous(),
325
+ qk.transpose(-1, -2).contiguous(),
326
+ attn_weights,
327
+ qk_scale.contiguous() if qk_scale.dtype == torch.float16 else qk_scale.to(torch.float16).contiguous(),
328
+ qk_zero.contiguous()if qk_zero.dtype == torch.float16 else qk_zero.to(torch.float16).contiguous())
329
+ # attn_weights = attn_weights.to(query.dtype).contiguous()
330
+ else:
331
+ key = dequantize_cache_torch(qk, qk_scale, qk_zero)
332
+ attn_weights = torch.matmul(query, key.transpose(-1, -2))
333
+ else:
334
+ attn_weights = torch.matmul(query, key.transpose(-1, -2))
335
+
336
+ if self.scale_attn_weights:
337
+ if self.use_cache_quantization:
338
+ size_temp = value[0].size(-1)
339
+ else:
340
+ size_temp = value.size(-1)
341
+ attn_weights = attn_weights / torch.full(
342
+ [],
343
+ size_temp ** 0.5,
344
+ dtype=attn_weights.dtype,
345
+ device=attn_weights.device,
346
+ )
347
+ if self.use_cache_quantization:
348
+ query_length, key_length = query.size(-2), key[0].size(-2)
349
+ else:
350
+ query_length, key_length = query.size(-2), key.size(-2)
351
+ causal_mask = registered_causal_mask[
352
+ :, :, key_length - query_length : key_length, :key_length
353
+ ]
354
+ mask_value = torch.finfo(attn_weights.dtype).min
355
+ mask_value = torch.full([], mask_value, dtype=attn_weights.dtype).to(
356
+ attn_weights.device
357
+ )
358
+ attn_weights = torch.where(
359
+ causal_mask, attn_weights.to(attn_weights.dtype), mask_value
360
+ )
361
+
362
+ if attention_mask is not None:
363
+ attn_weights = attn_weights + attention_mask
364
+
365
+ if self.softmax_in_fp32:
366
+ attn_weights = nn.functional.softmax(attn_weights.float(), dim=-1)
367
+ else:
368
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1)
369
+
370
+ attn_weights = attn_weights.type(query.dtype)
371
+ attn_weights = self.attn_dropout(attn_weights)
372
+
373
+ if head_mask is not None:
374
+ attn_weights = attn_weights * head_mask
375
+
376
+ if self.use_cache_quantization:
377
+ qv, qv_scale, qv_zero = value
378
+ if self.use_cache_kernel and self.cache_kernels is not None:
379
+ shape = attn_weights.shape[:-1] + (query.shape[-1],)
380
+ attn_output = torch.zeros(shape, dtype=torch.float16, device=device)
381
+ self.cache_kernels.vecquant8matmul_batched_column_compression_faster_old(
382
+ attn_weights.contiguous() if attn_weights.dtype == torch.float16 else attn_weights.to(torch.float16).contiguous(),
383
+ qv.contiguous(), # dtype: int32
384
+ attn_output,
385
+ qv_scale.contiguous() if qv_scale.dtype == torch.float16 else qv_scale.to(torch.float16).contiguous(),
386
+ qv_zero.contiguous() if qv_zero.dtype == torch.float16 else qv_zero.to(torch.float16).contiguous())
387
+ if attn_output.dtype != query.dtype:
388
+ attn_output = attn_output.to(query.dtype)
389
+ attn_weights = attn_weights.to(query.dtype)
390
+ else:
391
+ value = dequantize_cache_torch(qv, qv_scale, qv_zero)
392
+ attn_output = torch.matmul(attn_weights, value)
393
+ else:
394
+ attn_output = torch.matmul(attn_weights, value)
395
+
396
+ attn_output = attn_output.transpose(1, 2)
397
+
398
+ return attn_output, attn_weights
399
+
400
+ def _split_heads(self, tensor, num_heads, attn_head_size):
401
+ new_shape = tensor.size()[:-1] + (num_heads, attn_head_size)
402
+ tensor = tensor.view(new_shape)
403
+ return tensor
404
+
405
+ def _merge_heads(self, tensor, num_heads, attn_head_size):
406
+ tensor = tensor.contiguous()
407
+ new_shape = tensor.size()[:-2] + (num_heads * attn_head_size,)
408
+ return tensor.view(new_shape)
409
+
410
+ def forward(
411
+ self,
412
+ hidden_states: Optional[Tuple[torch.FloatTensor]],
413
+ rotary_pos_emb_list: Optional[List[List[torch.Tensor]]] = None,
414
+ layer_past: Optional[Tuple[torch.Tensor]] = None,
415
+ attention_mask: Optional[torch.FloatTensor] = None,
416
+ head_mask: Optional[torch.FloatTensor] = None,
417
+ encoder_hidden_states: Optional[torch.Tensor] = None,
418
+ encoder_attention_mask: Optional[torch.FloatTensor] = None,
419
+ output_attentions: Optional[bool] = False,
420
+ use_cache: Optional[bool] = False,
421
+ ):
422
+ mixed_x_layer = self.c_attn(hidden_states)
423
+
424
+ query, key, value = mixed_x_layer.split(self.split_size, dim=2)
425
+
426
+ query = self._split_heads(query, self.num_heads, self.head_dim)
427
+ key = self._split_heads(key, self.num_heads, self.head_dim)
428
+ value = self._split_heads(value, self.num_heads, self.head_dim)
429
+
430
+ if rotary_pos_emb_list is not None:
431
+ cur_len = query.shape[1]
432
+ if len(rotary_pos_emb_list) == 1:
433
+ rotary_pos_emb = rotary_pos_emb_list[0]
434
+ rotary_pos_emb = [i[:, -cur_len:, :, :] for i in rotary_pos_emb]
435
+ rotary_pos_emb = (rotary_pos_emb,) * 2
436
+ q_pos_emb, k_pos_emb = rotary_pos_emb
437
+ # Slice the pos emb for current inference
438
+ query = apply_rotary_pos_emb(query, q_pos_emb)
439
+ key = apply_rotary_pos_emb(key, k_pos_emb)
440
+ else:
441
+ query_list = []
442
+ key_list = []
443
+ for i, rotary_pos_emb in enumerate(rotary_pos_emb_list):
444
+ rotary_pos_emb = [i[:, -cur_len:, :, :] for i in rotary_pos_emb]
445
+ rotary_pos_emb = (rotary_pos_emb,) * 2
446
+ q_pos_emb, k_pos_emb = rotary_pos_emb
447
+ # Slice the pos emb for current inference
448
+ query_list += [apply_rotary_pos_emb(query[i:i+1, :, :], q_pos_emb)]
449
+ key_list += [apply_rotary_pos_emb(key[i:i+1, :, :], k_pos_emb)]
450
+ query = torch.cat(query_list, dim=0)
451
+ key = torch.cat(key_list, dim=0)
452
+
453
+ if self.use_cache_quantization:
454
+ key = quantize_cache_v(key.permute(0, 2, 1, 3),
455
+ bits=8,
456
+ qmin=self.cache_qmin,
457
+ qmax=self.cache_qmax)
458
+ value = quantize_cache_v(value.permute(0, 2, 1, 3),
459
+ bits=8,
460
+ qmin=self.cache_qmin,
461
+ qmax=self.cache_qmax)
462
+
463
+
464
+ if layer_past is not None:
465
+ past_key, past_value = layer_past[0], layer_past[1]
466
+ if self.use_cache_quantization:
467
+ # use_cache_quantization:
468
+ # present=((q_key,key_scale,key_zero_point),
469
+ # (q_value,value_scale,value_zero_point))
470
+ key = (torch.cat((past_key[0], key[0]), dim=2),
471
+ torch.cat((past_key[1], key[1]), dim=2),
472
+ torch.cat((past_key[2], key[2]), dim=2))
473
+ value = (torch.cat((past_value[0], value[0]), dim=2),
474
+ torch.cat((past_value[1], value[1]), dim=2),
475
+ torch.cat((past_value[2], value[2]), dim=2))
476
+ else:
477
+ # not use_cache_quantization:
478
+ # present=(key,value)
479
+ key = torch.cat((past_key, key), dim=1)
480
+ value = torch.cat((past_value, value), dim=1)
481
+
482
+ if use_cache:
483
+ present = (key, value)
484
+ else:
485
+ present = None
486
+
487
+ if self.use_logn_attn and not self.training:
488
+ if self.use_cache_quantization:
489
+ seq_start = key[0].size(2) - query.size(1)
490
+ seq_end = key[0].size(2)
491
+ else:
492
+ seq_start = key.size(1) - query.size(1)
493
+ seq_end = key.size(1)
494
+ logn_tensor = self.logn_tensor[:, seq_start:seq_end, :, :].type_as(query)
495
+ query = query * logn_tensor.expand_as(query)
496
+
497
+ if (
498
+ self.use_flash_attn
499
+ and flash_attn_unpadded_func is not None
500
+ and not self.is_fp32
501
+ and query.is_cuda
502
+ ):
503
+ q, k, v = query, key, value
504
+ attn_output = self.core_attention_flash(q, k, v, attention_mask=attention_mask)
505
+ else:
506
+ registered_causal_mask = torch.tril(
507
+ torch.ones((key.size(1), key.size(1)), dtype=torch.bool, device=key.device)
508
+ ).view(1, 1, key.size(1), key.size(1))
509
+ query = query.permute(0, 2, 1, 3)
510
+ if not self.use_cache_quantization:
511
+ key = key.permute(0, 2, 1, 3)
512
+ value = value.permute(0, 2, 1, 3)
513
+ if (
514
+ registered_causal_mask is None
515
+ and self.use_flash_attn
516
+ and flash_attn_unpadded_func is not None
517
+ and not self.is_fp32
518
+ and not query.is_cuda
519
+ ):
520
+ raise Exception(_ERROR_INPUT_CPU_QUERY_WITH_FLASH_ATTN_ACTIVATED)
521
+
522
+ if not self.use_cache_quantization and SUPPORT_TORCH2:
523
+ causal_mask = registered_causal_mask[
524
+ :, :, key.size(-2) - query.size(-2): key.size(-2), :key.size(-2)
525
+ ]
526
+ if attention_mask is not None:
527
+ attention_mask = attention_mask.expand(
528
+ -1, -1, causal_mask.size(2), -1
529
+ ).masked_fill(~causal_mask, torch.finfo(query.dtype).min)
530
+ else:
531
+ attention_mask = causal_mask
532
+ attn_output = F.scaled_dot_product_attention(
533
+ query, key, value, attn_mask=attention_mask
534
+ ).transpose(1, 2)
535
+ attn_weight = None
536
+ else:
537
+ attn_output, attn_weight = self._attn(
538
+ query, key, value, registered_causal_mask, attention_mask, head_mask
539
+ )
540
+ context_layer = self._merge_heads(
541
+ attn_output, self.num_heads, self.head_dim
542
+ )
543
+
544
+ attn_output = self.c_proj(context_layer)
545
+
546
+ outputs = (attn_output, present)
547
+ if output_attentions:
548
+ if (
549
+ self.use_flash_attn
550
+ and flash_attn_unpadded_func is not None
551
+ and not self.is_fp32
552
+ ):
553
+ raise ValueError("Cannot output attentions while using flash-attn")
554
+ else:
555
+ outputs += (attn_weight,)
556
+
557
+ return outputs
558
+
559
+
560
+ class QWenMLP(nn.Module):
561
+ def __init__(self, config):
562
+ super().__init__()
563
+ self.w1 = nn.Linear(
564
+ config.hidden_size, config.intermediate_size // 2, bias=not config.no_bias
565
+ )
566
+ self.w2 = nn.Linear(
567
+ config.hidden_size, config.intermediate_size // 2, bias=not config.no_bias
568
+ )
569
+ ff_dim_in = config.intermediate_size // 2
570
+ self.c_proj = nn.Linear(ff_dim_in, config.hidden_size, bias=not config.no_bias)
571
+
572
+ def forward(self, hidden_states):
573
+ a1 = self.w1(hidden_states)
574
+ a2 = self.w2(hidden_states)
575
+ intermediate_parallel = a1 * F.silu(a2)
576
+ output = self.c_proj(intermediate_parallel)
577
+ return output
578
+
579
+ class QWenBlock(nn.Module):
580
+ def __init__(self, config):
581
+ super().__init__()
582
+ hidden_size = config.hidden_size
583
+ self.bf16 = config.bf16
584
+
585
+ self.ln_1 = RMSNorm(
586
+ hidden_size,
587
+ eps=config.layer_norm_epsilon,
588
+ )
589
+ self.attn = QWenAttention(config)
590
+ self.ln_2 = RMSNorm(
591
+ hidden_size,
592
+ eps=config.layer_norm_epsilon,
593
+ )
594
+
595
+ self.mlp = QWenMLP(config)
596
+
597
+ def forward(
598
+ self,
599
+ hidden_states: Optional[Tuple[torch.FloatTensor]],
600
+ rotary_pos_emb_list: Optional[List[List[torch.Tensor]]] = None,
601
+ layer_past: Optional[Tuple[torch.Tensor]] = None,
602
+ attention_mask: Optional[torch.FloatTensor] = None,
603
+ head_mask: Optional[torch.FloatTensor] = None,
604
+ encoder_hidden_states: Optional[torch.Tensor] = None,
605
+ encoder_attention_mask: Optional[torch.FloatTensor] = None,
606
+ use_cache: Optional[bool] = False,
607
+ output_attentions: Optional[bool] = False,
608
+ ):
609
+ layernorm_output = self.ln_1(hidden_states)
610
+
611
+ attn_outputs = self.attn(
612
+ layernorm_output,
613
+ rotary_pos_emb_list,
614
+ layer_past=layer_past,
615
+ attention_mask=attention_mask,
616
+ head_mask=head_mask,
617
+ use_cache=use_cache,
618
+ output_attentions=output_attentions,
619
+ )
620
+ attn_output = attn_outputs[0]
621
+
622
+ outputs = attn_outputs[1:]
623
+
624
+ residual = hidden_states
625
+ layernorm_input = attn_output + residual
626
+
627
+ layernorm_output = self.ln_2(layernorm_input)
628
+
629
+ residual = layernorm_input
630
+ mlp_output = self.mlp(layernorm_output)
631
+ hidden_states = residual + mlp_output
632
+
633
+ if use_cache:
634
+ outputs = (hidden_states,) + outputs
635
+ else:
636
+ outputs = (hidden_states,) + outputs[1:]
637
+
638
+ return outputs
639
+
640
+
641
+ class QWenPreTrainedModel(PreTrainedModel):
642
+ config_class = QWenConfig
643
+ base_model_prefix = "transformer"
644
+ is_parallelizable = False
645
+ supports_gradient_checkpointing = True
646
+ _no_split_modules = ["QWenBlock"]
647
+
648
+ def __init__(self, *inputs, **kwargs):
649
+ super().__init__(*inputs, **kwargs)
650
+
651
+ def _init_weights(self, module):
652
+ """Initialize the weights."""
653
+ if isinstance(module, nn.Linear):
654
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
655
+ if module.bias is not None:
656
+ module.bias.data.zero_()
657
+ elif isinstance(module, nn.Embedding):
658
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
659
+ if module.padding_idx is not None:
660
+ module.weight.data[module.padding_idx].zero_()
661
+ elif isinstance(module, RMSNorm):
662
+ module.weight.data.fill_(1.0)
663
+
664
+ for name, p in module.named_parameters():
665
+ if name == "c_proj.weight":
666
+ p.data.normal_(
667
+ mean=0.0,
668
+ std=(
669
+ self.config.initializer_range
670
+ / math.sqrt(2 * self.config.num_hidden_layers)
671
+ ),
672
+ )
673
+
674
+ def _set_gradient_checkpointing(self, module, value=False):
675
+ if isinstance(module, QWenModel):
676
+ module.gradient_checkpointing = value
677
+
678
+
679
+ class QWenModel(QWenPreTrainedModel):
680
+ _keys_to_ignore_on_load_missing = ["attn.masked_bias"]
681
+
682
+ def __init__(self, config):
683
+ super().__init__(config)
684
+ self.vocab_size = config.vocab_size
685
+ self.num_hidden_layers = config.num_hidden_layers
686
+ self.embed_dim = config.hidden_size
687
+ self.use_cache_quantization = self.config.use_cache_quantization if hasattr(self.config, 'use_cache_quantization') else False
688
+
689
+ self.gradient_checkpointing = False
690
+ self.use_dynamic_ntk = config.use_dynamic_ntk
691
+ self.seq_length = config.seq_length
692
+
693
+ self.wte = nn.Embedding(self.vocab_size, self.embed_dim)
694
+
695
+ self.drop = nn.Dropout(config.emb_dropout_prob)
696
+
697
+ if config.rotary_pct == 1.0:
698
+ self.rotary_ndims = None
699
+ else:
700
+ assert config.rotary_pct < 1
701
+ self.rotary_ndims = int(
702
+ config.kv_channels * config.rotary_pct
703
+ )
704
+ dim = (
705
+ self.rotary_ndims
706
+ if self.rotary_ndims is not None
707
+ else config.kv_channels
708
+ )
709
+ self.rotary_emb = RotaryEmbedding(dim, base=config.rotary_emb_base)
710
+
711
+ self.use_flash_attn = config.use_flash_attn
712
+ self.is_fp32 = not (config.bf16 or config.fp16)
713
+
714
+ self.h = nn.ModuleList(
715
+ [
716
+ QWenBlock(
717
+ config
718
+ )
719
+ for i in range(config.num_hidden_layers)
720
+ ]
721
+ )
722
+ self.ln_f = RMSNorm(
723
+ self.embed_dim,
724
+ eps=config.layer_norm_epsilon,
725
+ )
726
+
727
+ self.audio = AudioEncoder(**config.audio)
728
+
729
+ self.post_init()
730
+
731
+ def get_input_embeddings(self):
732
+ return self.wte
733
+
734
+ def set_input_embeddings(self, new_embeddings):
735
+ self.wte = new_embeddings
736
+
737
+ def get_ntk_alpha(self, true_seq_len):
738
+ context_value = math.log(true_seq_len / self.seq_length, 2) + 1
739
+ ntk_alpha = 2 ** math.ceil(context_value) - 1
740
+ ntk_alpha = max(ntk_alpha, 1)
741
+ return ntk_alpha
742
+
743
+ def forward(
744
+ self,
745
+ input_ids: Optional[torch.LongTensor] = None,
746
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
747
+ attention_mask: Optional[torch.FloatTensor] = None,
748
+ token_type_ids: Optional[torch.LongTensor] = None,
749
+ position_ids: Optional[torch.LongTensor] = None,
750
+ head_mask: Optional[torch.FloatTensor] = None,
751
+ inputs_embeds: Optional[torch.FloatTensor] = None,
752
+ encoder_hidden_states: Optional[torch.Tensor] = None,
753
+ encoder_attention_mask: Optional[torch.FloatTensor] = None,
754
+ use_cache: Optional[bool] = None,
755
+ output_attentions: Optional[bool] = None,
756
+ output_hidden_states: Optional[bool] = None,
757
+ return_dict: Optional[bool] = None,
758
+ audio_info: Dict = None
759
+ ):
760
+ if past_key_values is None and torch.any(input_ids == self.config.audio['audio_start_id']):
761
+ bos_pos = torch.where(input_ids == self.config.audio['audio_start_id'])
762
+ eos_pos = torch.where(input_ids == self.config.audio['audio_start_id'] + 1)
763
+ assert (bos_pos[0] == eos_pos[0]).all()
764
+ audio_pos = torch.stack((bos_pos[0], bos_pos[1], eos_pos[1]), dim=1)
765
+ if isinstance(audio_info, Dict):
766
+ audios = audio_info["input_audios"]
767
+ audio_span_tokens = audio_info["audio_span_tokens"]
768
+ input_audio_lengths = audio_info["input_audio_lengths"]
769
+ audios = self.audio.encode(audios,input_audio_lengths, audio_span_tokens)
770
+ else:
771
+ audios = torch.concat([_["input_audios"] for _ in audio_info])
772
+ input_audio_lengths = torch.concat([_["input_audio_lengths"] for _ in audio_info])
773
+ audio_span_tokens = []
774
+ for _ in audio_info:
775
+ audio_span_tokens.extend(_['audio_span_tokens'])
776
+ audios = self.audio.encode(audios, input_audio_lengths, audio_span_tokens)
777
+
778
+ else:
779
+ audios = None
780
+
781
+ output_attentions = (
782
+ output_attentions
783
+ if output_attentions is not None
784
+ else self.config.output_attentions
785
+ )
786
+ output_hidden_states = (
787
+ output_hidden_states
788
+ if output_hidden_states is not None
789
+ else self.config.output_hidden_states
790
+ )
791
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
792
+ return_dict = (
793
+ return_dict if return_dict is not None else self.config.use_return_dict
794
+ )
795
+
796
+ if input_ids is not None and inputs_embeds is not None:
797
+ raise ValueError(
798
+ "You cannot specify both input_ids and inputs_embeds at the same time"
799
+ )
800
+ elif input_ids is not None:
801
+ input_shape = input_ids.size()
802
+ input_ids = input_ids.view(-1, input_shape[-1])
803
+ batch_size = input_ids.shape[0]
804
+ elif inputs_embeds is not None:
805
+ input_shape = inputs_embeds.size()[:-1]
806
+ batch_size = inputs_embeds.shape[0]
807
+ else:
808
+ raise ValueError("You have to specify either input_ids or inputs_embeds")
809
+
810
+ device = input_ids.device if input_ids is not None else inputs_embeds.device
811
+
812
+ if token_type_ids is not None:
813
+ token_type_ids = token_type_ids.view(-1, input_shape[-1])
814
+ if position_ids is not None:
815
+ position_ids = position_ids.view(-1, input_shape[-1])
816
+
817
+ if past_key_values is None:
818
+ past_length = 0
819
+ past_key_values = tuple([None] * len(self.h))
820
+ else:
821
+ if self.use_cache_quantization:
822
+ past_length = past_key_values[0][0][0].size(2)
823
+ else:
824
+ past_length = past_key_values[0][0].size(-2)
825
+ if position_ids is None:
826
+ position_ids = torch.arange(
827
+ past_length,
828
+ input_shape[-1] + past_length,
829
+ dtype=torch.long,
830
+ device=device,
831
+ )
832
+ position_ids = position_ids.unsqueeze(0).view(-1, input_shape[-1])
833
+
834
+ if attention_mask is not None:
835
+ if batch_size <= 0:
836
+ raise ValueError("batch_size has to be defined and > 0")
837
+ attention_mask = attention_mask.view(batch_size, -1)
838
+ attention_mask = attention_mask[:, None, None, :]
839
+ attention_mask = attention_mask.to(dtype=self.dtype)
840
+ attention_mask = (1.0 - attention_mask) * torch.finfo(self.dtype).min
841
+
842
+ encoder_attention_mask = None
843
+ head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
844
+
845
+ if inputs_embeds is None:
846
+ inputs_embeds = self.wte(input_ids)
847
+ hidden_states = inputs_embeds
848
+
849
+ kv_seq_len = hidden_states.size()[1]
850
+ if past_key_values[0] is not None:
851
+ # past key values[0][0] shape: bs * seq_len * head_num * dim
852
+ if self.use_cache_quantization:
853
+ kv_seq_len += past_key_values[0][0][0].shape[2]
854
+ else:
855
+ kv_seq_len += past_key_values[0][0].shape[1]
856
+
857
+ if self.training or not self.use_dynamic_ntk:
858
+ ntk_alpha_list = [1.0]
859
+ elif kv_seq_len != hidden_states.size()[1]:
860
+ ntk_alpha_list = self.rotary_emb._ntk_alpha_cached_list
861
+ else:
862
+ ntk_alpha_list = []
863
+ if attention_mask is not None and kv_seq_len > self.seq_length:
864
+ true_seq_lens = attention_mask.squeeze(1).squeeze(1).eq(0).sum(dim=-1, dtype=torch.int32)
865
+ for i in range(hidden_states.size()[0]):
866
+ true_seq_len = true_seq_lens[i].item()
867
+ ntk_alpha = self.get_ntk_alpha(true_seq_len)
868
+ ntk_alpha_list.append(ntk_alpha)
869
+ else:
870
+ ntk_alpha = self.get_ntk_alpha(kv_seq_len)
871
+ ntk_alpha_list.append(ntk_alpha)
872
+ self.rotary_emb._ntk_alpha_cached_list = ntk_alpha_list
873
+ rotary_pos_emb_list = [
874
+ self.rotary_emb(kv_seq_len, ntk_alpha=ntk_alpha) for ntk_alpha in ntk_alpha_list
875
+ ]
876
+
877
+ hidden_states = self.drop(hidden_states)
878
+ if audios is not None:
879
+ for idx, (i, a, b) in enumerate(audio_pos):
880
+ hidden_states[i][a : b+1] = audios[idx]
881
+ output_shape = input_shape + (hidden_states.size(-1),)
882
+
883
+ if self.gradient_checkpointing and self.training:
884
+ if use_cache:
885
+ logger.warning_once(
886
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
887
+ )
888
+ use_cache = False
889
+
890
+ presents = () if use_cache else None
891
+ all_self_attentions = () if output_attentions else None
892
+ all_hidden_states = () if output_hidden_states else None
893
+ for i, (block, layer_past) in enumerate(zip(self.h, past_key_values)):
894
+
895
+ if output_hidden_states:
896
+ all_hidden_states = all_hidden_states + (hidden_states,)
897
+
898
+ if self.gradient_checkpointing and self.training:
899
+
900
+ def create_custom_forward(module):
901
+ def custom_forward(*inputs):
902
+ # None for past_key_value
903
+ return module(*inputs, use_cache, output_attentions)
904
+
905
+ return custom_forward
906
+
907
+ outputs = torch.utils.checkpoint.checkpoint(
908
+ create_custom_forward(block),
909
+ hidden_states,
910
+ rotary_pos_emb_list,
911
+ None,
912
+ attention_mask,
913
+ head_mask[i],
914
+ encoder_hidden_states,
915
+ encoder_attention_mask,
916
+ )
917
+ else:
918
+
919
+ outputs = block(
920
+ hidden_states,
921
+ layer_past=layer_past,
922
+ rotary_pos_emb_list=rotary_pos_emb_list,
923
+ attention_mask=attention_mask,
924
+ head_mask=head_mask[i],
925
+ encoder_hidden_states=encoder_hidden_states,
926
+ encoder_attention_mask=encoder_attention_mask,
927
+ use_cache=use_cache,
928
+ output_attentions=output_attentions,
929
+ )
930
+
931
+ hidden_states = outputs[0]
932
+ if use_cache is True:
933
+ presents = presents + (outputs[1],)
934
+
935
+ if output_attentions:
936
+ all_self_attentions = all_self_attentions + (outputs[2 if use_cache else 1],)
937
+
938
+ hidden_states = self.ln_f(hidden_states)
939
+ hidden_states = hidden_states.view(output_shape)
940
+ # Add last hidden state
941
+ if output_hidden_states:
942
+ all_hidden_states = all_hidden_states + (hidden_states,)
943
+
944
+ if not return_dict:
945
+ return tuple(
946
+ v for v in [hidden_states, presents, all_hidden_states] if v is not None
947
+ )
948
+
949
+ return BaseModelOutputWithPast(
950
+ last_hidden_state=hidden_states,
951
+ past_key_values=presents,
952
+ hidden_states=all_hidden_states,
953
+ attentions=all_self_attentions,
954
+ )
955
+
956
+
957
+ class QWenLMHeadModel(QWenPreTrainedModel):
958
+ _keys_to_ignore_on_load_missing = [r"h\.\d+\.attn\.rotary_emb\.inv_freq"]
959
+ _keys_to_ignore_on_load_unexpected = [r"h\.\d+\.attn\.masked_bias"]
960
+
961
+ def __init__(self, config):
962
+ super().__init__(config)
963
+ assert (
964
+ config.bf16 + config.fp16 + config.fp32 <= 1
965
+ ), "Only one of \"bf16\", \"fp16\", \"fp32\" can be true"
966
+
967
+ autoset_precision = config.bf16 + config.fp16 + config.fp32 == 0
968
+
969
+ if autoset_precision:
970
+ if SUPPORT_BF16:
971
+ logger.warn(
972
+ "The model is automatically converting to bf16 for faster inference. "
973
+ "If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to \"AutoModelForCausalLM.from_pretrained\"."
974
+ )
975
+ config.bf16 = True
976
+ elif SUPPORT_FP16:
977
+ logger.warn(
978
+ "The model is automatically converting to fp16 for faster inference. "
979
+ "If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to \"AutoModelForCausalLM.from_pretrained\"."
980
+ )
981
+ config.fp16 = True
982
+ else:
983
+ config.fp32 = True
984
+
985
+ if config.bf16 and SUPPORT_CUDA and not SUPPORT_BF16:
986
+ logger.warn("Your device does NOT seem to support bf16, you can switch to fp16 or fp32 by by passing fp16/fp32=True in \"AutoModelForCausalLM.from_pretrained\".")
987
+ if config.fp16 and SUPPORT_CUDA and not SUPPORT_FP16:
988
+ logger.warn("Your device does NOT support faster inference with fp16, please switch to fp32 which is likely to be faster")
989
+ if config.fp32:
990
+ if SUPPORT_BF16:
991
+ logger.warn("Your device support faster inference by passing bf16=True in \"AutoModelForCausalLM.from_pretrained\".")
992
+ elif SUPPORT_FP16:
993
+ logger.warn("Your device support faster inference by passing fp16=True in \"AutoModelForCausalLM.from_pretrained\".")
994
+
995
+ if config.use_flash_attn == "auto":
996
+ if config.bf16 or config.fp16:
997
+ logger.warn("Try importing flash-attention for faster inference...")
998
+ config.use_flash_attn = True
999
+ else:
1000
+ config.use_flash_attn = False
1001
+ if config.use_flash_attn and config.fp32:
1002
+ logger.warn("Flash attention will be disabled because it does NOT support fp32.")
1003
+
1004
+ if config.use_flash_attn:
1005
+ _import_flash_attn()
1006
+
1007
+ self.transformer = QWenModel(config)
1008
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
1009
+
1010
+ if config.bf16:
1011
+ self.transformer.bfloat16()
1012
+ self.lm_head.bfloat16()
1013
+ if config.fp16:
1014
+ self.transformer.half()
1015
+ self.lm_head.half()
1016
+ self.post_init()
1017
+
1018
+ @classmethod
1019
+ def from_pretrained(
1020
+ cls,
1021
+ pretrained_model_name_or_path: Optional[Union[str, os.PathLike]],
1022
+ *model_args,
1023
+ config=None,
1024
+ cache_dir=None,
1025
+ **kwargs,
1026
+ ):
1027
+ if os.path.isdir(pretrained_model_name_or_path):
1028
+ # Local Directory of Models
1029
+ mel_filters_path = os.path.join(pretrained_model_name_or_path, 'mel_filters.npz')
1030
+ tgt_cache_path = os.path.join(os.path.dirname(__file__), 'mel_filters.npz')
1031
+ shutil.copy(mel_filters_path, tgt_cache_path)
1032
+ else:
1033
+ # Loading from huggingface repo
1034
+ from huggingface_hub import hf_hub_download
1035
+ hf_hub_download(repo_id=pretrained_model_name_or_path, filename="mel_filters.npz",
1036
+ token=kwargs.get('token', None), local_dir=os.path.dirname(__file__))
1037
+ return super().from_pretrained(pretrained_model_name_or_path, *model_args, config=config, cache_dir=cache_dir,
1038
+ **kwargs)
1039
+
1040
+ def get_output_embeddings(self):
1041
+ return self.lm_head
1042
+
1043
+ def set_output_embeddings(self, new_embeddings):
1044
+ self.lm_head = new_embeddings
1045
+
1046
+ def prepare_inputs_for_generation(
1047
+ self, input_ids, past_key_values=None, inputs_embeds=None, **kwargs
1048
+ ):
1049
+ audio_info = kwargs.pop("audio_info", None)
1050
+ token_type_ids = kwargs.get("token_type_ids", None)
1051
+ if past_key_values:
1052
+ input_ids = input_ids[:, -1].unsqueeze(-1)
1053
+ if token_type_ids is not None:
1054
+ token_type_ids = token_type_ids[:, -1].unsqueeze(-1)
1055
+
1056
+ attention_mask = kwargs.get("attention_mask", None)
1057
+ position_ids = kwargs.get("position_ids", None)
1058
+
1059
+ if attention_mask is not None and position_ids is None:
1060
+ position_ids = attention_mask.long().cumsum(-1) - 1
1061
+ position_ids.masked_fill_(attention_mask == 0, 1)
1062
+ if past_key_values:
1063
+ position_ids = position_ids[:, -1].unsqueeze(-1)
1064
+ else:
1065
+ position_ids = None
1066
+
1067
+ if inputs_embeds is not None and past_key_values is None:
1068
+ model_inputs = {"inputs_embeds": inputs_embeds}
1069
+ else:
1070
+ model_inputs = {"input_ids": input_ids}
1071
+
1072
+ model_inputs.update(
1073
+ {
1074
+ "past_key_values": past_key_values,
1075
+ "use_cache": kwargs.get("use_cache"),
1076
+ "position_ids": position_ids,
1077
+ "attention_mask": attention_mask,
1078
+ "token_type_ids": token_type_ids,
1079
+ "audio_info": audio_info
1080
+ }
1081
+ )
1082
+ return model_inputs
1083
+
1084
+ def forward(
1085
+ self,
1086
+ input_ids: Optional[torch.LongTensor] = None,
1087
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
1088
+ attention_mask: Optional[torch.FloatTensor] = None,
1089
+ token_type_ids: Optional[torch.LongTensor] = None,
1090
+ position_ids: Optional[torch.LongTensor] = None,
1091
+ head_mask: Optional[torch.FloatTensor] = None,
1092
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1093
+ encoder_hidden_states: Optional[torch.Tensor] = None,
1094
+ encoder_attention_mask: Optional[torch.FloatTensor] = None,
1095
+ labels: Optional[torch.LongTensor] = None,
1096
+ use_cache: Optional[bool] = None,
1097
+ output_attentions: Optional[bool] = None,
1098
+ output_hidden_states: Optional[bool] = None,
1099
+ return_dict: Optional[bool] = None,
1100
+ audio_info: Dict = None
1101
+ ) -> Union[Tuple, CausalLMOutputWithPast]:
1102
+
1103
+ return_dict = (
1104
+ return_dict if return_dict is not None else self.config.use_return_dict
1105
+ )
1106
+
1107
+ transformer_outputs = self.transformer(
1108
+ input_ids,
1109
+ past_key_values=past_key_values,
1110
+ attention_mask=attention_mask,
1111
+ token_type_ids=token_type_ids,
1112
+ position_ids=position_ids,
1113
+ head_mask=head_mask,
1114
+ inputs_embeds=inputs_embeds,
1115
+ encoder_hidden_states=encoder_hidden_states,
1116
+ encoder_attention_mask=encoder_attention_mask,
1117
+ use_cache=use_cache,
1118
+ output_attentions=output_attentions,
1119
+ output_hidden_states=output_hidden_states,
1120
+ return_dict=return_dict,
1121
+ audio_info=audio_info
1122
+ )
1123
+ hidden_states = transformer_outputs[0]
1124
+
1125
+ lm_logits = self.lm_head(hidden_states)
1126
+
1127
+ loss = None
1128
+ if labels is not None:
1129
+ labels = labels.to(lm_logits.device)
1130
+ shift_logits = lm_logits[..., :-1, :].contiguous()
1131
+ shift_labels = labels[..., 1:].contiguous()
1132
+ loss_fct = CrossEntropyLoss()
1133
+ loss = loss_fct(
1134
+ shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1)
1135
+ )
1136
+
1137
+ if not return_dict:
1138
+ output = (lm_logits,) + transformer_outputs[1:]
1139
+ return ((loss,) + output) if loss is not None else output
1140
+
1141
+ return CausalLMOutputWithPast(
1142
+ loss=loss,
1143
+ logits=lm_logits,
1144
+ past_key_values=transformer_outputs.past_key_values,
1145
+ hidden_states=transformer_outputs.hidden_states,
1146
+ attentions=transformer_outputs.attentions,
1147
+ )
1148
+
1149
+ @staticmethod
1150
+ def _reorder_cache(
1151
+ past_key_values: Tuple[Tuple[torch.Tensor]], beam_idx: torch.Tensor
1152
+ ) -> Tuple[Tuple[torch.Tensor]]:
1153
+
1154
+ return tuple(
1155
+ tuple(
1156
+ past_state.index_select(0, beam_idx.to(past_state.device))
1157
+ for past_state in layer_past
1158
+ )
1159
+ for layer_past in past_key_values
1160
+ )
1161
+
1162
+ def chat(
1163
+ self,
1164
+ tokenizer: PreTrainedTokenizer,
1165
+ query: str,
1166
+ history: Optional[HistoryType],
1167
+ system: str = "You are a helpful assistant.",
1168
+ append_history: bool = True,
1169
+ stream: Optional[bool] = _SENTINEL,
1170
+ stop_words_ids: Optional[List[List[int]]] = None,
1171
+ generation_config: Optional[GenerationConfig] = None,
1172
+ **kwargs,
1173
+ ) -> Tuple[str, HistoryType]:
1174
+ generation_config = generation_config if generation_config is not None else self.generation_config
1175
+
1176
+ assert stream is _SENTINEL, _ERROR_STREAM_IN_CHAT
1177
+ assert generation_config.chat_format == 'chatml', _ERROR_BAD_CHAT_FORMAT
1178
+ if history is None:
1179
+ history = []
1180
+ else:
1181
+ # make a copy of the user's input such that is is left untouched
1182
+ history = copy.deepcopy(history)
1183
+
1184
+ if stop_words_ids is None:
1185
+ stop_words_ids = []
1186
+
1187
+ max_window_size = kwargs.get('max_window_size', None)
1188
+ if max_window_size is None:
1189
+ max_window_size = generation_config.max_window_size
1190
+ raw_text, context_tokens, audio_info = make_context(
1191
+ tokenizer,
1192
+ query,
1193
+ history=history,
1194
+ system=system,
1195
+ max_window_size=max_window_size,
1196
+ chat_format=generation_config.chat_format,
1197
+ )
1198
+
1199
+ stop_words_ids.extend(get_stop_words_ids(
1200
+ generation_config.chat_format, tokenizer
1201
+ ))
1202
+ input_ids = torch.tensor([context_tokens]).to(self.device)
1203
+ kwargs['audio_info'] = audio_info
1204
+ outputs = self.generate(
1205
+ input_ids,
1206
+ stop_words_ids=stop_words_ids,
1207
+ return_dict_in_generate=False,
1208
+ generation_config=generation_config,
1209
+ **kwargs,
1210
+ )
1211
+
1212
+ response = decode_tokens(
1213
+ outputs[0],
1214
+ tokenizer,
1215
+ raw_text_len=len(raw_text),
1216
+ context_length=len(context_tokens),
1217
+ chat_format=generation_config.chat_format,
1218
+ verbose=False,
1219
+ errors='replace',
1220
+ audio_info=audio_info
1221
+ )
1222
+
1223
+ # as history is a copy of the user inputs,
1224
+ # we can always return the new turn to the user.
1225
+ # separating input history and output history also enables the user
1226
+ # to implement more complex history management
1227
+ history.append((query, response))
1228
+
1229
+ return response, history
1230
+
1231
+ def chat_stream(
1232
+ self,
1233
+ tokenizer: PreTrainedTokenizer,
1234
+ query: str,
1235
+ history: Optional[HistoryType],
1236
+ system: str = "You are a helpful assistant.",
1237
+ stop_words_ids: Optional[List[List[int]]] = None,
1238
+ logits_processor: Optional[LogitsProcessorList] = None,
1239
+ generation_config: Optional[GenerationConfig] = None,
1240
+ **kwargs,
1241
+ ) -> Generator[str, Any, None]:
1242
+ generation_config = generation_config if generation_config is not None else self.generation_config
1243
+ assert generation_config.chat_format == 'chatml', _ERROR_BAD_CHAT_FORMAT
1244
+ if history is None:
1245
+ history = []
1246
+ if stop_words_ids is None:
1247
+ stop_words_ids = []
1248
+
1249
+ max_window_size = kwargs.get('max_window_size', None)
1250
+ if max_window_size is None:
1251
+ max_window_size = generation_config.max_window_size
1252
+ raw_text, context_tokens = make_context(
1253
+ tokenizer,
1254
+ query,
1255
+ history=history,
1256
+ system=system,
1257
+ max_window_size=max_window_size,
1258
+ chat_format=generation_config.chat_format,
1259
+ )
1260
+
1261
+ stop_words_ids.extend(get_stop_words_ids(
1262
+ generation_config.chat_format, tokenizer
1263
+ ))
1264
+ if stop_words_ids is not None:
1265
+ stop_words_logits_processor = StopWordsLogitsProcessor(
1266
+ stop_words_ids=stop_words_ids,
1267
+ eos_token_id=generation_config.eos_token_id,
1268
+ )
1269
+ if logits_processor is None:
1270
+ logits_processor = LogitsProcessorList([stop_words_logits_processor])
1271
+ else:
1272
+ logits_processor.append(stop_words_logits_processor)
1273
+ input_ids = torch.tensor([context_tokens]).to(self.device)
1274
+
1275
+ from transformers_stream_generator.main import NewGenerationMixin, StreamGenerationConfig
1276
+ self.__class__.generate_stream = NewGenerationMixin.generate
1277
+ self.__class__.sample_stream = NewGenerationMixin.sample_stream
1278
+ stream_config = StreamGenerationConfig(**generation_config.to_dict(), do_stream=True)
1279
+
1280
+ def stream_generator():
1281
+ outputs = []
1282
+ for token in self.generate_stream(
1283
+ input_ids,
1284
+ return_dict_in_generate=False,
1285
+ generation_config=stream_config,
1286
+ logits_processor=logits_processor,
1287
+ seed=-1,
1288
+ **kwargs):
1289
+ outputs.append(token.item())
1290
+ yield tokenizer.decode(outputs, skip_special_tokens=True, errors='ignore')
1291
+
1292
+ return stream_generator()
1293
+
1294
+ def generate(
1295
+ self,
1296
+ inputs: Optional[torch.Tensor] = None,
1297
+ generation_config: Optional[GenerationConfig] = None,
1298
+ logits_processor: Optional[LogitsProcessorList] = None,
1299
+ stopping_criteria: Optional[StoppingCriteriaList] = None,
1300
+ prefix_allowed_tokens_fn: Optional[
1301
+ Callable[[int, torch.Tensor], List[int]]
1302
+ ] = None,
1303
+ synced_gpus: Optional[bool] = None,
1304
+ assistant_model: Optional["PreTrainedModel"] = None,
1305
+ streamer: Optional["BaseStreamer"] = None,
1306
+ **kwargs,
1307
+ ) -> Union[GenerateOutput, torch.LongTensor]:
1308
+ generation_config = generation_config if generation_config is not None else self.generation_config
1309
+
1310
+ # Process stop_words_ids.
1311
+ stop_words_ids = kwargs.pop("stop_words_ids", None)
1312
+ if stop_words_ids is None and generation_config is not None:
1313
+ stop_words_ids = getattr(generation_config, "stop_words_ids", None)
1314
+ if stop_words_ids is None:
1315
+ stop_words_ids = getattr(generation_config, "stop_words_ids", None)
1316
+
1317
+ if stop_words_ids is not None:
1318
+ stop_words_logits_processor = StopWordsLogitsProcessor(
1319
+ stop_words_ids=stop_words_ids,
1320
+ eos_token_id=generation_config.eos_token_id,
1321
+ )
1322
+ if logits_processor is None:
1323
+ logits_processor = LogitsProcessorList([stop_words_logits_processor])
1324
+ else:
1325
+ logits_processor.append(stop_words_logits_processor)
1326
+
1327
+ return super().generate(
1328
+ inputs,
1329
+ generation_config=generation_config,
1330
+ logits_processor=logits_processor,
1331
+ stopping_criteria=stopping_criteria,
1332
+ prefix_allowed_tokens_fn=prefix_allowed_tokens_fn,
1333
+ synced_gpus=synced_gpus,
1334
+ assistant_model=assistant_model,
1335
+ streamer=streamer,
1336
+ **kwargs,
1337
+ )
1338
+
1339
+
1340
+ class RotaryEmbedding(torch.nn.Module):
1341
+ def __init__(self, dim, base=10000):
1342
+ super().__init__()
1343
+ self.dim = dim
1344
+ self.base = base
1345
+ inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
1346
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
1347
+ if importlib.util.find_spec("einops") is None:
1348
+ raise RuntimeError("einops is required for Rotary Embedding")
1349
+
1350
+ self._rotary_pos_emb_cache = None
1351
+ self._seq_len_cached = 0
1352
+ self._ntk_alpha_cached = 1.0
1353
+ self._ntk_alpha_cached_list = [1.0]
1354
+
1355
+ def update_rotary_pos_emb_cache(self, max_seq_len, offset=0, ntk_alpha=1.0):
1356
+ seqlen = max_seq_len + offset
1357
+ if seqlen > self._seq_len_cached or ntk_alpha != self._ntk_alpha_cached:
1358
+ base = self.base * ntk_alpha ** (self.dim / (self.dim - 2))
1359
+ self.inv_freq = 1.0 / (
1360
+ base
1361
+ ** (
1362
+ torch.arange(0, self.dim, 2, device=self.inv_freq.device).float()
1363
+ / self.dim
1364
+ )
1365
+ )
1366
+ self._seq_len_cached = max(2 * seqlen, 16)
1367
+ self._ntk_alpha_cached = ntk_alpha
1368
+ seq = torch.arange(self._seq_len_cached, device=self.inv_freq.device)
1369
+ freqs = torch.outer(seq.type_as(self.inv_freq), self.inv_freq)
1370
+
1371
+ emb = torch.cat((freqs, freqs), dim=-1)
1372
+ from einops import rearrange
1373
+
1374
+ emb = rearrange(emb, "n d -> 1 n 1 d")
1375
+
1376
+ cos, sin = emb.cos(), emb.sin()
1377
+ self._rotary_pos_emb_cache = [cos, sin]
1378
+
1379
+ def forward(self, max_seq_len, offset=0, ntk_alpha=1.0):
1380
+ self.update_rotary_pos_emb_cache(max_seq_len, offset, ntk_alpha)
1381
+ cos, sin = self._rotary_pos_emb_cache
1382
+ return [cos[:, offset : offset + max_seq_len], sin[:, offset : offset + max_seq_len]]
1383
+
1384
+
1385
+ def _rotate_half(x):
1386
+ from einops import rearrange
1387
+
1388
+ x = rearrange(x, "... (j d) -> ... j d", j=2)
1389
+ x1, x2 = x.unbind(dim=-2)
1390
+ return torch.cat((-x2, x1), dim=-1)
1391
+
1392
+
1393
+ def apply_rotary_pos_emb(t, freqs):
1394
+ cos, sin = freqs
1395
+ if apply_rotary_emb_func is not None and t.is_cuda:
1396
+ t_ = t.float()
1397
+ cos = cos.squeeze(0).squeeze(1)[:, : cos.shape[-1] // 2]
1398
+ sin = sin.squeeze(0).squeeze(1)[:, : sin.shape[-1] // 2]
1399
+ output = apply_rotary_emb_func(t_, cos, sin).type_as(t)
1400
+ return output
1401
+ else:
1402
+ rot_dim = freqs[0].shape[-1]
1403
+ cos, sin = freqs
1404
+ t_, t_pass_ = t[..., :rot_dim], t[..., rot_dim:]
1405
+ t_ = t_.float()
1406
+ t_pass_ = t_pass_.float()
1407
+ t_ = (t_ * cos) + (_rotate_half(t_) * sin)
1408
+ return torch.cat((t_, t_pass_), dim=-1).type_as(t)
1409
+
1410
+
1411
+ class RMSNorm(torch.nn.Module):
1412
+ def __init__(self, dim: int, eps: float = 1e-6):
1413
+ super().__init__()
1414
+ self.eps = eps
1415
+ self.weight = nn.Parameter(torch.ones(dim))
1416
+
1417
+ def _norm(self, x):
1418
+ return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
1419
+
1420
+ def forward(self, x):
1421
+ if rms_norm is not None and x.is_cuda:
1422
+ return rms_norm(x, self.weight, self.eps)
1423
+ else:
1424
+ output = self._norm(x.float()).type_as(x)
1425
+ return output * self.weight
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6415987961a707da763cd76268de56c086d072d0962bdb11fab05212c5398160
3
+ size 7210852530
quantize_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bits": 4,
3
+ "group_size": 128,
4
+ "damp_percent": 0.01,
5
+ "desc_act": false,
6
+ "static_groups": false,
7
+ "sym": true,
8
+ "true_sequential": true,
9
+ "model_name_or_path": null,
10
+ "model_file_base_name": null
11
+ }
qwen.tiktoken ADDED
The diff for this file is too large to render. See raw diff
 
qwen_generation_utils.py ADDED
@@ -0,0 +1,431 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright (c) Alibaba Cloud.
2
+ #
3
+ # This source code is licensed under the license found in the
4
+ # LICENSE file in the root directory of this source tree.
5
+
6
+ """Generation support."""
7
+
8
+ from typing import Tuple, List, Union, Iterable, Dict
9
+
10
+ import numpy as np
11
+ import torch
12
+ import torch.nn.functional as F
13
+ from transformers import PreTrainedTokenizer
14
+ from transformers import logging
15
+ from transformers.generation import LogitsProcessor
16
+
17
+ logger = logging.get_logger(__name__)
18
+
19
+ # Types.
20
+ HistoryType = List[Tuple[str, str]]
21
+ TokensType = List[int]
22
+ BatchTokensType = List[List[int]]
23
+
24
+
25
+ def pad_batch(batch: BatchTokensType, pad_id: int, seq_length: int) -> BatchTokensType:
26
+ for tokens in batch:
27
+ context_length = len(tokens)
28
+ if context_length < seq_length:
29
+ tokens.extend([pad_id] * (seq_length - context_length))
30
+ return batch
31
+
32
+
33
+ def get_ltor_masks_and_position_ids(
34
+ data,
35
+ eod_token,
36
+ reset_position_ids,
37
+ reset_attention_mask,
38
+ eod_mask_loss,
39
+ ):
40
+ """Build masks and position id for left to right model."""
41
+
42
+ # Extract batch size and sequence length.
43
+ micro_batch_size, seq_length = data.size()
44
+
45
+ # Attention mask (lower triangular).
46
+ if reset_attention_mask:
47
+ att_mask_batch = micro_batch_size
48
+ else:
49
+ att_mask_batch = 1
50
+ attention_mask = torch.tril(
51
+ torch.ones((att_mask_batch, seq_length, seq_length), device=data.device)
52
+ ).view(att_mask_batch, 1, seq_length, seq_length)
53
+
54
+ # Loss mask.
55
+ loss_mask = torch.ones(data.size(), dtype=torch.float, device=data.device)
56
+ if eod_mask_loss:
57
+ loss_mask[data == eod_token] = 0.0
58
+
59
+ # Position ids.
60
+ position_ids = torch.arange(seq_length, dtype=torch.long, device=data.device)
61
+ position_ids = position_ids.unsqueeze(0).expand_as(data)
62
+ # We need to clone as the ids will be modifed based on batch index.
63
+ if reset_position_ids:
64
+ position_ids = position_ids.clone()
65
+
66
+ if reset_position_ids or reset_attention_mask:
67
+ # Loop through the batches:
68
+ for b in range(micro_batch_size):
69
+
70
+ # Find indecies where EOD token is.
71
+ eod_index = position_ids[b, data[b] == eod_token]
72
+ # Detach indecies from positions if going to modify positions.
73
+ if reset_position_ids:
74
+ eod_index = eod_index.clone()
75
+
76
+ # Loop through EOD indecies:
77
+ prev_index = 0
78
+ for j in range(eod_index.size()[0]):
79
+ i = eod_index[j]
80
+ # Mask attention loss.
81
+ if reset_attention_mask:
82
+ attention_mask[b, 0, (i + 1) :, : (i + 1)] = 0
83
+ # Reset positions.
84
+ if reset_position_ids:
85
+ position_ids[b, (i + 1) :] -= i + 1 - prev_index
86
+ prev_index = i + 1
87
+
88
+ # Convert attention mask to binary:
89
+ attention_mask = attention_mask < 0.5
90
+
91
+ return attention_mask, loss_mask, position_ids
92
+
93
+
94
+ def get_batch(context_tokens: torch.LongTensor, eod_id: int):
95
+ """Generate batch from context tokens."""
96
+ # Move to GPU.
97
+ tokens = context_tokens.contiguous().to(context_tokens.device)
98
+ # Get the attention mask and postition ids.
99
+ attention_mask, _, position_ids = get_ltor_masks_and_position_ids(
100
+ tokens,
101
+ eod_id,
102
+ reset_position_ids=False,
103
+ reset_attention_mask=False,
104
+ eod_mask_loss=False,
105
+ )
106
+ return tokens, attention_mask, position_ids
107
+
108
+
109
+ def get_stop_words_ids(chat_format, tokenizer):
110
+ if chat_format == "raw":
111
+ stop_words_ids = [tokenizer.encode("Human:"), [tokenizer.eod_id]]
112
+ elif chat_format == "chatml":
113
+ stop_words_ids = [[tokenizer.im_end_id], [tokenizer.im_start_id]]
114
+ else:
115
+ raise NotImplementedError(f"Unknown chat format {chat_format!r}")
116
+ return stop_words_ids
117
+
118
+
119
+ def make_context(
120
+ tokenizer: PreTrainedTokenizer,
121
+ query: str,
122
+ history: List[Tuple[str, str]] = None,
123
+ system: str = "",
124
+ max_window_size: int = 6144,
125
+ chat_format: str = "chatml",
126
+ ):
127
+ audio_info = None
128
+ if history is None:
129
+ history = []
130
+
131
+ if chat_format == "chatml":
132
+ im_start, im_end = "<|im_start|>", "<|im_end|>"
133
+ im_start_tokens = [tokenizer.im_start_id]
134
+ im_end_tokens = [tokenizer.im_end_id]
135
+ nl_tokens = tokenizer.encode("\n")
136
+
137
+ def _tokenize_str(role, content):
138
+ # import ipdb; ipdb.set_trace()
139
+ audio_info = tokenizer.process_audio(content)
140
+ return f"{role}\n{content}", tokenizer.encode(
141
+ role, allowed_special=set(tokenizer.AUDIO_ST), audio_info=audio_info
142
+ ) + nl_tokens + tokenizer.encode(content, allowed_special=set(tokenizer.AUDIO_ST), audio_info=audio_info),audio_info
143
+
144
+ system_text, system_tokens_part, audio_info = _tokenize_str("system", system)
145
+ system_tokens = im_start_tokens + system_tokens_part + im_end_tokens
146
+
147
+ raw_text = ""
148
+ context_tokens = []
149
+
150
+ for turn_query, turn_response in reversed(history):
151
+ query_text, query_tokens_part, _ = _tokenize_str("user", turn_query)
152
+ query_tokens = im_start_tokens + query_tokens_part + im_end_tokens
153
+ if turn_response is not None:
154
+ response_text, response_tokens_part, _ = _tokenize_str(
155
+ "assistant", turn_response
156
+ )
157
+ response_tokens = im_start_tokens + response_tokens_part + im_end_tokens
158
+
159
+ next_context_tokens = nl_tokens + query_tokens + nl_tokens + response_tokens
160
+ prev_chat = (
161
+ f"\n{im_start}{query_text}{im_end}\n{im_start}{response_text}{im_end}"
162
+ )
163
+ else:
164
+ next_context_tokens = nl_tokens + query_tokens + nl_tokens
165
+ prev_chat = f"\n{im_start}{query_text}{im_end}\n"
166
+
167
+ current_context_size = (
168
+ len(system_tokens) + len(next_context_tokens) + len(context_tokens)
169
+ )
170
+ if current_context_size < max_window_size:
171
+ context_tokens = next_context_tokens + context_tokens
172
+ raw_text = prev_chat + raw_text
173
+ else:
174
+ break
175
+
176
+ context_tokens = system_tokens + context_tokens
177
+ raw_text = f"{im_start}{system_text}{im_end}" + raw_text
178
+ context_tokens += (
179
+ nl_tokens
180
+ + im_start_tokens
181
+ + _tokenize_str("user", query)[1]
182
+ + im_end_tokens
183
+ + nl_tokens
184
+ + im_start_tokens
185
+ + tokenizer.encode("assistant")
186
+ + nl_tokens
187
+ )
188
+ raw_text += f"\n{im_start}user\n{query}{im_end}\n{im_start}assistant\n"
189
+ audio_info = tokenizer.process_audio(raw_text)
190
+
191
+ elif chat_format == "raw":
192
+ raw_text = query
193
+ context_tokens = tokenizer.encode(raw_text)
194
+ else:
195
+ raise NotImplementedError(f"Unknown chat format {chat_format!r}")
196
+
197
+ return raw_text, context_tokens, audio_info
198
+
199
+
200
+ def _decode_default(
201
+ tokens: List[int],
202
+ *,
203
+ stop_words: List[str],
204
+ eod_words: List[str],
205
+ tokenizer: PreTrainedTokenizer,
206
+ raw_text_len: int,
207
+ verbose: bool = False,
208
+ return_end_reason: bool = False,
209
+ errors: str='replace',
210
+ audio_info:Dict = None
211
+ ):
212
+ kwargs = {"audio_info": audio_info}
213
+ trim_decode_tokens = tokenizer.decode(tokens, errors=errors, **kwargs)[raw_text_len:]
214
+ if verbose:
215
+ print("\nRaw Generate: ", trim_decode_tokens)
216
+
217
+ end_reason = f"Gen length {len(tokens)}"
218
+ for stop_word in stop_words:
219
+ trim_decode_tokens = trim_decode_tokens.replace(stop_word, "").strip()
220
+ for eod_word in eod_words:
221
+ if eod_word in trim_decode_tokens:
222
+ end_reason = f"Gen {eod_word!r}"
223
+ trim_decode_tokens = trim_decode_tokens.split(eod_word)[0]
224
+ trim_decode_tokens = trim_decode_tokens.strip()
225
+ if verbose:
226
+ print("\nEnd Reason:", end_reason)
227
+ print("\nGenerate: ", trim_decode_tokens)
228
+
229
+ if return_end_reason:
230
+ return trim_decode_tokens, end_reason
231
+ else:
232
+ return trim_decode_tokens
233
+
234
+
235
+ def _decode_chatml(
236
+ tokens: List[int],
237
+ *,
238
+ stop_words: List[str],
239
+ eod_token_ids: List[int],
240
+ tokenizer: PreTrainedTokenizer,
241
+ raw_text_len: int,
242
+ context_length: int,
243
+ verbose: bool = False,
244
+ return_end_reason: bool = False,
245
+ errors: str='replace',
246
+ audio_info: Dict = None
247
+ ):
248
+ kwargs = {"audio_info": audio_info}
249
+ end_reason = f"Gen length {len(tokens)}"
250
+ eod_token_idx = context_length
251
+ for eod_token_idx in range(context_length, len(tokens)):
252
+ if tokens[eod_token_idx] in eod_token_ids:
253
+ end_reason = f"Gen {tokenizer.decode([tokens[eod_token_idx]],**kwargs)!r}"
254
+ break
255
+
256
+ trim_decode_tokens = tokenizer.decode(tokens[:eod_token_idx], errors=errors, **kwargs)[raw_text_len:]
257
+ if verbose:
258
+ print("\nRaw Generate w/o EOD:", tokenizer.decode(tokens, errors=errors, **kwargs)[raw_text_len:])
259
+ print("\nRaw Generate:", trim_decode_tokens)
260
+ print("\nEnd Reason:", end_reason)
261
+ for stop_word in stop_words:
262
+ trim_decode_tokens = trim_decode_tokens.replace(stop_word, "").strip()
263
+ trim_decode_tokens = trim_decode_tokens.strip()
264
+ if verbose:
265
+ print("\nGenerate:", trim_decode_tokens)
266
+
267
+ if return_end_reason:
268
+ return trim_decode_tokens, end_reason
269
+ else:
270
+ return trim_decode_tokens
271
+
272
+
273
+ def decode_tokens(
274
+ tokens: Union[torch.LongTensor, TokensType],
275
+ tokenizer: PreTrainedTokenizer,
276
+ raw_text_len: int,
277
+ context_length: int,
278
+ chat_format: str,
279
+ verbose: bool = False,
280
+ return_end_reason: bool = False,
281
+ errors: str="replace",
282
+ audio_info: Dict = None
283
+ ) -> str:
284
+ if torch.is_tensor(tokens):
285
+ tokens = tokens.cpu().numpy().tolist()
286
+
287
+ if chat_format == "chatml":
288
+ return _decode_chatml(
289
+ tokens,
290
+ stop_words=[],
291
+ eod_token_ids=[tokenizer.im_start_id, tokenizer.im_end_id],
292
+ tokenizer=tokenizer,
293
+ raw_text_len=raw_text_len,
294
+ context_length=context_length,
295
+ verbose=verbose,
296
+ return_end_reason=return_end_reason,
297
+ errors=errors,
298
+ audio_info=audio_info
299
+ )
300
+ elif chat_format == "raw":
301
+ return _decode_default(
302
+ tokens,
303
+ stop_words=["<|endoftext|>"],
304
+ eod_words=["<|endoftext|>"],
305
+ tokenizer=tokenizer,
306
+ raw_text_len=raw_text_len,
307
+ verbose=verbose,
308
+ return_end_reason=return_end_reason,
309
+ errors=errors,
310
+ audio_info=audio_info
311
+ )
312
+ else:
313
+ raise NotImplementedError(f"Unknown chat format {chat_format!r}")
314
+
315
+
316
+ class StopWordsLogitsProcessor(LogitsProcessor):
317
+ """
318
+ :class:`transformers.LogitsProcessor` that enforces that when specified sequences appear, stop geration.
319
+
320
+ Args:
321
+ stop_words_ids (:obj:`List[List[int]]`):
322
+ List of list of token ids of stop ids. In order to get the tokens of the words
323
+ that should not appear in the generated text, use :obj:`tokenizer(bad_word,
324
+ add_prefix_space=True).input_ids`.
325
+ eos_token_id (:obj:`int`):
326
+ The id of the `end-of-sequence` token.
327
+ """
328
+
329
+ def __init__(self, stop_words_ids: Iterable[Iterable[int]], eos_token_id: int):
330
+
331
+ if not isinstance(stop_words_ids, List) or len(stop_words_ids) == 0:
332
+ raise ValueError(
333
+ f"`stop_words_ids` has to be a non-emtpy list, but is {stop_words_ids}."
334
+ )
335
+ if any(not isinstance(bad_word_ids, list) for bad_word_ids in stop_words_ids):
336
+ raise ValueError(
337
+ f"`stop_words_ids` has to be a list of lists, but is {stop_words_ids}."
338
+ )
339
+ if any(
340
+ any(
341
+ (not isinstance(token_id, (int, np.integer)) or token_id < 0)
342
+ for token_id in stop_word_ids
343
+ )
344
+ for stop_word_ids in stop_words_ids
345
+ ):
346
+ raise ValueError(
347
+ f"Each list in `stop_words_ids` has to be a list of positive integers, but is {stop_words_ids}."
348
+ )
349
+
350
+ self.stop_words_ids = list(
351
+ filter(
352
+ lambda bad_token_seq: bad_token_seq != [eos_token_id], stop_words_ids
353
+ )
354
+ )
355
+ self.eos_token_id = eos_token_id
356
+ for stop_token_seq in self.stop_words_ids:
357
+ assert (
358
+ len(stop_token_seq) > 0
359
+ ), "Stop words token sequences {} cannot have an empty list".format(
360
+ stop_words_ids
361
+ )
362
+
363
+ def __call__(
364
+ self, input_ids: torch.LongTensor, scores: torch.FloatTensor
365
+ ) -> torch.FloatTensor:
366
+ stopped_samples = self._calc_stopped_samples(input_ids)
367
+ for i, should_stop in enumerate(stopped_samples):
368
+ if should_stop:
369
+ scores[i, self.eos_token_id] = float(2**15)
370
+ return scores
371
+
372
+ def _tokens_match(self, prev_tokens: torch.LongTensor, tokens: List[int]) -> bool:
373
+ if len(tokens) == 0:
374
+ # if bad word tokens is just one token always ban it
375
+ return True
376
+ elif len(tokens) > len(prev_tokens):
377
+ # if bad word tokens are longer then prev input_ids they can't be equal
378
+ return False
379
+ elif prev_tokens[-len(tokens) :].tolist() == tokens:
380
+ # if tokens match
381
+ return True
382
+ else:
383
+ return False
384
+
385
+ def _calc_stopped_samples(self, prev_input_ids: Iterable[int]) -> Iterable[int]:
386
+ stopped_samples = []
387
+ for prev_input_ids_slice in prev_input_ids:
388
+ match = False
389
+ for stop_token_seq in self.stop_words_ids:
390
+ if self._tokens_match(prev_input_ids_slice, stop_token_seq):
391
+ # if tokens do not match continue
392
+ match = True
393
+ break
394
+ stopped_samples.append(match)
395
+
396
+ return stopped_samples
397
+
398
+
399
+ def top_k_logits(logits, top_k=0, top_p=0.0, filter_value=-float("Inf")):
400
+ """This function has been mostly taken from huggingface conversational
401
+ ai code at
402
+ https://medium.com/huggingface/how-to-build-a-state-of-the-art-
403
+ conversational-ai-with-transfer-learning-2d818ac26313"""
404
+
405
+ if top_k > 0:
406
+ # Remove all tokens with a probability less than the
407
+ # last token of the top-k
408
+ indices_to_remove = logits < torch.topk(logits, top_k)[0][..., -1, None]
409
+ logits[indices_to_remove] = filter_value
410
+
411
+ if top_p > 0.0:
412
+ # Cconvert to 1D
413
+ sorted_logits, sorted_indices = torch.sort(logits, descending=True, dim=-1)
414
+ cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
415
+
416
+ # Remove tokens with cumulative probability above the threshold
417
+ sorted_indices_to_remove = cumulative_probs > top_p
418
+ # Shift the indices to the right to keep also the first token
419
+ # above the threshold
420
+ sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
421
+ sorted_indices_to_remove[..., 0] = 0
422
+ for i in range(sorted_indices.size(0)):
423
+ indices_to_remove = sorted_indices[i][sorted_indices_to_remove[i]]
424
+ logits[i][indices_to_remove] = filter_value
425
+
426
+ return logits
427
+
428
+
429
+ def switch(val1, val2, boolean):
430
+ boolean = boolean.type_as(val1)
431
+ return (1 - boolean) * val1 + boolean * val2
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {}
tokenization_qwen.py ADDED
@@ -0,0 +1,579 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright (c) Alibaba Cloud.
2
+ #
3
+ # This source code is licensed under the license found in the
4
+ # LICENSE file in the root directory of this source tree.
5
+
6
+ """Tokenization classes for QWen."""
7
+
8
+ import base64
9
+ import logging
10
+ import os
11
+ import re
12
+ import itertools
13
+
14
+ import requests
15
+ import unicodedata
16
+ from typing import Collection, Dict, List, Set, Tuple, Union, Any, Callable, Optional
17
+
18
+ import tiktoken
19
+ import numpy as np
20
+
21
+ from transformers import PreTrainedTokenizer, AddedToken
22
+ from transformers.utils import try_to_load_from_cache
23
+ from transformers.tokenization_utils_base import BatchEncoding, PaddingStrategy, TruncationStrategy, \
24
+ TextInput, TextInputPair, PreTokenizedInput, PreTokenizedInputPair, TensorType, EncodedInput, EncodedInputPair
25
+
26
+ import matplotlib.colors as mcolors
27
+ from matplotlib.font_manager import FontProperties
28
+ from .audio import *
29
+
30
+ logger = logging.getLogger(__name__)
31
+
32
+ VOCAB_FILES_NAMES = {"vocab_file": "qwen.tiktoken", "ttf": "SimSun.ttf"}
33
+
34
+ PAT_STR = r"""(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+"""
35
+ ENDOFTEXT = "<|endoftext|>"
36
+ IMSTART = "<|im_start|>"
37
+ IMEND = "<|im_end|>"
38
+ # as the default behavior is changed to allow special tokens in
39
+ # regular texts, the surface forms of special tokens need to be
40
+ # as different as possible to minimize the impact
41
+ EXTRAS = tuple((f"<|extra_{i}|>" for i in range(205)))
42
+ SPECIAL_TOKENS = (
43
+ ENDOFTEXT,
44
+ IMSTART,
45
+ IMEND,
46
+ ) + EXTRAS
47
+
48
+ LANGUAGES = {
49
+ "en": "english",
50
+ "zh": "chinese",
51
+ "de": "german",
52
+ "es": "spanish",
53
+ "ko": "korean",
54
+ "fr": "french",
55
+ "ja": "japanese",
56
+ "it": "italian",
57
+ }
58
+
59
+
60
+ def _load_tiktoken_bpe(tiktoken_bpe_file: str) -> Dict[bytes, int]:
61
+ with open(tiktoken_bpe_file, "rb") as f:
62
+ contents = f.read()
63
+ return {
64
+ base64.b64decode(token): int(rank)
65
+ for token, rank in (line.split() for line in contents.splitlines() if line)
66
+ }
67
+
68
+
69
+ def _list_find(
70
+ input_list: List[Any],
71
+ candidates: Tuple[Any],
72
+ start: int = 0,
73
+ ):
74
+ for i in range(start, len(input_list)):
75
+ if input_list[i] in candidates:
76
+ return i
77
+ return -1
78
+
79
+
80
+ def _replace_closed_tag(
81
+ input_tokens: List[Any],
82
+ start_tags: Union[Any, Tuple[Any]],
83
+ end_tags: Union[Any, Tuple[Any]],
84
+ inclusive_replace_func: Callable,
85
+ exclusive_replace_func: Callable = lambda x: x,
86
+ audio_info: Dict = None
87
+ ):
88
+ if isinstance(start_tags, (str, int)):
89
+ start_tags = (start_tags,)
90
+ if isinstance(end_tags, (str, int)):
91
+ end_tags = (end_tags,)
92
+ assert len(start_tags) == len(end_tags)
93
+
94
+ output_tokens = []
95
+ end = 0
96
+ audio_idx = 0
97
+ while True:
98
+ start = _list_find(input_tokens, start_tags, end)
99
+ if start == -1:
100
+ break
101
+ output_tokens.extend(exclusive_replace_func(input_tokens[end: start]))
102
+ tag_idx = start_tags.index(input_tokens[start])
103
+ end = _list_find(input_tokens, (end_tags[tag_idx],), start)
104
+ if end == -1:
105
+ raise ValueError("Unclosed audio token")
106
+ output_tokens.extend(inclusive_replace_func(input_tokens[start: end + 1], audio_info, audio_idx))
107
+ end += 1
108
+ audio_idx += 1
109
+ output_tokens.extend(exclusive_replace_func(input_tokens[end:]))
110
+ return output_tokens
111
+
112
+
113
+ class QWenTokenizer(PreTrainedTokenizer):
114
+ """QWen tokenizer."""
115
+
116
+ vocab_files_names = VOCAB_FILES_NAMES
117
+
118
+ def __init__(
119
+ self,
120
+ vocab_file,
121
+ errors="replace",
122
+ audio_start_tag='<audio>',
123
+ audio_end_tag='</audio>',
124
+ **kwargs,
125
+ ):
126
+ super().__init__(**kwargs)
127
+ self.audio_start_tag = audio_start_tag
128
+ self.audio_end_tag = audio_end_tag
129
+ self.audio_pad_tag = "[[[AUDIO:modality]]]"
130
+
131
+ self.AUDIO_ST = (
132
+ '[[[AUDIO:modality]]]',
133
+ # Transcription Tag
134
+ "<|startoftranscript|>", # Transcription
135
+ "<|startofanalysis|>", # Analysis
136
+ # Task Tag
137
+ "<|translate|>",
138
+ "<|transcribe|>",
139
+ "<|caption|>",
140
+ "<|keyword|>",
141
+ # Language Tag
142
+ "<|unknown|>", # unknown language
143
+ *[f"<|{lang}|>" for lang in LANGUAGES.keys()],
144
+ "<|zh_tr|>", # tranditional Chinese
145
+ # Timestamps Tag
146
+ "<|notimestamps|>",
147
+ "<|sil|>",
148
+ "<|timestamps|>",
149
+ *[f"<|{i * 0.01:.2f}|>" for i in range(3001)], # timestamps 0.00-30.00
150
+ # Output Instruction
151
+ "<|caption_audiocaps|>", # Audiocaps caption style
152
+ "<|caption_clotho|>", # Clotho caption style
153
+ "<|audioset_ontology|>", # Audioset ontology style
154
+ "<|caption_plain|>", # plain caption
155
+ "<|itn|>", # inversed text normalized
156
+ "<|wo_itn|>", # without inversed text normalized
157
+ "<|startofentityvalue|>",
158
+ "<|endofentityvalue|>",
159
+ "<|startofentitytype|>",
160
+ "<|endofentitytype|>",
161
+ "<|named_entity_recognition|>", # named entity recognition task
162
+ "<|audio_grounding|>",
163
+ "<|startofword|>",
164
+ "<|endofword|>",
165
+ "<|delim|>", # delimiter of timestamps pair in audio grounding
166
+ "<|emotion_recognition|>", # emotion recognition
167
+ "<|music_description|>", # music description
168
+ "<|note_analysis|>", # note analysis
169
+ "<|pitch|>", # note analysis: pitch
170
+ *[f"<|midi_pitch_{i}|>" for i in range(128)], # midi pitch 0-127
171
+ "<|velocity|>", # note analysis: velocity
172
+ *[f"<|midi_velocity_{i}|>" for i in range(128)], # midi velocity 0-127
173
+ "<|sonic|>", # note analysis: sonic
174
+ "<|instrument|>", # note analysis: instrument
175
+ "<|speaker_meta|>", # meta information of speaker
176
+ "<|song_meta|>", # meta information of song
177
+ "<|question|>", # AQA: question
178
+ "<|answer|>", # AQA: answer
179
+ "<|choice|>", # AQA: answer choice
180
+ "<|scene|>", # scene recognition
181
+ "<|event|>", # sound event
182
+ "<|vocal_classification|>", # vocal classification
183
+ "<|speech_understanding|>", # speech language understanding
184
+ "<|scenario|>", # speech language understanding: scenario
185
+ "<|action|>", # speech language understanding: action
186
+ "<|entities|>", # speech language understanding: entities
187
+ "<|speech_edit|>", # speech edit
188
+ audio_start_tag,
189
+ audio_end_tag
190
+ )
191
+
192
+ self.errors = errors # how to handle errors in decoding
193
+
194
+ self.mergeable_ranks = _load_tiktoken_bpe(vocab_file) # type: dict[bytes, int]
195
+ self.special_tokens = {
196
+ token: index
197
+ for index, token in enumerate(
198
+ SPECIAL_TOKENS + self.AUDIO_ST, start=len(self.mergeable_ranks)
199
+
200
+ )
201
+ }
202
+ self.audio_start_id = self.special_tokens[self.audio_start_tag]
203
+ self.audio_end_id = self.special_tokens[self.audio_end_tag]
204
+ self.audio_pad_id = self.special_tokens[self.audio_pad_tag]
205
+ print(f"audio_start_id: {self.audio_start_id}, "
206
+ f"audio_end_id: {self.audio_end_id}, "
207
+ f"audio_pad_id: {self.audio_pad_id}.")
208
+
209
+ enc = tiktoken.Encoding(
210
+ "Qwen",
211
+ pat_str=PAT_STR,
212
+ mergeable_ranks=self.mergeable_ranks,
213
+ special_tokens=self.special_tokens,
214
+ )
215
+ assert (
216
+ len(self.mergeable_ranks) + len(self.special_tokens) == enc.n_vocab
217
+ ), f"{len(self.mergeable_ranks) + len(self.special_tokens)} != {enc.n_vocab} in encoding"
218
+
219
+ self.decoder = {
220
+ v: k for k, v in self.mergeable_ranks.items()
221
+ } # type: dict[int, bytes|str]
222
+ self.decoder.update({v: k for k, v in self.special_tokens.items()})
223
+
224
+ self.tokenizer = enc # type: tiktoken.Encoding
225
+
226
+ self.eod_id = self.tokenizer.eot_token
227
+ self.im_start_id = self.special_tokens[IMSTART]
228
+ self.im_end_id = self.special_tokens[IMEND]
229
+
230
+ def __getstate__(self):
231
+ # for pickle lovers
232
+ state = self.__dict__.copy()
233
+ del state['tokenizer']
234
+ return state
235
+
236
+ def __setstate__(self, state):
237
+ # tokenizer is not python native; don't pass it; rebuild it
238
+ self.__dict__.update(state)
239
+ enc = tiktoken.Encoding(
240
+ "Qwen",
241
+ pat_str=PAT_STR,
242
+ mergeable_ranks=self.mergeable_ranks,
243
+ special_tokens=self.special_tokens,
244
+ )
245
+ self.tokenizer = enc
246
+
247
+ def __len__(self) -> int:
248
+ return self.tokenizer.n_vocab
249
+
250
+ def get_vocab(self) -> Dict[bytes, int]:
251
+ return self.mergeable_ranks
252
+
253
+ def convert_tokens_to_ids(
254
+ self, tokens: Union[bytes, str, List[Union[bytes, str]]]
255
+ ) -> List[int]:
256
+ ids = []
257
+ if isinstance(tokens, (str, bytes)):
258
+ if tokens in self.special_tokens:
259
+ return self.special_tokens[tokens]
260
+ else:
261
+ return self.mergeable_ranks.get(tokens)
262
+ for token in tokens:
263
+ if token in self.special_tokens:
264
+ ids.append(self.special_tokens[token])
265
+ else:
266
+ ids.append(self.mergeable_ranks.get(token))
267
+ return ids
268
+
269
+ def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int:
270
+ if not special_tokens and new_tokens:
271
+ raise ValueError('Adding regular tokens is not supported')
272
+ for token in new_tokens:
273
+ surface_form = token.content if isinstance(token, AddedToken) else token
274
+ if surface_form not in SPECIAL_TOKENS + self.AUDIO_ST:
275
+ raise ValueError('Adding unknown special tokens is not supported')
276
+ return 0
277
+
278
+ def save_vocabulary(self, save_directory: str, **kwargs) -> Tuple[str]:
279
+ """
280
+ Save only the vocabulary of the tokenizer (vocabulary).
281
+
282
+ Returns:
283
+ `Tuple(str)`: Paths to the files saved.
284
+ """
285
+ file_path = os.path.join(save_directory, "qwen.tiktoken")
286
+ with open(file_path, "w", encoding="utf8") as w:
287
+ for k, v in self.mergeable_ranks.items():
288
+ line = base64.b64encode(k).decode("utf8") + " " + str(v) + "\n"
289
+ w.write(line)
290
+ return (file_path,)
291
+
292
+ def tokenize(
293
+ self,
294
+ text: str,
295
+ allowed_special: Union[Set, str] = "all",
296
+ disallowed_special: Union[Collection, str] = (),
297
+ audio_info: Dict = None,
298
+ **kwargs,
299
+ ) -> List[Union[bytes, str]]:
300
+ """
301
+ Converts a string in a sequence of tokens.
302
+
303
+ Args:
304
+ text (`str`):
305
+ The sequence to be encoded.
306
+ allowed_special (`Literal["all"]` or `set`):
307
+ The surface forms of the tokens to be encoded as special tokens in regular texts.
308
+ Default to "all".
309
+ disallowed_special (`Literal["all"]` or `Collection`):
310
+ The surface forms of the tokens that should not be in regular texts and trigger errors.
311
+ Default to an empty tuple.
312
+
313
+ kwargs (additional keyword arguments, *optional*):
314
+ Will be passed to the underlying model specific encode method.
315
+
316
+ Returns:
317
+ `List[bytes|str]`: The list of tokens.
318
+ """
319
+ tokens = []
320
+ text = unicodedata.normalize("NFC", text)
321
+
322
+ # this implementation takes a detour: text -> token id -> token surface forms
323
+ for t in self.tokenizer.encode(
324
+ text, allowed_special=allowed_special, disallowed_special=disallowed_special
325
+ ):
326
+ tokens.append(self.decoder[t])
327
+
328
+ def _encode_audiourl(audio_tokens, audio_info, audio_idx):
329
+ assert audio_tokens[0] == self.audio_start_tag and audio_tokens[-1] == self.audio_end_tag
330
+ audio_token_span = audio_info['audio_span_tokens'][audio_idx]
331
+ out_audio_tokens = [self.audio_start_tag] + [self.audio_pad_tag] * (audio_token_span - 2) + [
332
+ self.audio_end_tag]
333
+ return out_audio_tokens
334
+
335
+ return _replace_closed_tag(tokens, self.audio_start_tag, self.audio_end_tag, _encode_audiourl,
336
+ audio_info=audio_info)
337
+
338
+ def _batch_encode_plus(
339
+ self,
340
+ batch_text_or_text_pairs: Union[
341
+ List[TextInput],
342
+ List[TextInputPair],
343
+ List[PreTokenizedInput],
344
+ List[PreTokenizedInputPair],
345
+ List[EncodedInput],
346
+ List[EncodedInputPair],
347
+ ],
348
+ add_special_tokens: bool = True,
349
+ padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
350
+ truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE,
351
+ max_length: Optional[int] = None,
352
+ stride: int = 0,
353
+ is_split_into_words: bool = False,
354
+ pad_to_multiple_of: Optional[int] = None,
355
+ return_tensors: Optional[Union[str, TensorType]] = None,
356
+ return_token_type_ids: Optional[bool] = None,
357
+ return_attention_mask: Optional[bool] = None,
358
+ return_overflowing_tokens: bool = False,
359
+ return_special_tokens_mask: bool = False,
360
+ return_offsets_mapping: bool = False,
361
+ return_length: bool = False,
362
+ verbose: bool = True,
363
+ **kwargs,
364
+ ) -> BatchEncoding:
365
+
366
+ def get_input_ids(text):
367
+ if isinstance(text, str):
368
+ tokens = self.tokenize(text, **kwargs)
369
+ return self.convert_tokens_to_ids(tokens)
370
+ elif isinstance(text, (list, tuple)) and len(text) > 0 and isinstance(text[0], str):
371
+ if is_split_into_words:
372
+ tokens = list(
373
+ itertools.chain(*(self.tokenize(t, is_split_into_words=True, **kwargs) for t in text))
374
+ )
375
+ return self.convert_tokens_to_ids(tokens)
376
+ else:
377
+ return self.convert_tokens_to_ids(text)
378
+ elif isinstance(text, (list, tuple)) and len(text) > 0 and isinstance(text[0], int):
379
+ return text
380
+ else:
381
+ raise ValueError(
382
+ "Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers."
383
+ )
384
+
385
+ if return_offsets_mapping:
386
+ raise NotImplementedError(
387
+ "return_offset_mapping is not available when using Python tokenizers. "
388
+ "To use this feature, change your tokenizer to one deriving from "
389
+ "transformers.PreTrainedTokenizerFast."
390
+ )
391
+
392
+ input_ids = []
393
+ audio_info = kwargs.pop("audio_info", None)
394
+ for pair_id in range(len(batch_text_or_text_pairs)):
395
+ kwargs['audio_info'] = audio_info[pair_id]
396
+ ids_or_pair_ids = batch_text_or_text_pairs[pair_id]
397
+ # for ids_or_pair_ids in batch_text_or_text_pairs:
398
+ if not isinstance(ids_or_pair_ids, (list, tuple)):
399
+ ids, pair_ids = ids_or_pair_ids, None
400
+ elif is_split_into_words and not isinstance(ids_or_pair_ids[0], (list, tuple)):
401
+ ids, pair_ids = ids_or_pair_ids, None
402
+ else:
403
+ ids, pair_ids = ids_or_pair_ids
404
+
405
+ first_ids = get_input_ids(ids)
406
+ second_ids = get_input_ids(pair_ids) if pair_ids is not None else None
407
+ input_ids.append((first_ids, second_ids))
408
+
409
+ batch_outputs = self._batch_prepare_for_model(
410
+ input_ids,
411
+ add_special_tokens=add_special_tokens,
412
+ padding_strategy=padding_strategy,
413
+ truncation_strategy=truncation_strategy,
414
+ max_length=max_length,
415
+ stride=stride,
416
+ pad_to_multiple_of=pad_to_multiple_of,
417
+ return_attention_mask=return_attention_mask,
418
+ return_token_type_ids=return_token_type_ids,
419
+ return_overflowing_tokens=return_overflowing_tokens,
420
+ return_special_tokens_mask=return_special_tokens_mask,
421
+ return_length=return_length,
422
+ return_tensors=return_tensors,
423
+ verbose=verbose,
424
+ )
425
+
426
+ return BatchEncoding(batch_outputs)
427
+
428
+ def convert_tokens_to_string(self, tokens: List[Union[bytes, str]]) -> str:
429
+ """
430
+ Converts a sequence of tokens in a single string.
431
+ """
432
+ text = ""
433
+ temp = b""
434
+ for t in tokens:
435
+ if isinstance(t, str):
436
+ if temp:
437
+ text += temp.decode("utf-8", errors=self.errors)
438
+ temp = b""
439
+ text += t
440
+ elif isinstance(t, bytes):
441
+ temp += t
442
+ else:
443
+ raise TypeError("token should only be of type types or str")
444
+ if temp:
445
+ text += temp.decode("utf-8", errors=self.errors)
446
+ return text
447
+
448
+ @property
449
+ def vocab_size(self):
450
+ return self.tokenizer.n_vocab
451
+
452
+ def _convert_id_to_token(self, index: int) -> Union[bytes, str]:
453
+ """Converts an id to a token, special tokens included"""
454
+ if index in self.decoder:
455
+ return self.decoder[index]
456
+ raise ValueError("unknown ids")
457
+
458
+ def _convert_token_to_id(self, token: Union[bytes, str]) -> int:
459
+ """Converts a token to an id using the vocab, special tokens included"""
460
+ if token in self.special_tokens:
461
+ return self.special_tokens[token]
462
+ if token in self.mergeable_ranks:
463
+ return self.mergeable_ranks[token]
464
+ raise ValueError("unknown token")
465
+
466
+ def _tokenize(self, text: str, **kwargs):
467
+ """
468
+ Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based
469
+ vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces).
470
+
471
+ Do NOT take care of added tokens.
472
+ """
473
+ raise NotImplementedError
474
+
475
+ def _decode(
476
+ self,
477
+ token_ids: Union[int, List[int]],
478
+ skip_special_tokens: bool = False,
479
+ errors: str = None,
480
+ **kwargs,
481
+ ) -> str:
482
+ if isinstance(token_ids, int):
483
+ token_ids = [token_ids]
484
+ audio_info = kwargs.pop("audio_info", None)
485
+
486
+ def _decode_audiourl(audio_token_ids, audio_info, audio_idx):
487
+ assert audio_token_ids[0] == self.audio_start_id and audio_token_ids[-1] == self.audio_end_id
488
+ audio_url = audio_info["audio_urls"][audio_idx]
489
+ return [self.audio_start_id] + self.tokenizer.encode(audio_url) + [self.audio_end_id]
490
+
491
+ token_ids = _replace_closed_tag(token_ids, self.audio_start_id, self.audio_end_id, _decode_audiourl,
492
+ audio_info=audio_info)
493
+
494
+ if skip_special_tokens:
495
+ token_ids = [i for i in token_ids if i < self.eod_id]
496
+ return self.tokenizer.decode(token_ids, errors=errors or self.errors)
497
+
498
+ def to_list_format(self, text: str):
499
+ text = unicodedata.normalize("NFC", text)
500
+ token_ids = self.tokenizer.encode(
501
+ text, allowed_special=set(self.AUDIO_ST + (ENDOFTEXT,)))
502
+
503
+ def _encode_audio_info(tokens):
504
+ if len(tokens) == 0:
505
+ return []
506
+ if tokens[0] == self.audio_start_id and tokens[-1] == self.audio_end_id:
507
+ key = 'audio'
508
+ else:
509
+ _tobytes = lambda x: x.encode('utf-8') if isinstance(x, str) else x
510
+ return [{'text': b''.join(map(_tobytes, map(self.decoder.get, tokens))).decode('utf-8')}]
511
+ _tobytes = lambda x: x.encode('utf-8') if isinstance(x, str) else x
512
+ val = b''.join(map(_tobytes, map(self.decoder.get, tokens[1:-1]))).decode('utf-8')
513
+ return [{key: val}]
514
+
515
+ return _replace_closed_tag(
516
+ token_ids,
517
+ (self.audio_start_id),
518
+ (self.audio_end_id),
519
+ _encode_audio_info,
520
+ _encode_audio_info,
521
+ )
522
+
523
+ def from_list_format(self, list_format: List[Dict]):
524
+ text = ''
525
+ num_audios = 0
526
+ for ele in list_format:
527
+ if 'audio' in ele:
528
+ num_audios += 1
529
+ text += f'Audio {num_audios}:'
530
+ text += self.audio_start_tag + ele['audio'] + self.audio_end_tag
531
+ text += '\n'
532
+ elif 'text' in ele:
533
+ text += ele['text']
534
+ elif 'box' in ele:
535
+ if 'ref' in ele:
536
+ text += self.ref_start_tag + ele['ref'] + self.ref_end_tag
537
+ for box in ele['box']:
538
+ text += self.box_start_tag + '(%d,%d),(%d,%d)' % (box[0], box[1], box[2], box[3]) + self.box_end_tag
539
+ else:
540
+ raise ValueError("Unsupport element: " + str(ele))
541
+ return text
542
+
543
+ def extract_audio_urls(self, text):
544
+ pattern = rf"{self.audio_start_tag}(.*?){self.audio_end_tag}"
545
+ return re.findall(pattern, text)
546
+
547
+ def process_audio(self, text):
548
+ audio_urls = self.extract_audio_urls(text)
549
+ if len(audio_urls) > 0:
550
+ audios, audio_lens, audio_span_tokens = [], [], []
551
+ for audio_path in audio_urls:
552
+ if audio_path.startswith("http://") or audio_path.startswith("https://"): # http
553
+ data = bytes(requests.get(audio_path, stream=True).content)
554
+ audio = load_bytesio_audio(data)
555
+ else:
556
+ audio = load_audio(audio_path)
557
+ L = (audio.shape[0] if audio.shape[0] <= 480000 else 480000) # max_length < 30s
558
+ mel_len = L // 160
559
+ audio = pad_or_trim(audio.flatten())
560
+ mel = log_mel_spectrogram(audio)
561
+ audio_len_after_cnn = get_T_after_cnn(mel_len)
562
+ audio_token_num = (audio_len_after_cnn - 2) // 2 + 1
563
+ audio_len = [audio_len_after_cnn, audio_token_num]
564
+ audios.append(mel)
565
+ audio_lens.append(audio_len)
566
+ audio_span_tokens.append(audio_token_num + 2) # add audio bos eos
567
+ input_audio_lengths = torch.IntTensor(audio_lens)
568
+ input_audios = torch.stack(audios, dim=0)
569
+ return {"input_audios": input_audios,
570
+ "input_audio_lengths": input_audio_lengths,
571
+ "audio_span_tokens": audio_span_tokens,
572
+ "audio_urls": audio_urls}
573
+ else:
574
+ return None
575
+
576
+
577
+
578
+
579
+
tokenizer_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_map": {
3
+ "AutoTokenizer": [
4
+ "tokenization_qwen.QWenTokenizer",
5
+ null
6
+ ]
7
+ },
8
+ "clean_up_tokenization_spaces": true,
9
+ "model_max_length": 2048,
10
+ "tokenizer_class": "QWenTokenizer"
11
+ }