Commit
•
5691eac
1
Parent(s):
8f4c61d
Upload folder using huggingface_hub
Browse files- LICENSE +209 -0
- README.md +57 -0
- added_tokens.json +40 -0
- config.json +41 -0
- configuration_quasar.py +253 -0
- generation_config.json +6 -0
- merges.txt +0 -0
- model-00001-of-00004.safetensors +3 -0
- model-00002-of-00004.safetensors +3 -0
- model-00003-of-00004.safetensors +3 -0
- model-00004-of-00004.safetensors +3 -0
- model.safetensors.index.json +696 -0
- modeling_quasar.py +1494 -0
- special_tokens_map.json +23 -0
- tokenizer.json +0 -0
- tokenizer_config.json +323 -0
- vocab.json +0 -0
LICENSE
ADDED
|
@@ -0,0 +1,209 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Apache License
|
| 2 |
+
Version 2.0, January 2004
|
| 3 |
+
http://www.apache.org/licenses/
|
| 4 |
+
|
| 5 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 6 |
+
|
| 7 |
+
1. Definitions.
|
| 8 |
+
|
| 9 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 10 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 11 |
+
|
| 12 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 13 |
+
the copyright owner that is granting the License.
|
| 14 |
+
|
| 15 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 16 |
+
other entities that control, are controlled by, or are under common
|
| 17 |
+
control with that entity. For the purposes of this definition,
|
| 18 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 19 |
+
direction or management of such entity, whether by contract or
|
| 20 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 21 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 22 |
+
|
| 23 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 24 |
+
exercising permissions granted by this License.
|
| 25 |
+
|
| 26 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 27 |
+
including but not limited to software source code, documentation
|
| 28 |
+
source, and configuration files.
|
| 29 |
+
|
| 30 |
+
"Object" form shall mean any form resulting from mechanical
|
| 31 |
+
transformation or translation of a Source form, including but
|
| 32 |
+
not limited to compiled object code, generated documentation,
|
| 33 |
+
and conversions to other media types.
|
| 34 |
+
|
| 35 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 36 |
+
Object form, made available under the License, as indicated by a
|
| 37 |
+
copyright notice that is included in or attached to the work
|
| 38 |
+
(an example is provided in the Appendix below).
|
| 39 |
+
|
| 40 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 41 |
+
form, that is based on (or derived from) the Work and for which the
|
| 42 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 43 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 44 |
+
of this License, Derivative Works shall not include works that remain
|
| 45 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 46 |
+
the Work and Derivative Works thereof.
|
| 47 |
+
|
| 48 |
+
"Contribution" shall mean any work of authorship, including
|
| 49 |
+
the original version of the Work and any modifications or additions
|
| 50 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 51 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 52 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 53 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 54 |
+
means any form of electronic, verbal, or written communication sent
|
| 55 |
+
to the Licensor or its representatives, including but not limited to
|
| 56 |
+
communication on electronic mailing lists, source code control systems,
|
| 57 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 58 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 59 |
+
excluding communication that is conspicuously marked or otherwise
|
| 60 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 61 |
+
|
| 62 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 63 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 64 |
+
subsequently incorporated within the Work.
|
| 65 |
+
|
| 66 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 67 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 68 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 69 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 70 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 71 |
+
Work and such Derivative Works in Source or Object form.
|
| 72 |
+
|
| 73 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 74 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 75 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 76 |
+
(except as stated in this section) patent license to make, have made,
|
| 77 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 78 |
+
where such license applies only to those patent claims licensable
|
| 79 |
+
by such Contributor that are necessarily infringed by their
|
| 80 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 81 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 82 |
+
institute patent litigation against any entity (including a
|
| 83 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 84 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 85 |
+
or contributory patent infringement, then any patent licenses
|
| 86 |
+
granted to You under this License for that Work shall terminate
|
| 87 |
+
as of the date such litigation is filed.
|
| 88 |
+
|
| 89 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 90 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 91 |
+
modifications, and in Source or Object form, provided that You
|
| 92 |
+
meet the following conditions:
|
| 93 |
+
|
| 94 |
+
(a) You must give any other recipients of the Work or
|
| 95 |
+
Derivative Works a copy of this License; and
|
| 96 |
+
|
| 97 |
+
(b) You must cause any modified files to carry prominent notices
|
| 98 |
+
stating that You changed the files; and
|
| 99 |
+
|
| 100 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 101 |
+
that You distribute, all copyright, patent, trademark, and
|
| 102 |
+
attribution notices from the Source form of the Work,
|
| 103 |
+
excluding those notices that do not pertain to any part of
|
| 104 |
+
the Derivative Works; and
|
| 105 |
+
|
| 106 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 107 |
+
distribution, then any Derivative Works that You distribute must
|
| 108 |
+
include a readable copy of the attribution notices contained
|
| 109 |
+
within such NOTICE file, excluding those notices that do not
|
| 110 |
+
pertain to any part of the Derivative Works, in at least one
|
| 111 |
+
of the following places: within a NOTICE text file distributed
|
| 112 |
+
as part of the Derivative Works; within the Source form or
|
| 113 |
+
documentation, if provided along with the Derivative Works; or,
|
| 114 |
+
within a display generated by the Derivative Works, if and
|
| 115 |
+
wherever such third-party notices normally appear. The contents
|
| 116 |
+
of the NOTICE file are for informational purposes only and
|
| 117 |
+
do not modify the License. You may add Your own attribution
|
| 118 |
+
notices within Derivative Works that You distribute, alongside
|
| 119 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 120 |
+
that such additional attribution notices cannot be construed
|
| 121 |
+
as modifying the License.
|
| 122 |
+
|
| 123 |
+
You may add Your own copyright statement to Your modifications and
|
| 124 |
+
may provide additional or different license terms and conditions
|
| 125 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 126 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 127 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 128 |
+
the conditions stated in this License.
|
| 129 |
+
|
| 130 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 131 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 132 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 133 |
+
this License, without any additional terms or conditions.
|
| 134 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 135 |
+
the terms of any separate license agreement you may have executed
|
| 136 |
+
with Licensor regarding such Contributions.
|
| 137 |
+
|
| 138 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 139 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 140 |
+
except as required for reasonable and customary use in describing the
|
| 141 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 142 |
+
|
| 143 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 144 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 145 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 146 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 147 |
+
implied, including, without limitation, any warranties or conditions
|
| 148 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 149 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 150 |
+
appropriateness of using or redistributing the Work and assume any
|
| 151 |
+
risks associated with Your exercise of permissions under this License.
|
| 152 |
+
|
| 153 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 154 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 155 |
+
unless required by applicable law (such as deliberate and grossly
|
| 156 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 157 |
+
liable to You for damages, including any direct, indirect, special,
|
| 158 |
+
incidental, or consequential damages of any character arising as a
|
| 159 |
+
result of this License or out of the use or inability to use the
|
| 160 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 161 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 162 |
+
other commercial damages or losses), even if such Contributor
|
| 163 |
+
has been advised of the possibility of such damages.
|
| 164 |
+
|
| 165 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 166 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 167 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 168 |
+
or other liability obligations and/or rights consistent with this
|
| 169 |
+
License. However, in accepting such obligations, You may act only
|
| 170 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 171 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 172 |
+
defend, and hold each Contributor harmless for any liability
|
| 173 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 174 |
+
of your accepting any such warranty or additional liability.
|
| 175 |
+
|
| 176 |
+
10. Addendum to the Apache License, Version 2.0 - Flexible Termination Provision
|
| 177 |
+
This Addendum is an integral part of the Apache License, Version 2.0
|
| 178 |
+
(the "License") under which [Your Project Name] ("The Software")
|
| 179 |
+
is licensed. All terms of the Apache License, Version 2.0, apply to
|
| 180 |
+
the Software, except as modified herein:
|
| 181 |
+
Notwithstanding any other provision of this License, AstraMind reserves
|
| 182 |
+
the right to modify the terms of the License for future versions of
|
| 183 |
+
The Software or to terminate the License for any new versions of
|
| 184 |
+
The Software that have not yet been released. AstraMind commits to
|
| 185 |
+
providing reasonable notice of such changes or termination to the community
|
| 186 |
+
and stakeholders.
|
| 187 |
+
This right is asserted with the understanding that it will be exercised in
|
| 188 |
+
exceptional circumstances and with a commitment to fairness and transparency.
|
| 189 |
+
The goal of this provision is to allow AstraMind the flexibility to respond
|
| 190 |
+
to changing legal, technical, or business environments.
|
| 191 |
+
For clarity, this provision does not affect the License as it applies to versions
|
| 192 |
+
of The Software that have already been released under the Apache License,
|
| 193 |
+
Version 2.0, prior to any such modification or termination of the License
|
| 194 |
+
for future versions.
|
| 195 |
+
END OF TERMS AND CONDITIONS
|
| 196 |
+
|
| 197 |
+
Copyright 2024 AstraMindAI
|
| 198 |
+
|
| 199 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 200 |
+
you may not use this file except in compliance with the License.
|
| 201 |
+
You may obtain a copy of the License at
|
| 202 |
+
|
| 203 |
+
https://huggingface.co/AstraMindAI/AstraPulsar-Phi/blob/main/LICENSE
|
| 204 |
+
|
| 205 |
+
Unless required by applicable law or agreed to in writing, software
|
| 206 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 207 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 208 |
+
See the License for the specific language governing permissions and
|
| 209 |
+
limitations under the License.
|
README.md
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license:
|
| 3 |
+
- other
|
| 4 |
+
language:
|
| 5 |
+
- en
|
| 6 |
+
pipeline_tag: text-generation
|
| 7 |
+
tags:
|
| 8 |
+
- pretrained
|
| 9 |
+
inference:
|
| 10 |
+
parameters:
|
| 11 |
+
temperature: 0.7
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
## Model sheet for AstraQuasar-4.5B-v.0.1
|
| 15 |
+
|
| 16 |
+
**AstraQuasar-4.5B-v.0.1** is our first pre-trained Large Language Model (LLM) for text generation. It is a model with **4.5B parameters**.
|
| 17 |
+
AstraQuasar-4.5B-v.0.1 is built upon the foundation of the Phi-2 architecture, with **significant enhancements including an increased number of layers and the innovative introduction of a novel technique known as the duplicate trick.**
|
| 18 |
+
|
| 19 |
+
AstraQuasar-4.5B-v.0.1 at the moment is an under trained model. Serving as a demonstration of the potential of the duplication trick and its implications for future advancements in language modeling. Despite its nascent status, our model has already demonstrated superior performance compared to both the base Phi-2 model and earlier iterations of AstraQuasar-4.5B that do not utilize the duplication trick.
|
| 20 |
+
|
| 21 |
+
One of the key milestones achieved by AstraQuasar-4.5B-v.0.1 is its successful application of backpropagation on the duplication trick, setting a precedent for future research and development in this area.
|
| 22 |
+
|
| 23 |
+
Our model's architecture is fully compatible with leading training frameworks such as Axolotl and LLaMA Factory, ensuring seamless integration into existing workflows leveraging the standard Hugging Face Transformers library.
|
| 24 |
+
|
| 25 |
+
## Example:
|
| 26 |
+
AstraQuasar-4.5B-v.0.1 can be easily instantiated using the Hugging Face Transformers library:
|
| 27 |
+
|
| 28 |
+
from transformers import AutoTokenizer, AutoModelForCausalLM
|
| 29 |
+
|
| 30 |
+
model = AutoModelForCausalLM.from_pretrained("AstraMindAI/AstraQuasar-4.5B", trust_remote_code=True)
|
| 31 |
+
tokenizer = AutoTokenizer.from_pretrained("AstraMindAI/AstraQuasar-4.5B")
|
| 32 |
+
|
| 33 |
+
# you can optionally disable the duplicate trick
|
| 34 |
+
# model.model.duplicate_trick = False
|
| 35 |
+
|
| 36 |
+
# You can also disable the duplicate gradient calculation
|
| 37 |
+
# model.model.duplicate_grad = False
|
| 38 |
+
|
| 39 |
+
# You can specify the layer ranges for the duplicate trick
|
| 40 |
+
# model.model.layer_ranges = [(0, 16),(8, 24),(17, 32),(25, 40),(33, 49),(40, 56)]
|
| 41 |
+
|
| 42 |
+
prompt = "This is an example script ."
|
| 43 |
+
inputs = tokenizer(prompt, return_tensors="pt")
|
| 44 |
+
|
| 45 |
+
# Generate
|
| 46 |
+
generate_ids = model.generate(inputs.input_ids, max_length=30)
|
| 47 |
+
tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
|
| 48 |
+
|
| 49 |
+
Pre-training and fine-tuning can be performed using **accelerate**.
|
| 50 |
+
|
| 51 |
+
## Notice
|
| 52 |
+
|
| 53 |
+
It's important to note that AstraQuasar-4.5B-v.0.1 is a pre-trained base model and does not incorporate any moderation mechanisms.
|
| 54 |
+
|
| 55 |
+
## NEWS
|
| 56 |
+
|
| 57 |
+
Stay tuned for exciting developments! A new architecture, **AstraPulsar**, is on the horizon, promising further advancements in language modeling.
|
added_tokens.json
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"\t\t": 50294,
|
| 3 |
+
"\t\t\t": 50293,
|
| 4 |
+
"\t\t\t\t": 50292,
|
| 5 |
+
"\t\t\t\t\t": 50291,
|
| 6 |
+
"\t\t\t\t\t\t": 50290,
|
| 7 |
+
"\t\t\t\t\t\t\t": 50289,
|
| 8 |
+
"\t\t\t\t\t\t\t\t": 50288,
|
| 9 |
+
"\t\t\t\t\t\t\t\t\t": 50287,
|
| 10 |
+
" ": 50286,
|
| 11 |
+
" ": 50285,
|
| 12 |
+
" ": 50284,
|
| 13 |
+
" ": 50283,
|
| 14 |
+
" ": 50282,
|
| 15 |
+
" ": 50281,
|
| 16 |
+
" ": 50280,
|
| 17 |
+
" ": 50279,
|
| 18 |
+
" ": 50278,
|
| 19 |
+
" ": 50277,
|
| 20 |
+
" ": 50276,
|
| 21 |
+
" ": 50275,
|
| 22 |
+
" ": 50274,
|
| 23 |
+
" ": 50273,
|
| 24 |
+
" ": 50272,
|
| 25 |
+
" ": 50271,
|
| 26 |
+
" ": 50270,
|
| 27 |
+
" ": 50269,
|
| 28 |
+
" ": 50268,
|
| 29 |
+
" ": 50267,
|
| 30 |
+
" ": 50266,
|
| 31 |
+
" ": 50265,
|
| 32 |
+
" ": 50264,
|
| 33 |
+
" ": 50263,
|
| 34 |
+
" ": 50262,
|
| 35 |
+
" ": 50261,
|
| 36 |
+
" ": 50260,
|
| 37 |
+
" ": 50259,
|
| 38 |
+
" ": 50258,
|
| 39 |
+
" ": 50257
|
| 40 |
+
}
|
config.json
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_name_or_path": "AstraMindAI/AstraQuasar-4.5B",
|
| 3 |
+
"architectures": [
|
| 4 |
+
"QuasarForCausalLM"
|
| 5 |
+
],
|
| 6 |
+
"attention_dropout": 0.0,
|
| 7 |
+
"auto_map": {
|
| 8 |
+
"AutoConfig": "AstraMindAI/AstraQuasar-4.5B--configuration_quasar.QuasarConfig",
|
| 9 |
+
"AutoModelForCausalLM": "AstraMindAI/AstraQuasar-4.5B--modeling_quasar.QuasarForCausalLM"
|
| 10 |
+
},
|
| 11 |
+
"bos_token_id": 50256,
|
| 12 |
+
"embd_pdrop": 0.0,
|
| 13 |
+
"eos_token_id": 50256,
|
| 14 |
+
"gated_activation": false,
|
| 15 |
+
"remove_ff_bias": true,
|
| 16 |
+
"simple_norm": false,
|
| 17 |
+
"hidden_act": "gelu_new",
|
| 18 |
+
"hidden_size": 2560,
|
| 19 |
+
"initializer_range": 0.02,
|
| 20 |
+
"intermediate_size": 10240,
|
| 21 |
+
"layer_norm_eps": 1e-05,
|
| 22 |
+
"max_position_embeddings": 32768,
|
| 23 |
+
"model_type": "phi",
|
| 24 |
+
"num_attention_heads": 32,
|
| 25 |
+
"num_hidden_layers": 57,
|
| 26 |
+
"num_key_value_heads": 32,
|
| 27 |
+
"partial_rotary_factor": 0.4,
|
| 28 |
+
"qk_layernorm": false,
|
| 29 |
+
"resid_pdrop": 0.1,
|
| 30 |
+
"rope_scaling": null,
|
| 31 |
+
"rope_theta": 10000.0,
|
| 32 |
+
"tie_word_embeddings": false,
|
| 33 |
+
"torch_dtype": "float16",
|
| 34 |
+
"transformers_version": "4.38.0.dev0",
|
| 35 |
+
"use_cache": true,
|
| 36 |
+
"vocab_size": 51200,
|
| 37 |
+
"duplicate_trick": true,
|
| 38 |
+
"duplicate_grad": true,
|
| 39 |
+
"layer_ranges": [[0, 16],[8, 24],[17, 32],[25, 40],[33, 49],[40, 56]],
|
| 40 |
+
"sliding_window": 2048
|
| 41 |
+
}
|
configuration_quasar.py
ADDED
|
@@ -0,0 +1,253 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# coding=utf-8
|
| 2 |
+
# Copyright 2024 AstraMind and the HuggingFace Inc. team. All rights reserved.
|
| 3 |
+
#
|
| 4 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 5 |
+
# you may not use this file except in compliance with the License.
|
| 6 |
+
# You may obtain a copy of the License at
|
| 7 |
+
#
|
| 8 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 9 |
+
#
|
| 10 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 11 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 12 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 13 |
+
# See the License for the specific language governing permissions and
|
| 14 |
+
# limitations under the License.
|
| 15 |
+
|
| 16 |
+
""" Quasar model configuration"""
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
from transformers.configuration_utils import PretrainedConfig
|
| 20 |
+
from transformers.utils import logging
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
logger = logging.get_logger(__name__)
|
| 24 |
+
|
| 25 |
+
QUASAR_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
| 26 |
+
"AstraMindAI/AstraQuasar-4.5B": "https://huggingface.co/AstraMindAI/AstraQuasar-4.5B/resolve/main/config.json",
|
| 27 |
+
}
|
| 28 |
+
|
| 29 |
+
#From phi-2 Phi -> Quasar
|
| 30 |
+
class QuasarConfig(PretrainedConfig):
|
| 31 |
+
r"""
|
| 32 |
+
This is the configuration class to store the configuration of a [`QuasarModel`]. It is used to instantiate an Quasar
|
| 33 |
+
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
|
| 34 |
+
defaults will yield a similar configuration to that of the Quasar
|
| 35 |
+
[microsoft/quasar-1](https://huggingface.co/microsoft/quasar-1).
|
| 36 |
+
|
| 37 |
+
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
|
| 38 |
+
documentation from [`PretrainedConfig`] for more information.
|
| 39 |
+
|
| 40 |
+
Args:
|
| 41 |
+
vocab_size (`int`, *optional*, defaults to 51200):
|
| 42 |
+
Vocabulary size of the Quasar model. Defines the number of different tokens that can be represented by the
|
| 43 |
+
`inputs_ids` passed when calling [`QuasarModel`].
|
| 44 |
+
hidden_size (`int`, *optional*, defaults to 2048):
|
| 45 |
+
Dimension of the hidden representations.
|
| 46 |
+
intermediate_size (`int`, *optional*, defaults to 8192):
|
| 47 |
+
Dimension of the MLP representations.
|
| 48 |
+
num_hidden_layers (`int`, *optional*, defaults to 24):
|
| 49 |
+
Number of hidden layers in the Transformer decoder.
|
| 50 |
+
num_attention_heads (`int`, *optional*, defaults to 32):
|
| 51 |
+
Number of attention heads for each attention layer in the Transformer decoder.
|
| 52 |
+
num_key_value_heads (`int`, *optional*):
|
| 53 |
+
This is the number of key_value heads that should be used to implement Grouped Query Attention. If
|
| 54 |
+
`num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
|
| 55 |
+
`num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
|
| 56 |
+
converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
|
| 57 |
+
by meanpooling all the original heads within that group. For more details checkout [this
|
| 58 |
+
paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
|
| 59 |
+
`num_attention_heads`.
|
| 60 |
+
resid_pdrop (`float`, *optional*, defaults to 0.0):
|
| 61 |
+
Dropout probability for mlp outputs.
|
| 62 |
+
embd_pdrop (`int`, *optional*, defaults to 0.0):
|
| 63 |
+
The dropout ratio for the embeddings.
|
| 64 |
+
attention_dropout (`float`, *optional*, defaults to 0.0):
|
| 65 |
+
The dropout ratio after computing the attention scores.
|
| 66 |
+
hidden_act (`str` or `function`, *optional*, defaults to `"gelu_new"`):
|
| 67 |
+
The non-linear activation function (function or string) in the decoder.
|
| 68 |
+
max_position_embeddings (`int`, *optional*, defaults to 2048):
|
| 69 |
+
The maximum sequence length that this model might ever be used with. Quasar-1 and Quasar-1.5 supports up to 2048
|
| 70 |
+
tokens.
|
| 71 |
+
initializer_range (`float`, *optional*, defaults to 0.02):
|
| 72 |
+
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
| 73 |
+
layer_norm_eps (`float`, *optional*, defaults to 1e-05):
|
| 74 |
+
The epsilon used by the rms normalization layers.
|
| 75 |
+
use_cache (`bool`, *optional*, defaults to `True`):
|
| 76 |
+
Whether or not the model should return the last key/values attentions (not used by all models). Only
|
| 77 |
+
relevant if `config.is_decoder=True`. Whether to tie weight embeddings or not.
|
| 78 |
+
tie_word_embeddings (`bool`, *optional*, defaults to `False`):
|
| 79 |
+
Whether to tie weight embeddings
|
| 80 |
+
rope_theta (`float`, *optional*, defaults to 10000.0):
|
| 81 |
+
The base period of the RoPE embeddings.
|
| 82 |
+
rope_scaling (`Dict`, *optional*):
|
| 83 |
+
Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
|
| 84 |
+
strategies: linear and dynamic. Their scaling factor must be an float greater than 1. The expected format
|
| 85 |
+
is `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
|
| 86 |
+
`max_position_embeddings` to the expected new maximum. See the following thread for more information on how
|
| 87 |
+
these scaling strategies behave:
|
| 88 |
+
https://www.reddit.com/r/LocalPersimmon/comments/14mrgpr/dynamically_scaled_rope_further_increases/. This
|
| 89 |
+
is an experimental feature, subject to breaking API changes in future versions.
|
| 90 |
+
partial_rotary_factor (`float`, *optional*, defaults to 0.5):
|
| 91 |
+
Percentage of the query and keys which will have rotary embedding.
|
| 92 |
+
qk_layernorm (`bool`, *optional*, defaults to `False`):
|
| 93 |
+
Whether or not to normalize the Queries and Keys after projecting the hidden states.
|
| 94 |
+
bos_token_id (`int`, *optional*, defaults to 1):
|
| 95 |
+
Denotes beginning of sequences token id.
|
| 96 |
+
eos_token_id (`int`, *optional*, defaults to 2):
|
| 97 |
+
Denotes end of sequences token id.
|
| 98 |
+
|
| 99 |
+
Example:
|
| 100 |
+
|
| 101 |
+
```python
|
| 102 |
+
>>> from transformers import AutoModel, QuasarConfig
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
>>> # Initializing a Quasar-1 style configuration
|
| 106 |
+
>>> configuration = QuasarConfig.from_pretrained("AstraMindAI/AstraQuasar-4.5B")
|
| 107 |
+
|
| 108 |
+
>>> # Initializing a model from the configuration
|
| 109 |
+
>>> model = QuasarModel(configuration, trust_remote_code=True)
|
| 110 |
+
|
| 111 |
+
>>> # Accessing the model configuration
|
| 112 |
+
>>> configuration = model.config
|
| 113 |
+
```"""
|
| 114 |
+
|
| 115 |
+
model_type = "quasar"
|
| 116 |
+
keys_to_ignore_at_inference = ["past_key_values"]
|
| 117 |
+
|
| 118 |
+
def __init__(
|
| 119 |
+
self,
|
| 120 |
+
vocab_size=51200,
|
| 121 |
+
hidden_size=2560,
|
| 122 |
+
intermediate_size=8192,
|
| 123 |
+
num_hidden_layers=24,
|
| 124 |
+
num_attention_heads=32,
|
| 125 |
+
num_key_value_heads=None,
|
| 126 |
+
resid_pdrop=0.0,
|
| 127 |
+
embd_pdrop=0.0,
|
| 128 |
+
attention_dropout=0.0,
|
| 129 |
+
hidden_act="gelu_new",
|
| 130 |
+
max_position_embeddings=32768,
|
| 131 |
+
initializer_range=0.02,
|
| 132 |
+
layer_norm_eps=1e-5,
|
| 133 |
+
use_cache=True,
|
| 134 |
+
tie_word_embeddings=False,
|
| 135 |
+
rope_theta=10000.0,
|
| 136 |
+
rope_scaling=None,
|
| 137 |
+
partial_rotary_factor=0.5,
|
| 138 |
+
qk_layernorm=False,
|
| 139 |
+
bos_token_id=1,
|
| 140 |
+
eos_token_id=2,
|
| 141 |
+
## Aggiunto
|
| 142 |
+
#duplicate_trick_v2=True,
|
| 143 |
+
#duplicate_rank=8,
|
| 144 |
+
#duplicate_dropout=0.0,
|
| 145 |
+
sliding_window=4096,
|
| 146 |
+
simple_norm=False,
|
| 147 |
+
remove_ff_bias=True,
|
| 148 |
+
gated_activation=False,
|
| 149 |
+
duplicate_trick=True,
|
| 150 |
+
duplicate_grad=True,
|
| 151 |
+
layer_ranges=[[0, 16],[8, 21],[12, 25],[16, 29],[25, 32]],
|
| 152 |
+
**kwargs,
|
| 153 |
+
):
|
| 154 |
+
##Aggiunti
|
| 155 |
+
self.sliding_window = sliding_window
|
| 156 |
+
self.simple_norm = simple_norm
|
| 157 |
+
self.remove_ff_bias = remove_ff_bias
|
| 158 |
+
self.gated_activation = gated_activation
|
| 159 |
+
self.duplicate_trick = duplicate_trick
|
| 160 |
+
self.duplicate_grad = duplicate_grad
|
| 161 |
+
self.layer_ranges = layer_ranges if layer_ranges is not None else []
|
| 162 |
+
|
| 163 |
+
##V2###
|
| 164 |
+
#self.duplicate_trick_v2 = duplicate_trick_v2
|
| 165 |
+
#self.layer_ranges_duplicate_v2 = []
|
| 166 |
+
#self._assing_layer_ranges_duplicate_v2()
|
| 167 |
+
#self.duplicate_rank = duplicate_rank
|
| 168 |
+
#self.duplicate_dropout = duplicate_dropout
|
| 169 |
+
#self._duplicate_trick_v2_validation()
|
| 170 |
+
####
|
| 171 |
+
|
| 172 |
+
self.vocab_size = vocab_size
|
| 173 |
+
self.hidden_size = hidden_size
|
| 174 |
+
self.intermediate_size = intermediate_size
|
| 175 |
+
self.num_hidden_layers = num_hidden_layers
|
| 176 |
+
self.num_attention_heads = num_attention_heads
|
| 177 |
+
|
| 178 |
+
if num_key_value_heads is None:
|
| 179 |
+
num_key_value_heads = num_attention_heads
|
| 180 |
+
|
| 181 |
+
self.num_key_value_heads = num_key_value_heads
|
| 182 |
+
self.resid_pdrop = resid_pdrop
|
| 183 |
+
self.embd_pdrop = embd_pdrop
|
| 184 |
+
self.attention_dropout = attention_dropout
|
| 185 |
+
self.hidden_act = hidden_act
|
| 186 |
+
self.max_position_embeddings = max_position_embeddings
|
| 187 |
+
self.initializer_range = initializer_range
|
| 188 |
+
self.layer_norm_eps = layer_norm_eps
|
| 189 |
+
self.use_cache = use_cache
|
| 190 |
+
self.rope_theta = rope_theta
|
| 191 |
+
self.rope_scaling = rope_scaling
|
| 192 |
+
self.partial_rotary_factor = partial_rotary_factor
|
| 193 |
+
self.qk_layernorm = qk_layernorm
|
| 194 |
+
self._rope_scaling_validation()
|
| 195 |
+
|
| 196 |
+
super().__init__(
|
| 197 |
+
bos_token_id=bos_token_id,
|
| 198 |
+
eos_token_id=eos_token_id,
|
| 199 |
+
tie_word_embeddings=tie_word_embeddings,
|
| 200 |
+
**kwargs,
|
| 201 |
+
)
|
| 202 |
+
|
| 203 |
+
def _assing_layer_ranges_duplicate_v2(self):
|
| 204 |
+
# Calcolo gli offset iniziali per ciascun intervallo nella lista unica
|
| 205 |
+
offsets = [0]
|
| 206 |
+
for i in range(1, len(self.layer_ranges)):
|
| 207 |
+
offset = offsets[-1] + self.layer_ranges[i - 1][1] - self.layer_ranges[i - 1][0]
|
| 208 |
+
offsets.append(offset)
|
| 209 |
+
|
| 210 |
+
# Seleziono solo gli intervalli dispari e calcolo le loro posizioni assolute
|
| 211 |
+
odd_intervals_positions = []
|
| 212 |
+
for i in range(1, len(self.layer_ranges), 2):
|
| 213 |
+
start, end = self.layer_ranges[i]
|
| 214 |
+
for n in range(start, end):
|
| 215 |
+
position = offsets[i] + (n - start)
|
| 216 |
+
odd_intervals_positions.append(position)
|
| 217 |
+
|
| 218 |
+
self.layer_ranges_duplicate_v2 = list(set(odd_intervals_positions))
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
def _duplicate_trick_v2_validation(self):
|
| 222 |
+
if self.duplicate_trick_v2 and self.duplicate_trick:
|
| 223 |
+
# warn just one time that only one of the two flags will be used
|
| 224 |
+
logger.warning(
|
| 225 |
+
"Both `duplicate_trick` and `duplicate_trick_v2` are set to True. Only `duplicate_trick_v2` will be used."
|
| 226 |
+
)
|
| 227 |
+
if self.duplicate_trick_v2 and self.duplicate_rank < 1:
|
| 228 |
+
raise ValueError("`duplicate_rank` must be a positive integer")
|
| 229 |
+
if self.duplicate_trick_v2 and not self.layer_ranges:
|
| 230 |
+
raise ValueError("`layer_ranges` must be set when `duplicate_trick_v2` is True")
|
| 231 |
+
|
| 232 |
+
|
| 233 |
+
# Copied from transformers.models.llama.configuration_llama.LlamaConfig._rope_scaling_validation
|
| 234 |
+
def _rope_scaling_validation(self):
|
| 235 |
+
"""
|
| 236 |
+
Validate the `rope_scaling` configuration.
|
| 237 |
+
"""
|
| 238 |
+
if self.rope_scaling is None:
|
| 239 |
+
return
|
| 240 |
+
|
| 241 |
+
if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
|
| 242 |
+
raise ValueError(
|
| 243 |
+
"`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
|
| 244 |
+
f"got {self.rope_scaling}"
|
| 245 |
+
)
|
| 246 |
+
rope_scaling_type = self.rope_scaling.get("type", None)
|
| 247 |
+
rope_scaling_factor = self.rope_scaling.get("factor", None)
|
| 248 |
+
if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
|
| 249 |
+
raise ValueError(
|
| 250 |
+
f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
|
| 251 |
+
)
|
| 252 |
+
if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor <= 1.0:
|
| 253 |
+
raise ValueError(f"`rope_scaling`'s factor field must be a float > 1, got {rope_scaling_factor}")
|
generation_config.json
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_from_model_config": true,
|
| 3 |
+
"bos_token_id": 50256,
|
| 4 |
+
"eos_token_id": 50256,
|
| 5 |
+
"transformers_version": "4.38.0.dev0"
|
| 6 |
+
}
|
merges.txt
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
model-00001-of-00004.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e97b156a9bb5de39ecb30c1c69d682e28e56450abf91045ba615a83333c8f717
|
| 3 |
+
size 4981635896
|
model-00002-of-00004.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3ecb2fdf93fa1838bd1721086f2ce688f20c6de33fe9987e87ef400f40b95d7b
|
| 3 |
+
size 4981719560
|
model-00003-of-00004.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8939a34c5d15c04e7d796131f45effb4005a6a27efdd284ab62c177e5879b167
|
| 3 |
+
size 4929290880
|
model-00004-of-00004.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d02e41e5c1b8f16477a8050f2692e552486210ee0482f0891758aafa74352f2c
|
| 3 |
+
size 4090383480
|
model.safetensors.index.json
ADDED
|
@@ -0,0 +1,696 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"metadata": {
|
| 3 |
+
"total_size": 18982952960
|
| 4 |
+
},
|
| 5 |
+
"weight_map": {
|
| 6 |
+
"lm_head.bias": "model-00004-of-00004.safetensors",
|
| 7 |
+
"lm_head.weight": "model-00004-of-00004.safetensors",
|
| 8 |
+
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
| 9 |
+
"model.final_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 10 |
+
"model.final_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 11 |
+
"model.layers.0.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 12 |
+
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 13 |
+
"model.layers.0.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 14 |
+
"model.layers.0.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 15 |
+
"model.layers.0.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 16 |
+
"model.layers.0.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 17 |
+
"model.layers.0.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 18 |
+
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 19 |
+
"model.layers.0.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 20 |
+
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 21 |
+
"model.layers.0.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 22 |
+
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 23 |
+
"model.layers.1.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 24 |
+
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 25 |
+
"model.layers.1.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 26 |
+
"model.layers.1.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 27 |
+
"model.layers.1.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 28 |
+
"model.layers.1.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 29 |
+
"model.layers.1.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 30 |
+
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 31 |
+
"model.layers.1.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 32 |
+
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 33 |
+
"model.layers.1.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 34 |
+
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 35 |
+
"model.layers.10.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 36 |
+
"model.layers.10.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 37 |
+
"model.layers.10.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 38 |
+
"model.layers.10.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 39 |
+
"model.layers.10.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 40 |
+
"model.layers.10.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 41 |
+
"model.layers.10.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 42 |
+
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 43 |
+
"model.layers.10.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 44 |
+
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 45 |
+
"model.layers.10.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 46 |
+
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 47 |
+
"model.layers.11.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 48 |
+
"model.layers.11.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 49 |
+
"model.layers.11.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 50 |
+
"model.layers.11.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 51 |
+
"model.layers.11.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 52 |
+
"model.layers.11.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 53 |
+
"model.layers.11.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 54 |
+
"model.layers.11.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 55 |
+
"model.layers.11.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 56 |
+
"model.layers.11.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 57 |
+
"model.layers.11.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 58 |
+
"model.layers.11.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 59 |
+
"model.layers.12.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 60 |
+
"model.layers.12.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 61 |
+
"model.layers.12.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 62 |
+
"model.layers.12.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 63 |
+
"model.layers.12.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 64 |
+
"model.layers.12.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 65 |
+
"model.layers.12.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 66 |
+
"model.layers.12.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 67 |
+
"model.layers.12.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 68 |
+
"model.layers.12.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 69 |
+
"model.layers.12.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 70 |
+
"model.layers.12.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 71 |
+
"model.layers.13.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 72 |
+
"model.layers.13.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 73 |
+
"model.layers.13.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 74 |
+
"model.layers.13.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 75 |
+
"model.layers.13.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 76 |
+
"model.layers.13.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 77 |
+
"model.layers.13.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 78 |
+
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 79 |
+
"model.layers.13.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 80 |
+
"model.layers.13.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 81 |
+
"model.layers.13.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 82 |
+
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 83 |
+
"model.layers.14.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 84 |
+
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 85 |
+
"model.layers.14.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 86 |
+
"model.layers.14.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 87 |
+
"model.layers.14.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 88 |
+
"model.layers.14.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 89 |
+
"model.layers.14.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 90 |
+
"model.layers.14.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 91 |
+
"model.layers.14.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 92 |
+
"model.layers.14.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 93 |
+
"model.layers.14.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 94 |
+
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 95 |
+
"model.layers.15.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 96 |
+
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 97 |
+
"model.layers.15.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 98 |
+
"model.layers.15.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 99 |
+
"model.layers.15.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 100 |
+
"model.layers.15.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 101 |
+
"model.layers.15.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 102 |
+
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 103 |
+
"model.layers.15.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 104 |
+
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 105 |
+
"model.layers.15.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 106 |
+
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 107 |
+
"model.layers.16.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 108 |
+
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 109 |
+
"model.layers.16.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 110 |
+
"model.layers.16.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 111 |
+
"model.layers.16.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 112 |
+
"model.layers.16.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 113 |
+
"model.layers.16.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 114 |
+
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 115 |
+
"model.layers.16.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 116 |
+
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 117 |
+
"model.layers.16.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 118 |
+
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 119 |
+
"model.layers.17.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 120 |
+
"model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 121 |
+
"model.layers.17.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 122 |
+
"model.layers.17.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 123 |
+
"model.layers.17.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 124 |
+
"model.layers.17.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 125 |
+
"model.layers.17.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 126 |
+
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 127 |
+
"model.layers.17.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 128 |
+
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 129 |
+
"model.layers.17.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 130 |
+
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 131 |
+
"model.layers.18.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 132 |
+
"model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 133 |
+
"model.layers.18.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 134 |
+
"model.layers.18.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 135 |
+
"model.layers.18.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 136 |
+
"model.layers.18.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 137 |
+
"model.layers.18.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 138 |
+
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 139 |
+
"model.layers.18.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 140 |
+
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 141 |
+
"model.layers.18.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 142 |
+
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 143 |
+
"model.layers.19.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 144 |
+
"model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 145 |
+
"model.layers.19.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 146 |
+
"model.layers.19.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 147 |
+
"model.layers.19.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 148 |
+
"model.layers.19.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 149 |
+
"model.layers.19.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 150 |
+
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 151 |
+
"model.layers.19.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 152 |
+
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 153 |
+
"model.layers.19.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 154 |
+
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 155 |
+
"model.layers.2.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 156 |
+
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 157 |
+
"model.layers.2.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 158 |
+
"model.layers.2.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 159 |
+
"model.layers.2.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 160 |
+
"model.layers.2.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 161 |
+
"model.layers.2.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 162 |
+
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 163 |
+
"model.layers.2.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 164 |
+
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 165 |
+
"model.layers.2.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 166 |
+
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 167 |
+
"model.layers.20.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 168 |
+
"model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 169 |
+
"model.layers.20.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 170 |
+
"model.layers.20.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 171 |
+
"model.layers.20.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 172 |
+
"model.layers.20.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 173 |
+
"model.layers.20.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 174 |
+
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 175 |
+
"model.layers.20.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 176 |
+
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 177 |
+
"model.layers.20.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 178 |
+
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 179 |
+
"model.layers.21.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 180 |
+
"model.layers.21.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 181 |
+
"model.layers.21.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 182 |
+
"model.layers.21.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 183 |
+
"model.layers.21.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 184 |
+
"model.layers.21.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 185 |
+
"model.layers.21.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 186 |
+
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 187 |
+
"model.layers.21.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 188 |
+
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 189 |
+
"model.layers.21.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 190 |
+
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 191 |
+
"model.layers.22.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 192 |
+
"model.layers.22.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 193 |
+
"model.layers.22.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 194 |
+
"model.layers.22.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 195 |
+
"model.layers.22.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 196 |
+
"model.layers.22.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 197 |
+
"model.layers.22.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 198 |
+
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 199 |
+
"model.layers.22.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 200 |
+
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 201 |
+
"model.layers.22.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 202 |
+
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 203 |
+
"model.layers.23.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 204 |
+
"model.layers.23.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 205 |
+
"model.layers.23.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 206 |
+
"model.layers.23.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 207 |
+
"model.layers.23.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 208 |
+
"model.layers.23.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 209 |
+
"model.layers.23.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 210 |
+
"model.layers.23.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 211 |
+
"model.layers.23.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 212 |
+
"model.layers.23.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 213 |
+
"model.layers.23.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 214 |
+
"model.layers.23.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 215 |
+
"model.layers.24.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 216 |
+
"model.layers.24.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 217 |
+
"model.layers.24.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 218 |
+
"model.layers.24.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 219 |
+
"model.layers.24.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 220 |
+
"model.layers.24.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 221 |
+
"model.layers.24.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 222 |
+
"model.layers.24.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 223 |
+
"model.layers.24.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 224 |
+
"model.layers.24.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 225 |
+
"model.layers.24.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 226 |
+
"model.layers.24.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 227 |
+
"model.layers.25.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 228 |
+
"model.layers.25.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 229 |
+
"model.layers.25.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 230 |
+
"model.layers.25.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 231 |
+
"model.layers.25.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 232 |
+
"model.layers.25.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 233 |
+
"model.layers.25.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 234 |
+
"model.layers.25.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 235 |
+
"model.layers.25.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 236 |
+
"model.layers.25.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 237 |
+
"model.layers.25.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 238 |
+
"model.layers.25.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 239 |
+
"model.layers.26.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 240 |
+
"model.layers.26.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 241 |
+
"model.layers.26.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 242 |
+
"model.layers.26.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 243 |
+
"model.layers.26.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 244 |
+
"model.layers.26.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 245 |
+
"model.layers.26.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 246 |
+
"model.layers.26.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 247 |
+
"model.layers.26.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 248 |
+
"model.layers.26.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 249 |
+
"model.layers.26.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 250 |
+
"model.layers.26.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 251 |
+
"model.layers.27.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 252 |
+
"model.layers.27.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 253 |
+
"model.layers.27.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 254 |
+
"model.layers.27.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 255 |
+
"model.layers.27.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 256 |
+
"model.layers.27.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 257 |
+
"model.layers.27.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 258 |
+
"model.layers.27.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 259 |
+
"model.layers.27.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 260 |
+
"model.layers.27.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 261 |
+
"model.layers.27.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 262 |
+
"model.layers.27.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 263 |
+
"model.layers.28.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 264 |
+
"model.layers.28.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 265 |
+
"model.layers.28.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 266 |
+
"model.layers.28.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 267 |
+
"model.layers.28.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 268 |
+
"model.layers.28.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 269 |
+
"model.layers.28.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 270 |
+
"model.layers.28.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 271 |
+
"model.layers.28.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 272 |
+
"model.layers.28.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 273 |
+
"model.layers.28.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 274 |
+
"model.layers.28.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 275 |
+
"model.layers.29.input_layernorm.bias": "model-00002-of-00004.safetensors",
|
| 276 |
+
"model.layers.29.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 277 |
+
"model.layers.29.mlp.fc1.weight": "model-00002-of-00004.safetensors",
|
| 278 |
+
"model.layers.29.mlp.fc2.weight": "model-00002-of-00004.safetensors",
|
| 279 |
+
"model.layers.29.self_attn.dense.bias": "model-00002-of-00004.safetensors",
|
| 280 |
+
"model.layers.29.self_attn.dense.weight": "model-00002-of-00004.safetensors",
|
| 281 |
+
"model.layers.29.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
|
| 282 |
+
"model.layers.29.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 283 |
+
"model.layers.29.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
|
| 284 |
+
"model.layers.29.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 285 |
+
"model.layers.29.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
|
| 286 |
+
"model.layers.29.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 287 |
+
"model.layers.3.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 288 |
+
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 289 |
+
"model.layers.3.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 290 |
+
"model.layers.3.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 291 |
+
"model.layers.3.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 292 |
+
"model.layers.3.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 293 |
+
"model.layers.3.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 294 |
+
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 295 |
+
"model.layers.3.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 296 |
+
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 297 |
+
"model.layers.3.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 298 |
+
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 299 |
+
"model.layers.30.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 300 |
+
"model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 301 |
+
"model.layers.30.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 302 |
+
"model.layers.30.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 303 |
+
"model.layers.30.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 304 |
+
"model.layers.30.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 305 |
+
"model.layers.30.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 306 |
+
"model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 307 |
+
"model.layers.30.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 308 |
+
"model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 309 |
+
"model.layers.30.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 310 |
+
"model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 311 |
+
"model.layers.31.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 312 |
+
"model.layers.31.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 313 |
+
"model.layers.31.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 314 |
+
"model.layers.31.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 315 |
+
"model.layers.31.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 316 |
+
"model.layers.31.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 317 |
+
"model.layers.31.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 318 |
+
"model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 319 |
+
"model.layers.31.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 320 |
+
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 321 |
+
"model.layers.31.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 322 |
+
"model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 323 |
+
"model.layers.32.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 324 |
+
"model.layers.32.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 325 |
+
"model.layers.32.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 326 |
+
"model.layers.32.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 327 |
+
"model.layers.32.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 328 |
+
"model.layers.32.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 329 |
+
"model.layers.32.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 330 |
+
"model.layers.32.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 331 |
+
"model.layers.32.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 332 |
+
"model.layers.32.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 333 |
+
"model.layers.32.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 334 |
+
"model.layers.32.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 335 |
+
"model.layers.33.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 336 |
+
"model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 337 |
+
"model.layers.33.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 338 |
+
"model.layers.33.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 339 |
+
"model.layers.33.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 340 |
+
"model.layers.33.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 341 |
+
"model.layers.33.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 342 |
+
"model.layers.33.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 343 |
+
"model.layers.33.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 344 |
+
"model.layers.33.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 345 |
+
"model.layers.33.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 346 |
+
"model.layers.33.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 347 |
+
"model.layers.34.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 348 |
+
"model.layers.34.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 349 |
+
"model.layers.34.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 350 |
+
"model.layers.34.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 351 |
+
"model.layers.34.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 352 |
+
"model.layers.34.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 353 |
+
"model.layers.34.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 354 |
+
"model.layers.34.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 355 |
+
"model.layers.34.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 356 |
+
"model.layers.34.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 357 |
+
"model.layers.34.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 358 |
+
"model.layers.34.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 359 |
+
"model.layers.35.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 360 |
+
"model.layers.35.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 361 |
+
"model.layers.35.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 362 |
+
"model.layers.35.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 363 |
+
"model.layers.35.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 364 |
+
"model.layers.35.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 365 |
+
"model.layers.35.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 366 |
+
"model.layers.35.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 367 |
+
"model.layers.35.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 368 |
+
"model.layers.35.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 369 |
+
"model.layers.35.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 370 |
+
"model.layers.35.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 371 |
+
"model.layers.36.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 372 |
+
"model.layers.36.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 373 |
+
"model.layers.36.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 374 |
+
"model.layers.36.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 375 |
+
"model.layers.36.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 376 |
+
"model.layers.36.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 377 |
+
"model.layers.36.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 378 |
+
"model.layers.36.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 379 |
+
"model.layers.36.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 380 |
+
"model.layers.36.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 381 |
+
"model.layers.36.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 382 |
+
"model.layers.36.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 383 |
+
"model.layers.37.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 384 |
+
"model.layers.37.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 385 |
+
"model.layers.37.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 386 |
+
"model.layers.37.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 387 |
+
"model.layers.37.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 388 |
+
"model.layers.37.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 389 |
+
"model.layers.37.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 390 |
+
"model.layers.37.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 391 |
+
"model.layers.37.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 392 |
+
"model.layers.37.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 393 |
+
"model.layers.37.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 394 |
+
"model.layers.37.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 395 |
+
"model.layers.38.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 396 |
+
"model.layers.38.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 397 |
+
"model.layers.38.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 398 |
+
"model.layers.38.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 399 |
+
"model.layers.38.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 400 |
+
"model.layers.38.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 401 |
+
"model.layers.38.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 402 |
+
"model.layers.38.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 403 |
+
"model.layers.38.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 404 |
+
"model.layers.38.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 405 |
+
"model.layers.38.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 406 |
+
"model.layers.38.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 407 |
+
"model.layers.39.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 408 |
+
"model.layers.39.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 409 |
+
"model.layers.39.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 410 |
+
"model.layers.39.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 411 |
+
"model.layers.39.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 412 |
+
"model.layers.39.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 413 |
+
"model.layers.39.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 414 |
+
"model.layers.39.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 415 |
+
"model.layers.39.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 416 |
+
"model.layers.39.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 417 |
+
"model.layers.39.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 418 |
+
"model.layers.39.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 419 |
+
"model.layers.4.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 420 |
+
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 421 |
+
"model.layers.4.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 422 |
+
"model.layers.4.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 423 |
+
"model.layers.4.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 424 |
+
"model.layers.4.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 425 |
+
"model.layers.4.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 426 |
+
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 427 |
+
"model.layers.4.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 428 |
+
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 429 |
+
"model.layers.4.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 430 |
+
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 431 |
+
"model.layers.40.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 432 |
+
"model.layers.40.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 433 |
+
"model.layers.40.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 434 |
+
"model.layers.40.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 435 |
+
"model.layers.40.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 436 |
+
"model.layers.40.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 437 |
+
"model.layers.40.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 438 |
+
"model.layers.40.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 439 |
+
"model.layers.40.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 440 |
+
"model.layers.40.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 441 |
+
"model.layers.40.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 442 |
+
"model.layers.40.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 443 |
+
"model.layers.41.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 444 |
+
"model.layers.41.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 445 |
+
"model.layers.41.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 446 |
+
"model.layers.41.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 447 |
+
"model.layers.41.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 448 |
+
"model.layers.41.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 449 |
+
"model.layers.41.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 450 |
+
"model.layers.41.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 451 |
+
"model.layers.41.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 452 |
+
"model.layers.41.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 453 |
+
"model.layers.41.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 454 |
+
"model.layers.41.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 455 |
+
"model.layers.42.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 456 |
+
"model.layers.42.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 457 |
+
"model.layers.42.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 458 |
+
"model.layers.42.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 459 |
+
"model.layers.42.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 460 |
+
"model.layers.42.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 461 |
+
"model.layers.42.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 462 |
+
"model.layers.42.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 463 |
+
"model.layers.42.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 464 |
+
"model.layers.42.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 465 |
+
"model.layers.42.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 466 |
+
"model.layers.42.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 467 |
+
"model.layers.43.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 468 |
+
"model.layers.43.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 469 |
+
"model.layers.43.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 470 |
+
"model.layers.43.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 471 |
+
"model.layers.43.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 472 |
+
"model.layers.43.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 473 |
+
"model.layers.43.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 474 |
+
"model.layers.43.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 475 |
+
"model.layers.43.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 476 |
+
"model.layers.43.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 477 |
+
"model.layers.43.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 478 |
+
"model.layers.43.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 479 |
+
"model.layers.44.input_layernorm.bias": "model-00003-of-00004.safetensors",
|
| 480 |
+
"model.layers.44.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 481 |
+
"model.layers.44.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 482 |
+
"model.layers.44.mlp.fc2.weight": "model-00003-of-00004.safetensors",
|
| 483 |
+
"model.layers.44.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 484 |
+
"model.layers.44.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 485 |
+
"model.layers.44.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 486 |
+
"model.layers.44.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 487 |
+
"model.layers.44.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 488 |
+
"model.layers.44.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 489 |
+
"model.layers.44.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 490 |
+
"model.layers.44.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 491 |
+
"model.layers.45.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 492 |
+
"model.layers.45.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 493 |
+
"model.layers.45.mlp.fc1.weight": "model-00003-of-00004.safetensors",
|
| 494 |
+
"model.layers.45.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 495 |
+
"model.layers.45.self_attn.dense.bias": "model-00003-of-00004.safetensors",
|
| 496 |
+
"model.layers.45.self_attn.dense.weight": "model-00003-of-00004.safetensors",
|
| 497 |
+
"model.layers.45.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
|
| 498 |
+
"model.layers.45.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 499 |
+
"model.layers.45.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
|
| 500 |
+
"model.layers.45.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 501 |
+
"model.layers.45.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
|
| 502 |
+
"model.layers.45.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 503 |
+
"model.layers.46.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 504 |
+
"model.layers.46.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 505 |
+
"model.layers.46.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 506 |
+
"model.layers.46.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 507 |
+
"model.layers.46.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 508 |
+
"model.layers.46.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 509 |
+
"model.layers.46.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 510 |
+
"model.layers.46.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 511 |
+
"model.layers.46.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 512 |
+
"model.layers.46.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 513 |
+
"model.layers.46.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 514 |
+
"model.layers.46.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 515 |
+
"model.layers.47.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 516 |
+
"model.layers.47.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 517 |
+
"model.layers.47.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 518 |
+
"model.layers.47.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 519 |
+
"model.layers.47.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 520 |
+
"model.layers.47.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 521 |
+
"model.layers.47.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 522 |
+
"model.layers.47.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 523 |
+
"model.layers.47.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 524 |
+
"model.layers.47.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 525 |
+
"model.layers.47.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 526 |
+
"model.layers.47.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 527 |
+
"model.layers.48.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 528 |
+
"model.layers.48.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 529 |
+
"model.layers.48.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 530 |
+
"model.layers.48.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 531 |
+
"model.layers.48.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 532 |
+
"model.layers.48.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 533 |
+
"model.layers.48.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 534 |
+
"model.layers.48.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 535 |
+
"model.layers.48.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 536 |
+
"model.layers.48.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 537 |
+
"model.layers.48.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 538 |
+
"model.layers.48.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 539 |
+
"model.layers.49.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 540 |
+
"model.layers.49.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 541 |
+
"model.layers.49.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 542 |
+
"model.layers.49.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 543 |
+
"model.layers.49.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 544 |
+
"model.layers.49.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 545 |
+
"model.layers.49.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 546 |
+
"model.layers.49.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 547 |
+
"model.layers.49.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 548 |
+
"model.layers.49.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 549 |
+
"model.layers.49.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 550 |
+
"model.layers.49.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 551 |
+
"model.layers.5.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 552 |
+
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 553 |
+
"model.layers.5.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 554 |
+
"model.layers.5.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 555 |
+
"model.layers.5.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 556 |
+
"model.layers.5.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 557 |
+
"model.layers.5.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 558 |
+
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 559 |
+
"model.layers.5.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 560 |
+
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 561 |
+
"model.layers.5.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 562 |
+
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 563 |
+
"model.layers.50.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 564 |
+
"model.layers.50.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 565 |
+
"model.layers.50.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 566 |
+
"model.layers.50.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 567 |
+
"model.layers.50.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 568 |
+
"model.layers.50.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 569 |
+
"model.layers.50.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 570 |
+
"model.layers.50.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 571 |
+
"model.layers.50.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 572 |
+
"model.layers.50.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 573 |
+
"model.layers.50.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 574 |
+
"model.layers.50.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 575 |
+
"model.layers.51.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 576 |
+
"model.layers.51.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 577 |
+
"model.layers.51.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 578 |
+
"model.layers.51.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 579 |
+
"model.layers.51.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 580 |
+
"model.layers.51.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 581 |
+
"model.layers.51.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 582 |
+
"model.layers.51.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 583 |
+
"model.layers.51.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 584 |
+
"model.layers.51.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 585 |
+
"model.layers.51.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 586 |
+
"model.layers.51.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 587 |
+
"model.layers.52.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 588 |
+
"model.layers.52.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 589 |
+
"model.layers.52.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 590 |
+
"model.layers.52.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 591 |
+
"model.layers.52.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 592 |
+
"model.layers.52.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 593 |
+
"model.layers.52.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 594 |
+
"model.layers.52.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 595 |
+
"model.layers.52.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 596 |
+
"model.layers.52.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 597 |
+
"model.layers.52.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 598 |
+
"model.layers.52.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 599 |
+
"model.layers.53.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 600 |
+
"model.layers.53.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 601 |
+
"model.layers.53.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 602 |
+
"model.layers.53.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 603 |
+
"model.layers.53.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 604 |
+
"model.layers.53.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 605 |
+
"model.layers.53.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 606 |
+
"model.layers.53.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 607 |
+
"model.layers.53.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 608 |
+
"model.layers.53.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 609 |
+
"model.layers.53.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 610 |
+
"model.layers.53.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 611 |
+
"model.layers.54.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 612 |
+
"model.layers.54.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 613 |
+
"model.layers.54.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 614 |
+
"model.layers.54.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 615 |
+
"model.layers.54.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 616 |
+
"model.layers.54.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 617 |
+
"model.layers.54.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 618 |
+
"model.layers.54.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 619 |
+
"model.layers.54.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 620 |
+
"model.layers.54.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 621 |
+
"model.layers.54.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 622 |
+
"model.layers.54.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 623 |
+
"model.layers.55.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 624 |
+
"model.layers.55.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 625 |
+
"model.layers.55.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 626 |
+
"model.layers.55.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 627 |
+
"model.layers.55.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 628 |
+
"model.layers.55.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 629 |
+
"model.layers.55.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 630 |
+
"model.layers.55.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 631 |
+
"model.layers.55.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 632 |
+
"model.layers.55.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 633 |
+
"model.layers.55.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 634 |
+
"model.layers.55.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 635 |
+
"model.layers.56.input_layernorm.bias": "model-00004-of-00004.safetensors",
|
| 636 |
+
"model.layers.56.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 637 |
+
"model.layers.56.mlp.fc1.weight": "model-00004-of-00004.safetensors",
|
| 638 |
+
"model.layers.56.mlp.fc2.weight": "model-00004-of-00004.safetensors",
|
| 639 |
+
"model.layers.56.self_attn.dense.bias": "model-00004-of-00004.safetensors",
|
| 640 |
+
"model.layers.56.self_attn.dense.weight": "model-00004-of-00004.safetensors",
|
| 641 |
+
"model.layers.56.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
|
| 642 |
+
"model.layers.56.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 643 |
+
"model.layers.56.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
|
| 644 |
+
"model.layers.56.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 645 |
+
"model.layers.56.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
|
| 646 |
+
"model.layers.56.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 647 |
+
"model.layers.6.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 648 |
+
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 649 |
+
"model.layers.6.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 650 |
+
"model.layers.6.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 651 |
+
"model.layers.6.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 652 |
+
"model.layers.6.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 653 |
+
"model.layers.6.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 654 |
+
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 655 |
+
"model.layers.6.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 656 |
+
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 657 |
+
"model.layers.6.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 658 |
+
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 659 |
+
"model.layers.7.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 660 |
+
"model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 661 |
+
"model.layers.7.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 662 |
+
"model.layers.7.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 663 |
+
"model.layers.7.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 664 |
+
"model.layers.7.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 665 |
+
"model.layers.7.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 666 |
+
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 667 |
+
"model.layers.7.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 668 |
+
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 669 |
+
"model.layers.7.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 670 |
+
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 671 |
+
"model.layers.8.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 672 |
+
"model.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 673 |
+
"model.layers.8.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 674 |
+
"model.layers.8.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 675 |
+
"model.layers.8.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 676 |
+
"model.layers.8.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 677 |
+
"model.layers.8.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 678 |
+
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 679 |
+
"model.layers.8.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 680 |
+
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 681 |
+
"model.layers.8.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 682 |
+
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 683 |
+
"model.layers.9.input_layernorm.bias": "model-00001-of-00004.safetensors",
|
| 684 |
+
"model.layers.9.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 685 |
+
"model.layers.9.mlp.fc1.weight": "model-00001-of-00004.safetensors",
|
| 686 |
+
"model.layers.9.mlp.fc2.weight": "model-00001-of-00004.safetensors",
|
| 687 |
+
"model.layers.9.self_attn.dense.bias": "model-00001-of-00004.safetensors",
|
| 688 |
+
"model.layers.9.self_attn.dense.weight": "model-00001-of-00004.safetensors",
|
| 689 |
+
"model.layers.9.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
|
| 690 |
+
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 691 |
+
"model.layers.9.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
|
| 692 |
+
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 693 |
+
"model.layers.9.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
|
| 694 |
+
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00004.safetensors"
|
| 695 |
+
}
|
| 696 |
+
}
|
modeling_quasar.py
ADDED
|
@@ -0,0 +1,1494 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# coding=utf-8
|
| 2 |
+
# Copyright 2023 Microsoft and the HuggingFace Inc. team. All rights reserved.
|
| 3 |
+
#
|
| 4 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 5 |
+
# you may not use this file except in compliance with the License.
|
| 6 |
+
# You may obtain a copy of the License at
|
| 7 |
+
#
|
| 8 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 9 |
+
#
|
| 10 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 11 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 12 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 13 |
+
# See the License for the specific language governing permissions and
|
| 14 |
+
# limitations under the License.
|
| 15 |
+
|
| 16 |
+
""" PyTorch Quasar model."""
|
| 17 |
+
import inspect
|
| 18 |
+
import math
|
| 19 |
+
from typing import List, Optional, Tuple, Union, Callable
|
| 20 |
+
|
| 21 |
+
import torch
|
| 22 |
+
import torch.nn.functional as F
|
| 23 |
+
import torch.utils.checkpoint
|
| 24 |
+
from torch import nn
|
| 25 |
+
from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
|
| 26 |
+
|
| 27 |
+
from transformers.activations import ACT2FN
|
| 28 |
+
from transformers.cache_utils import Cache, DynamicCache
|
| 29 |
+
from transformers.modeling_attn_mask_utils import _prepare_4d_causal_attention_mask
|
| 30 |
+
from transformers.modeling_outputs import (
|
| 31 |
+
BaseModelOutputWithPast,
|
| 32 |
+
CausalLMOutputWithPast,
|
| 33 |
+
SequenceClassifierOutputWithPast,
|
| 34 |
+
TokenClassifierOutput,
|
| 35 |
+
)
|
| 36 |
+
from transformers.modeling_utils import PreTrainedModel
|
| 37 |
+
from transformers.utils import (
|
| 38 |
+
add_code_sample_docstrings,
|
| 39 |
+
add_start_docstrings,
|
| 40 |
+
add_start_docstrings_to_model_forward,
|
| 41 |
+
is_flash_attn_2_available,
|
| 42 |
+
is_flash_attn_greater_or_equal_2_10,
|
| 43 |
+
logging,
|
| 44 |
+
replace_return_docstrings,
|
| 45 |
+
)
|
| 46 |
+
from .configuration_quasar import QuasarConfig
|
| 47 |
+
|
| 48 |
+
try:
|
| 49 |
+
from flash_attn import flash_attn_func, flash_attn_varlen_func
|
| 50 |
+
from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input # noqa
|
| 51 |
+
|
| 52 |
+
_flash_supports_window_size = "window_size" in list(inspect.signature(flash_attn_func).parameters)
|
| 53 |
+
except:
|
| 54 |
+
pass
|
| 55 |
+
|
| 56 |
+
logger = logging.get_logger(__name__)
|
| 57 |
+
|
| 58 |
+
_CHECKPOINT_FOR_DOC = "AstraMindAI/AstraQuasar-4.5B"
|
| 59 |
+
_CONFIG_FOR_DOC = "QuasarConfig"
|
| 60 |
+
|
| 61 |
+
QUASAR_PRETRAINED_MODEL_ARCHIVE_LIST = [
|
| 62 |
+
"AstraMindAI/AstraQuasar-4.5B",
|
| 63 |
+
]
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
# Copied from transformers.models.llama.modeling_llama._get_unpad_data
|
| 70 |
+
def _get_unpad_data(attention_mask):
|
| 71 |
+
seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32)
|
| 72 |
+
indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
|
| 73 |
+
max_seqlen_in_batch = seqlens_in_batch.max().item()
|
| 74 |
+
cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.torch.int32), (1, 0))
|
| 75 |
+
return (
|
| 76 |
+
indices,
|
| 77 |
+
cu_seqlens,
|
| 78 |
+
max_seqlen_in_batch,
|
| 79 |
+
)
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaRotaryEmbedding with Llama->Quasar
|
| 83 |
+
class QuasarRotaryEmbedding(nn.Module):
|
| 84 |
+
def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None):
|
| 85 |
+
super().__init__()
|
| 86 |
+
|
| 87 |
+
self.dim = dim
|
| 88 |
+
self.max_position_embeddings = max_position_embeddings
|
| 89 |
+
self.base = base
|
| 90 |
+
inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
|
| 91 |
+
self.register_buffer("inv_freq", inv_freq, persistent=False)
|
| 92 |
+
|
| 93 |
+
# Build here to make `torch.jit.trace` work.
|
| 94 |
+
self._set_cos_sin_cache(
|
| 95 |
+
seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.get_default_dtype()
|
| 96 |
+
)
|
| 97 |
+
|
| 98 |
+
def _set_cos_sin_cache(self, seq_len, device, dtype):
|
| 99 |
+
self.max_seq_len_cached = seq_len
|
| 100 |
+
t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
|
| 101 |
+
|
| 102 |
+
freqs = torch.outer(t, self.inv_freq)
|
| 103 |
+
# Different from paper, but it uses a different permutation in order to obtain the same calculation
|
| 104 |
+
emb = torch.cat((freqs, freqs), dim=-1)
|
| 105 |
+
self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
|
| 106 |
+
self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
|
| 107 |
+
|
| 108 |
+
def forward(self, x, seq_len=None):
|
| 109 |
+
# x: [bs, num_attention_heads, seq_len, head_size]
|
| 110 |
+
if seq_len > self.max_seq_len_cached:
|
| 111 |
+
self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=x.dtype)
|
| 112 |
+
|
| 113 |
+
return (
|
| 114 |
+
self.cos_cached[:seq_len].to(dtype=x.dtype),
|
| 115 |
+
self.sin_cached[:seq_len].to(dtype=x.dtype),
|
| 116 |
+
)
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaLinearScalingRotaryEmbedding with Llama->Quasar
|
| 120 |
+
class QuasarLinearScalingRotaryEmbedding(QuasarRotaryEmbedding):
|
| 121 |
+
"""QuasarRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev"""
|
| 122 |
+
|
| 123 |
+
def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):
|
| 124 |
+
self.scaling_factor = scaling_factor
|
| 125 |
+
super().__init__(dim, max_position_embeddings, base, device)
|
| 126 |
+
|
| 127 |
+
def _set_cos_sin_cache(self, seq_len, device, dtype):
|
| 128 |
+
self.max_seq_len_cached = seq_len
|
| 129 |
+
t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
|
| 130 |
+
t = t / self.scaling_factor
|
| 131 |
+
|
| 132 |
+
freqs = torch.outer(t, self.inv_freq)
|
| 133 |
+
# Different from paper, but it uses a different permutation in order to obtain the same calculation
|
| 134 |
+
emb = torch.cat((freqs, freqs), dim=-1)
|
| 135 |
+
self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
|
| 136 |
+
self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaDynamicNTKScalingRotaryEmbedding with Llama->Quasar
|
| 140 |
+
class QuasarDynamicNTKScalingRotaryEmbedding(QuasarRotaryEmbedding):
|
| 141 |
+
"""QuasarRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla"""
|
| 142 |
+
|
| 143 |
+
def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):
|
| 144 |
+
self.scaling_factor = scaling_factor
|
| 145 |
+
super().__init__(dim, max_position_embeddings, base, device)
|
| 146 |
+
|
| 147 |
+
def _set_cos_sin_cache(self, seq_len, device, dtype):
|
| 148 |
+
self.max_seq_len_cached = seq_len
|
| 149 |
+
|
| 150 |
+
if seq_len > self.max_position_embeddings:
|
| 151 |
+
base = self.base * (
|
| 152 |
+
(self.scaling_factor * seq_len / self.max_position_embeddings) - (self.scaling_factor - 1)
|
| 153 |
+
) ** (self.dim / (self.dim - 2))
|
| 154 |
+
inv_freq = 1.0 / (base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
|
| 155 |
+
self.register_buffer("inv_freq", inv_freq, persistent=False)
|
| 156 |
+
|
| 157 |
+
t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
|
| 158 |
+
|
| 159 |
+
freqs = torch.outer(t, self.inv_freq)
|
| 160 |
+
# Different from paper, but it uses a different permutation in order to obtain the same calculation
|
| 161 |
+
emb = torch.cat((freqs, freqs), dim=-1)
|
| 162 |
+
self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
|
| 163 |
+
self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
|
| 164 |
+
|
| 165 |
+
|
| 166 |
+
##################################################################à
|
| 167 |
+
|
| 168 |
+
class QuasarMLP(nn.Module):
|
| 169 |
+
def __init__(self, config):
|
| 170 |
+
super().__init__()
|
| 171 |
+
self.config = config
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
self.activation_fn = ACT2FN[config.hidden_act]
|
| 175 |
+
self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size)
|
| 176 |
+
self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size)
|
| 177 |
+
|
| 178 |
+
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
|
| 179 |
+
hidden_states = self.fc1(hidden_states)
|
| 180 |
+
hidden_states = self.activation_fn(hidden_states)
|
| 181 |
+
hidden_states = self.fc2(hidden_states)
|
| 182 |
+
return hidden_states
|
| 183 |
+
|
| 184 |
+
|
| 185 |
+
###################################################################
|
| 186 |
+
# Copied from transformers.models.llama.modeling_llama.rotate_half
|
| 187 |
+
def rotate_half(x):
|
| 188 |
+
"""Rotates half the hidden dims of the input."""
|
| 189 |
+
x1 = x[..., : x.shape[-1] // 2]
|
| 190 |
+
x2 = x[..., x.shape[-1] // 2:]
|
| 191 |
+
return torch.cat((-x2, x1), dim=-1)
|
| 192 |
+
|
| 193 |
+
|
| 194 |
+
# Copied from transformers.models.llama.modeling_llama.apply_rotary_pos_emb
|
| 195 |
+
def apply_rotary_pos_emb(q, k, cos, sin, position_ids, unsqueeze_dim=1):
|
| 196 |
+
"""Applies Rotary Position Embedding to the query and key tensors.
|
| 197 |
+
Args:
|
| 198 |
+
q (`torch.Tensor`): The query tensor.
|
| 199 |
+
k (`torch.Tensor`): The key tensor.
|
| 200 |
+
cos (`torch.Tensor`): The cosine part of the rotary embedding.
|
| 201 |
+
sin (`torch.Tensor`): The sine part of the rotary embedding.
|
| 202 |
+
position_ids (`torch.Tensor`):
|
| 203 |
+
The position indices of the tokens corresponding to the query and key tensors. For example, this can be
|
| 204 |
+
used to pass offsetted position ids when working with a KV-cache.
|
| 205 |
+
unsqueeze_dim (`int`, *optional*, defaults to 1):
|
| 206 |
+
The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
|
| 207 |
+
sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
|
| 208 |
+
that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
|
| 209 |
+
k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
|
| 210 |
+
cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
|
| 211 |
+
the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
|
| 212 |
+
Returns:
|
| 213 |
+
`tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
|
| 214 |
+
"""
|
| 215 |
+
cos = cos[position_ids].unsqueeze(unsqueeze_dim)
|
| 216 |
+
sin = sin[position_ids].unsqueeze(unsqueeze_dim)
|
| 217 |
+
q_embed = (q * cos) + (rotate_half(q) * sin)
|
| 218 |
+
k_embed = (k * cos) + (rotate_half(k) * sin)
|
| 219 |
+
return q_embed, k_embed
|
| 220 |
+
|
| 221 |
+
|
| 222 |
+
# Copied from transformers.models.llama.modeling_llama.repeat_kv with llama-> pulsar
|
| 223 |
+
def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
|
| 224 |
+
"""
|
| 225 |
+
This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
|
| 226 |
+
num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
|
| 227 |
+
"""
|
| 228 |
+
batch, num_key_value_heads, slen, head_dim = hidden_states.shape
|
| 229 |
+
if n_rep == 1:
|
| 230 |
+
return hidden_states
|
| 231 |
+
hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
|
| 232 |
+
return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
|
| 233 |
+
|
| 234 |
+
|
| 235 |
+
class QuasarAttention(nn.Module):
|
| 236 |
+
"""Multi-headed attention from 'Attention Is All You Need' paper"""
|
| 237 |
+
|
| 238 |
+
def __init__(self, config: QuasarConfig, layer_idx: Optional[int] = None):
|
| 239 |
+
super().__init__()
|
| 240 |
+
self.config = config
|
| 241 |
+
self.layer_idx = layer_idx
|
| 242 |
+
if layer_idx is None:
|
| 243 |
+
logger.warning_once(
|
| 244 |
+
f"Instantiating {self.__class__.__name__} without passing `layer_idx` is not recommended and will "
|
| 245 |
+
"to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` "
|
| 246 |
+
"when creating this class."
|
| 247 |
+
)
|
| 248 |
+
|
| 249 |
+
self.attention_dropout = config.attention_dropout
|
| 250 |
+
self.hidden_size = config.hidden_size
|
| 251 |
+
self.num_heads = config.num_attention_heads
|
| 252 |
+
self.head_dim = self.hidden_size // self.num_heads
|
| 253 |
+
self.num_key_value_heads = config.num_key_value_heads
|
| 254 |
+
self.num_key_value_groups = self.num_heads // self.num_key_value_heads
|
| 255 |
+
self.max_position_embeddings = config.max_position_embeddings
|
| 256 |
+
self.rope_theta = config.rope_theta
|
| 257 |
+
self.partial_rotary_factor = config.partial_rotary_factor
|
| 258 |
+
self.is_causal = True
|
| 259 |
+
|
| 260 |
+
if (self.head_dim * self.num_heads) != self.hidden_size:
|
| 261 |
+
raise ValueError(
|
| 262 |
+
f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}"
|
| 263 |
+
f" and `num_heads`: {self.num_heads})."
|
| 264 |
+
)
|
| 265 |
+
|
| 266 |
+
self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=True)
|
| 267 |
+
self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=True)
|
| 268 |
+
self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=True)
|
| 269 |
+
self.dense = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=True)
|
| 270 |
+
|
| 271 |
+
self.qk_layernorm = config.qk_layernorm
|
| 272 |
+
if self.qk_layernorm:
|
| 273 |
+
|
| 274 |
+
self.q_layernorm = (
|
| 275 |
+
nn.LayerNorm(
|
| 276 |
+
config.hidden_size // self.num_heads, eps=config.layer_norm_eps, elementwise_affine=True
|
| 277 |
+
))
|
| 278 |
+
self.k_layernorm = nn.LayerNorm(
|
| 279 |
+
config.hidden_size // self.num_heads, eps=config.layer_norm_eps, elementwise_affine=True
|
| 280 |
+
)
|
| 281 |
+
|
| 282 |
+
self._init_rope()
|
| 283 |
+
|
| 284 |
+
def _init_rope(self):
|
| 285 |
+
if self.config.rope_scaling is None:
|
| 286 |
+
self.rotary_emb = QuasarRotaryEmbedding(
|
| 287 |
+
int(self.partial_rotary_factor * self.head_dim),
|
| 288 |
+
max_position_embeddings=self.max_position_embeddings,
|
| 289 |
+
base=self.rope_theta,
|
| 290 |
+
)
|
| 291 |
+
else:
|
| 292 |
+
scaling_type = self.config.rope_scaling["type"]
|
| 293 |
+
scaling_factor = self.config.rope_scaling["factor"]
|
| 294 |
+
if scaling_type == "linear":
|
| 295 |
+
self.rotary_emb = QuasarLinearScalingRotaryEmbedding(
|
| 296 |
+
int(self.partial_rotary_factor * self.head_dim),
|
| 297 |
+
max_position_embeddings=self.max_position_embeddings,
|
| 298 |
+
scaling_factor=scaling_factor,
|
| 299 |
+
base=self.rope_theta,
|
| 300 |
+
)
|
| 301 |
+
elif scaling_type == "dynamic":
|
| 302 |
+
self.rotary_emb = QuasarDynamicNTKScalingRotaryEmbedding(
|
| 303 |
+
int(self.partial_rotary_factor * self.head_dim),
|
| 304 |
+
max_position_embeddings=self.max_position_embeddings,
|
| 305 |
+
scaling_factor=scaling_factor,
|
| 306 |
+
base=self.rope_theta,
|
| 307 |
+
)
|
| 308 |
+
else:
|
| 309 |
+
raise ValueError(f"Unknown RoPE scaling type {scaling_type}")
|
| 310 |
+
|
| 311 |
+
# Quasar-2 has an attention overflow issue (with FP16) and requires autocast to be disabled
|
| 312 |
+
@torch.autocast("cpu", enabled=False)
|
| 313 |
+
@torch.autocast("cuda", enabled=False)
|
| 314 |
+
def forward(
|
| 315 |
+
self,
|
| 316 |
+
hidden_states: torch.Tensor,
|
| 317 |
+
attention_mask: Optional[torch.Tensor] = None,
|
| 318 |
+
position_ids: Optional[torch.LongTensor] = None,
|
| 319 |
+
past_key_value: Optional[Cache] = None,
|
| 320 |
+
output_attentions: bool = False,
|
| 321 |
+
use_cache: bool = False,
|
| 322 |
+
) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
|
| 323 |
+
bsz, q_len, _ = hidden_states.size()
|
| 324 |
+
|
| 325 |
+
query_states = self.q_proj(hidden_states)
|
| 326 |
+
key_states = self.k_proj(hidden_states)
|
| 327 |
+
value_states = self.v_proj(hidden_states)
|
| 328 |
+
|
| 329 |
+
if self.qk_layernorm:
|
| 330 |
+
query_states = self.q_layernorm(query_states)
|
| 331 |
+
key_states = self.k_layernorm(key_states)
|
| 332 |
+
|
| 333 |
+
query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
|
| 334 |
+
key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
|
| 335 |
+
value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
|
| 336 |
+
|
| 337 |
+
kv_seq_len = key_states.shape[-2]
|
| 338 |
+
if past_key_value is not None:
|
| 339 |
+
if self.layer_idx is None:
|
| 340 |
+
raise ValueError(
|
| 341 |
+
f"The cache structure has changed since version v4.36. If you are using {self.__class__.__name__} "
|
| 342 |
+
"for auto-regressive decoding with k/v caching, please make sure to initialize the attention class "
|
| 343 |
+
"with a layer index."
|
| 344 |
+
)
|
| 345 |
+
kv_seq_len += past_key_value.get_usable_length(kv_seq_len, self.layer_idx)
|
| 346 |
+
cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
|
| 347 |
+
|
| 348 |
+
# Partial rotary embedding
|
| 349 |
+
query_rot, query_pass = (
|
| 350 |
+
query_states[..., : self.rotary_emb.dim],
|
| 351 |
+
query_states[..., self.rotary_emb.dim:],
|
| 352 |
+
)
|
| 353 |
+
key_rot, key_pass = (
|
| 354 |
+
key_states[..., : self.rotary_emb.dim],
|
| 355 |
+
key_states[..., self.rotary_emb.dim:],
|
| 356 |
+
)
|
| 357 |
+
# [batch_size, seq_length, num_heads, head_dim // config.partial_rotary_factor]
|
| 358 |
+
query_rot, key_rot = apply_rotary_pos_emb(query_rot, key_rot, cos, sin, position_ids)
|
| 359 |
+
|
| 360 |
+
# [batch_size, seq_length, num_heads, head_dim]
|
| 361 |
+
query_states = torch.cat((query_rot, query_pass), dim=-1)
|
| 362 |
+
key_states = torch.cat((key_rot, key_pass), dim=-1)
|
| 363 |
+
|
| 364 |
+
if past_key_value is not None:
|
| 365 |
+
cache_kwargs = {"sin": sin, "cos": cos, "partial_rotation_size": self.rotary_emb.dim}
|
| 366 |
+
key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
|
| 367 |
+
|
| 368 |
+
key_states = repeat_kv(key_states, self.num_key_value_groups)
|
| 369 |
+
value_states = repeat_kv(value_states, self.num_key_value_groups)
|
| 370 |
+
|
| 371 |
+
# Queries and keys upcast to fp32 is required by Quasar-2 to avoid overflow
|
| 372 |
+
attn_weights = torch.matmul(
|
| 373 |
+
query_states.to(torch.float32), key_states.to(torch.float32).transpose(2, 3)
|
| 374 |
+
) / math.sqrt(self.head_dim)
|
| 375 |
+
|
| 376 |
+
if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len):
|
| 377 |
+
raise ValueError(
|
| 378 |
+
f"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is"
|
| 379 |
+
f" {attn_weights.size()}"
|
| 380 |
+
)
|
| 381 |
+
|
| 382 |
+
if attention_mask is not None:
|
| 383 |
+
if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
|
| 384 |
+
raise ValueError(
|
| 385 |
+
f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
|
| 386 |
+
)
|
| 387 |
+
attn_weights = attn_weights + attention_mask
|
| 388 |
+
|
| 389 |
+
# upcast attention to fp32
|
| 390 |
+
attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(value_states.dtype)
|
| 391 |
+
attn_weights = nn.functional.dropout(attn_weights, p=self.attention_dropout, training=self.training)
|
| 392 |
+
|
| 393 |
+
attn_output = torch.matmul(attn_weights, value_states)
|
| 394 |
+
|
| 395 |
+
if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):
|
| 396 |
+
raise ValueError(
|
| 397 |
+
f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is"
|
| 398 |
+
f" {attn_output.size()}"
|
| 399 |
+
)
|
| 400 |
+
|
| 401 |
+
attn_output = attn_output.transpose(1, 2).contiguous()
|
| 402 |
+
attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
|
| 403 |
+
|
| 404 |
+
attn_output = self.dense(attn_output)
|
| 405 |
+
|
| 406 |
+
if not output_attentions:
|
| 407 |
+
attn_weights = None
|
| 408 |
+
|
| 409 |
+
return attn_output, attn_weights, past_key_value
|
| 410 |
+
|
| 411 |
+
|
| 412 |
+
class QuasarFlashAttention2(QuasarAttention):
|
| 413 |
+
"""
|
| 414 |
+
Quasar flash attention module. This module inherits from `QuasarAttention` as the weights of the module stays
|
| 415 |
+
untouched. The only required change would be on the forward pass where it needs to correctly call the public API of
|
| 416 |
+
flash attention and deal with padding tokens in case the input contains any of them.
|
| 417 |
+
"""
|
| 418 |
+
|
| 419 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaFlashAttention2.__init__
|
| 420 |
+
def __init__(self, *args, **kwargs):
|
| 421 |
+
super().__init__(*args, **kwargs)
|
| 422 |
+
|
| 423 |
+
# TODO: Should be removed once Flash Attention for RoCm is bumped to 2.1.
|
| 424 |
+
# flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. Reference: https://github.com/Dao-AILab/flash-attention/releases/tag/v2.1.0.
|
| 425 |
+
# Beware that with flash_attn<2.1, using q_seqlen != k_seqlen (except for the case q_seqlen == 1) produces a wrong mask (top-left).
|
| 426 |
+
self._flash_attn_uses_top_left_mask = not is_flash_attn_greater_or_equal_2_10()
|
| 427 |
+
|
| 428 |
+
def forward(
|
| 429 |
+
self,
|
| 430 |
+
hidden_states: torch.Tensor,
|
| 431 |
+
attention_mask: Optional[torch.LongTensor] = None,
|
| 432 |
+
position_ids: Optional[torch.LongTensor] = None,
|
| 433 |
+
past_key_value: Optional[Cache] = None,
|
| 434 |
+
output_attentions: bool = False,
|
| 435 |
+
use_cache: bool = False,
|
| 436 |
+
**kwargs,
|
| 437 |
+
) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
|
| 438 |
+
# QuasarFlashAttention2 attention does not support output_attentions
|
| 439 |
+
|
| 440 |
+
output_attentions = False
|
| 441 |
+
|
| 442 |
+
bsz, q_len, _ = hidden_states.size()
|
| 443 |
+
|
| 444 |
+
query_states = self.q_proj(hidden_states)
|
| 445 |
+
key_states = self.k_proj(hidden_states)
|
| 446 |
+
value_states = self.v_proj(hidden_states)
|
| 447 |
+
|
| 448 |
+
if self.qk_layernorm:
|
| 449 |
+
query_states = self.q_layernorm(query_states)
|
| 450 |
+
key_states = self.k_layernorm(key_states)
|
| 451 |
+
|
| 452 |
+
# Flash attention requires the input to have the shape
|
| 453 |
+
# batch_size x seq_length x head_dim x hidden_dim
|
| 454 |
+
# therefore we just need to keep the original shape
|
| 455 |
+
query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
|
| 456 |
+
key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
|
| 457 |
+
value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
|
| 458 |
+
|
| 459 |
+
kv_seq_len = key_states.shape[-2]
|
| 460 |
+
if past_key_value is not None:
|
| 461 |
+
kv_seq_len += past_key_value.get_usable_length(kv_seq_len, self.layer_idx)
|
| 462 |
+
cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
|
| 463 |
+
|
| 464 |
+
# Partial rotary embedding
|
| 465 |
+
query_rot, query_pass = (
|
| 466 |
+
query_states[..., : self.rotary_emb.dim],
|
| 467 |
+
query_states[..., self.rotary_emb.dim:],
|
| 468 |
+
)
|
| 469 |
+
key_rot, key_pass = (
|
| 470 |
+
key_states[..., : self.rotary_emb.dim],
|
| 471 |
+
key_states[..., self.rotary_emb.dim:],
|
| 472 |
+
)
|
| 473 |
+
# [batch_size, seq_length, num_heads, head_dim // config.partial_rotary_factor]
|
| 474 |
+
query_rot, key_rot = apply_rotary_pos_emb(query_rot, key_rot, cos, sin, position_ids)
|
| 475 |
+
|
| 476 |
+
use_sliding_windows = (
|
| 477 |
+
_flash_supports_window_size
|
| 478 |
+
and getattr(self.config, "sliding_window", None) is not None
|
| 479 |
+
and kv_seq_len > self.config.sliding_window
|
| 480 |
+
)
|
| 481 |
+
|
| 482 |
+
if not _flash_supports_window_size:
|
| 483 |
+
logger.warning_once(
|
| 484 |
+
"The current flash attention version does not support sliding window attention, for a more memory efficient implementation"
|
| 485 |
+
" make sure to upgrade flash-attn library."
|
| 486 |
+
)
|
| 487 |
+
# [batch_size, seq_length, num_heads, head_dim]
|
| 488 |
+
query_states = torch.cat((query_rot, query_pass), dim=-1)
|
| 489 |
+
key_states = torch.cat((key_rot, key_pass), dim=-1)
|
| 490 |
+
|
| 491 |
+
if past_key_value is not None:
|
| 492 |
+
# Activate slicing cache only if the config has a value `sliding_windows` attribute
|
| 493 |
+
cache_has_contents = past_key_value.get_seq_length(self.layer_idx) > 0
|
| 494 |
+
if (
|
| 495 |
+
getattr(self.config, "sliding_window", None) is not None
|
| 496 |
+
and kv_seq_len > self.config.sliding_window
|
| 497 |
+
and cache_has_contents
|
| 498 |
+
):
|
| 499 |
+
slicing_tokens = 1 - self.config.sliding_window
|
| 500 |
+
|
| 501 |
+
past_key = past_key_value[self.layer_idx][0]
|
| 502 |
+
past_value = past_key_value[self.layer_idx][1]
|
| 503 |
+
|
| 504 |
+
past_key = past_key[:, :, slicing_tokens:, :].contiguous()
|
| 505 |
+
past_value = past_value[:, :, slicing_tokens:, :].contiguous()
|
| 506 |
+
|
| 507 |
+
if past_key.shape[-2] != self.config.sliding_window - 1:
|
| 508 |
+
raise ValueError(
|
| 509 |
+
f"past key must have a shape of (`batch_size, num_heads, self.config.sliding_window-1, head_dim`), got"
|
| 510 |
+
f" {past_key.shape}"
|
| 511 |
+
)
|
| 512 |
+
|
| 513 |
+
if attention_mask is not None:
|
| 514 |
+
attention_mask = attention_mask[:, slicing_tokens:]
|
| 515 |
+
attention_mask = torch.cat([attention_mask, torch.ones_like(attention_mask[:, -1:])], dim=-1)
|
| 516 |
+
cache_kwargs = {"sin": sin, "cos": cos, "partial_rotation_size": self.rotary_emb.dim}
|
| 517 |
+
key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
|
| 518 |
+
|
| 519 |
+
# TODO: These transpose are quite inefficient but Flash Attention requires the layout [batch_size, sequence_length, num_heads, head_dim]. We would need to refactor the KV cache
|
| 520 |
+
# to be able to avoid many of these transpose/reshape/view.
|
| 521 |
+
query_states = query_states.transpose(1, 2)
|
| 522 |
+
key_states = key_states.transpose(1, 2)
|
| 523 |
+
value_states = value_states.transpose(1, 2)
|
| 524 |
+
|
| 525 |
+
attn_dropout = self.attention_dropout if self.training else 0.0
|
| 526 |
+
|
| 527 |
+
# In PEFT, usually we cast the layer norms in float32 for training stability reasons
|
| 528 |
+
# therefore the input hidden states gets silently casted in float32. Hence, we need
|
| 529 |
+
# cast them back in the correct dtype just to be sure everything works as expected.
|
| 530 |
+
# This might slowdown training & inference so it is recommended to not cast the LayerNorms
|
| 531 |
+
# in fp32.
|
| 532 |
+
|
| 533 |
+
if query_states.dtype == torch.float32:
|
| 534 |
+
if torch.is_autocast_enabled():
|
| 535 |
+
target_dtype = torch.get_autocast_gpu_dtype()
|
| 536 |
+
# Handle the case where the model is quantized
|
| 537 |
+
elif hasattr(self.config, "_pre_quantization_dtype"):
|
| 538 |
+
target_dtype = self.config._pre_quantization_dtype
|
| 539 |
+
else:
|
| 540 |
+
target_dtype = self.q_proj.weight.dtype
|
| 541 |
+
|
| 542 |
+
logger.warning_once(
|
| 543 |
+
f"The input hidden states seems to be silently casted in float32, this might be related to"
|
| 544 |
+
f" the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in"
|
| 545 |
+
f" {target_dtype}."
|
| 546 |
+
)
|
| 547 |
+
|
| 548 |
+
query_states = query_states.to(target_dtype)
|
| 549 |
+
key_states = key_states.to(target_dtype)
|
| 550 |
+
value_states = value_states.to(target_dtype)
|
| 551 |
+
|
| 552 |
+
attn_output = self._flash_attention_forward(
|
| 553 |
+
query_states, key_states, value_states, attention_mask, q_len, dropout=attn_dropout, softmax_scale=None,
|
| 554 |
+
use_sliding_windows=use_sliding_windows
|
| 555 |
+
)
|
| 556 |
+
|
| 557 |
+
attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
|
| 558 |
+
attn_output = self.dense(attn_output)
|
| 559 |
+
|
| 560 |
+
if not output_attentions:
|
| 561 |
+
attn_weights = None
|
| 562 |
+
|
| 563 |
+
return attn_output, attn_weights, past_key_value
|
| 564 |
+
|
| 565 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaFlashAttention2._flash_attention_forward
|
| 566 |
+
def _flash_attention_forward(
|
| 567 |
+
self, query_states, key_states, value_states, attention_mask, query_length, dropout=0.0, softmax_scale=None,
|
| 568 |
+
use_sliding_windows=False
|
| 569 |
+
):
|
| 570 |
+
"""
|
| 571 |
+
Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token
|
| 572 |
+
first unpad the input, then computes the attention scores and pad the final attention scores.
|
| 573 |
+
Args:
|
| 574 |
+
query_states (`torch.Tensor`):
|
| 575 |
+
Input query states to be passed to Flash Attention API
|
| 576 |
+
key_states (`torch.Tensor`):
|
| 577 |
+
Input key states to be passed to Flash Attention API
|
| 578 |
+
value_states (`torch.Tensor`):
|
| 579 |
+
Input value states to be passed to Flash Attention API
|
| 580 |
+
attention_mask (`torch.Tensor`):
|
| 581 |
+
The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the
|
| 582 |
+
position of padding tokens and 1 for the position of non-padding tokens.
|
| 583 |
+
dropout (`int`, *optional*):
|
| 584 |
+
Attention dropout
|
| 585 |
+
softmax_scale (`float`, *optional*):
|
| 586 |
+
The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim)
|
| 587 |
+
use_sliding_windows (`bool`, *optional*):
|
| 588 |
+
Whether to use sliding windows for the attention computation. Default to False.
|
| 589 |
+
"""
|
| 590 |
+
if not self._flash_attn_uses_top_left_mask:
|
| 591 |
+
causal = self.is_causal
|
| 592 |
+
else:
|
| 593 |
+
# TODO: Remove the `query_length != 1` check once Flash Attention for RoCm is bumped to 2.1. For details, please see the comment in LlamaFlashAttention2 __init__.
|
| 594 |
+
causal = self.is_causal and query_length != 1
|
| 595 |
+
|
| 596 |
+
# Contains at least one padding token in the sequence
|
| 597 |
+
if attention_mask is not None:
|
| 598 |
+
batch_size = query_states.shape[0]
|
| 599 |
+
query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = self._upad_input(
|
| 600 |
+
query_states, key_states, value_states, attention_mask, query_length
|
| 601 |
+
)
|
| 602 |
+
|
| 603 |
+
cu_seqlens_q, cu_seqlens_k = cu_seq_lens
|
| 604 |
+
max_seqlen_in_batch_q, max_seqlen_in_batch_k = max_seq_lens
|
| 605 |
+
if not use_sliding_windows:
|
| 606 |
+
attn_output_unpad = flash_attn_varlen_func(
|
| 607 |
+
query_states,
|
| 608 |
+
key_states,
|
| 609 |
+
value_states,
|
| 610 |
+
cu_seqlens_q=cu_seqlens_q,
|
| 611 |
+
cu_seqlens_k=cu_seqlens_k,
|
| 612 |
+
max_seqlen_q=max_seqlen_in_batch_q,
|
| 613 |
+
max_seqlen_k=max_seqlen_in_batch_k,
|
| 614 |
+
dropout_p=dropout,
|
| 615 |
+
softmax_scale=softmax_scale,
|
| 616 |
+
causal=causal,
|
| 617 |
+
)
|
| 618 |
+
else:
|
| 619 |
+
attn_output_unpad = flash_attn_varlen_func(
|
| 620 |
+
query_states,
|
| 621 |
+
key_states,
|
| 622 |
+
value_states,
|
| 623 |
+
cu_seqlens_q=cu_seqlens_q,
|
| 624 |
+
cu_seqlens_k=cu_seqlens_k,
|
| 625 |
+
max_seqlen_q=max_seqlen_in_batch_q,
|
| 626 |
+
max_seqlen_k=max_seqlen_in_batch_k,
|
| 627 |
+
dropout_p=dropout,
|
| 628 |
+
softmax_scale=softmax_scale,
|
| 629 |
+
causal=causal,
|
| 630 |
+
window_size=(self.config.sliding_window, self.config.sliding_window),
|
| 631 |
+
)
|
| 632 |
+
|
| 633 |
+
attn_output = pad_input(attn_output_unpad, indices_q, batch_size, query_length)
|
| 634 |
+
else:
|
| 635 |
+
if not use_sliding_windows:
|
| 636 |
+
attn_output = flash_attn_func(
|
| 637 |
+
query_states, key_states, value_states, dropout, softmax_scale=softmax_scale, causal=causal
|
| 638 |
+
)
|
| 639 |
+
else:
|
| 640 |
+
attn_output = flash_attn_func(
|
| 641 |
+
query_states, key_states, value_states, dropout, softmax_scale=softmax_scale, causal=causal,
|
| 642 |
+
window_size=(self.config.sliding_window, self.config.sliding_window)
|
| 643 |
+
)
|
| 644 |
+
|
| 645 |
+
return attn_output
|
| 646 |
+
|
| 647 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaFlashAttention2._upad_input
|
| 648 |
+
def _upad_input(self, query_layer, key_layer, value_layer, attention_mask, query_length):
|
| 649 |
+
indices_k, cu_seqlens_k, max_seqlen_in_batch_k = _get_unpad_data(attention_mask)
|
| 650 |
+
batch_size, kv_seq_len, num_key_value_heads, head_dim = key_layer.shape
|
| 651 |
+
|
| 652 |
+
key_layer = index_first_axis(
|
| 653 |
+
key_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k
|
| 654 |
+
)
|
| 655 |
+
value_layer = index_first_axis(
|
| 656 |
+
value_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k
|
| 657 |
+
)
|
| 658 |
+
if query_length == kv_seq_len:
|
| 659 |
+
query_layer = index_first_axis(
|
| 660 |
+
query_layer.reshape(batch_size * kv_seq_len, self.num_heads, head_dim), indices_k
|
| 661 |
+
)
|
| 662 |
+
cu_seqlens_q = cu_seqlens_k
|
| 663 |
+
max_seqlen_in_batch_q = max_seqlen_in_batch_k
|
| 664 |
+
indices_q = indices_k
|
| 665 |
+
elif query_length == 1:
|
| 666 |
+
max_seqlen_in_batch_q = 1
|
| 667 |
+
cu_seqlens_q = torch.arange(
|
| 668 |
+
batch_size + 1, dtype=torch.int32, device=query_layer.device
|
| 669 |
+
) # There is a memcpy here, that is very bad.
|
| 670 |
+
indices_q = cu_seqlens_q[:-1]
|
| 671 |
+
query_layer = query_layer.squeeze(1)
|
| 672 |
+
else:
|
| 673 |
+
# The -q_len: slice assumes left padding.
|
| 674 |
+
attention_mask = attention_mask[:, -query_length:]
|
| 675 |
+
query_layer, indices_q, cu_seqlens_q, max_seqlen_in_batch_q = unpad_input(query_layer, attention_mask)
|
| 676 |
+
|
| 677 |
+
return (
|
| 678 |
+
query_layer,
|
| 679 |
+
key_layer,
|
| 680 |
+
value_layer,
|
| 681 |
+
indices_q,
|
| 682 |
+
(cu_seqlens_q, cu_seqlens_k),
|
| 683 |
+
(max_seqlen_in_batch_q, max_seqlen_in_batch_k),
|
| 684 |
+
)
|
| 685 |
+
|
| 686 |
+
|
| 687 |
+
QUASAR_ATTENTION_CLASSES = {
|
| 688 |
+
"eager": QuasarAttention,
|
| 689 |
+
"flash_attention_2": QuasarFlashAttention2,
|
| 690 |
+
}
|
| 691 |
+
|
| 692 |
+
|
| 693 |
+
class QuasarDecoderLayer(nn.Module):
|
| 694 |
+
def __init__(self, config: QuasarConfig, layer_idx: int):
|
| 695 |
+
super().__init__()
|
| 696 |
+
self.self_attn = QUASAR_ATTENTION_CLASSES[config._attn_implementation](config, layer_idx=layer_idx)
|
| 697 |
+
self.mlp = QuasarMLP(config)
|
| 698 |
+
|
| 699 |
+
self.input_layernorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
|
| 700 |
+
|
| 701 |
+
self.resid_dropout = nn.Dropout(config.resid_pdrop)
|
| 702 |
+
|
| 703 |
+
def forward(
|
| 704 |
+
self,
|
| 705 |
+
hidden_states: torch.Tensor,
|
| 706 |
+
attention_mask: Optional[torch.Tensor] = None,
|
| 707 |
+
position_ids: Optional[torch.LongTensor] = None,
|
| 708 |
+
output_attentions: Optional[bool] = False,
|
| 709 |
+
use_cache: Optional[bool] = False,
|
| 710 |
+
past_key_value: Optional[Tuple[torch.Tensor]] = None,
|
| 711 |
+
) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
|
| 712 |
+
"""
|
| 713 |
+
Args:
|
| 714 |
+
hidden_states (`torch.FloatTensor`):
|
| 715 |
+
input to the layer of shape `(batch, seq_len, embed_dim)`
|
| 716 |
+
attention_mask (`torch.FloatTensor`, *optional*): attention mask of size
|
| 717 |
+
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
|
| 718 |
+
position_ids (`torch.LongTensor` of shape `({0})`, *optional*):
|
| 719 |
+
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range
|
| 720 |
+
`[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids)
|
| 721 |
+
output_attentions (`bool`, *optional*):
|
| 722 |
+
Whether or not to return the attentions tensors of all attention layers. See `attentions` under
|
| 723 |
+
returned tensors for more detail.
|
| 724 |
+
use_cache (`bool`, *optional*):
|
| 725 |
+
If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
|
| 726 |
+
(see `past_key_values`).
|
| 727 |
+
past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states
|
| 728 |
+
"""
|
| 729 |
+
|
| 730 |
+
residual = hidden_states
|
| 731 |
+
|
| 732 |
+
hidden_states = self.input_layernorm(hidden_states)
|
| 733 |
+
|
| 734 |
+
# Self Attention
|
| 735 |
+
attn_outputs, self_attn_weights, present_key_value = self.self_attn(
|
| 736 |
+
hidden_states=hidden_states,
|
| 737 |
+
attention_mask=attention_mask,
|
| 738 |
+
position_ids=position_ids,
|
| 739 |
+
past_key_value=past_key_value,
|
| 740 |
+
output_attentions=output_attentions,
|
| 741 |
+
use_cache=use_cache,
|
| 742 |
+
)
|
| 743 |
+
attn_outputs = self.resid_dropout(attn_outputs)
|
| 744 |
+
|
| 745 |
+
feed_forward_hidden_states = self.resid_dropout(self.mlp(hidden_states))
|
| 746 |
+
hidden_states = attn_outputs + feed_forward_hidden_states + residual
|
| 747 |
+
outputs = (hidden_states,)
|
| 748 |
+
|
| 749 |
+
if output_attentions:
|
| 750 |
+
outputs += (self_attn_weights,)
|
| 751 |
+
|
| 752 |
+
if use_cache:
|
| 753 |
+
outputs += (present_key_value,)
|
| 754 |
+
|
| 755 |
+
return outputs
|
| 756 |
+
|
| 757 |
+
|
| 758 |
+
QUASAR_START_DOCSTRING = r"""
|
| 759 |
+
This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
|
| 760 |
+
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
|
| 761 |
+
etc.)
|
| 762 |
+
This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
|
| 763 |
+
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
|
| 764 |
+
and behavior.
|
| 765 |
+
Parameters:
|
| 766 |
+
config ([`QuasarConfig`]):
|
| 767 |
+
Model configuration class with all the parameters of the model. Initializing with a config file does not
|
| 768 |
+
load the weights associated with the model, only the configuration. Check out the
|
| 769 |
+
[`~PreTrainedModel.from_pretrained`] method to load the model weights.
|
| 770 |
+
"""
|
| 771 |
+
|
| 772 |
+
|
| 773 |
+
@add_start_docstrings(
|
| 774 |
+
"The bare Quasar Model outputting raw hidden-states without any specific head on top.",
|
| 775 |
+
QUASAR_START_DOCSTRING,
|
| 776 |
+
)
|
| 777 |
+
class QuasarPreTrainedModel(PreTrainedModel):
|
| 778 |
+
config_class = QuasarConfig
|
| 779 |
+
base_model_prefix = "model"
|
| 780 |
+
supports_gradient_checkpointing = True
|
| 781 |
+
_no_split_modules = ["QuasarDecoderLayer"]
|
| 782 |
+
_skip_keys_device_placement = "past_key_values"
|
| 783 |
+
_supports_flash_attn_2 = True
|
| 784 |
+
_supports_cache_class = True
|
| 785 |
+
|
| 786 |
+
def _init_weights(self, module):
|
| 787 |
+
std = self.config.initializer_range
|
| 788 |
+
if isinstance(module, nn.Linear):
|
| 789 |
+
module.weight.data.normal_(mean=0.0, std=std)
|
| 790 |
+
if module.bias is not None:
|
| 791 |
+
module.bias.data.zero_()
|
| 792 |
+
elif isinstance(module, nn.Embedding):
|
| 793 |
+
module.weight.data.normal_(mean=0.0, std=std)
|
| 794 |
+
if module.padding_idx is not None:
|
| 795 |
+
module.weight.data[module.padding_idx].zero_()
|
| 796 |
+
|
| 797 |
+
|
| 798 |
+
QUASAR_INPUTS_DOCSTRING = r"""
|
| 799 |
+
Args:
|
| 800 |
+
input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
|
| 801 |
+
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
|
| 802 |
+
it.
|
| 803 |
+
Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
|
| 804 |
+
[`PreTrainedTokenizer.__call__`] for details.
|
| 805 |
+
[What are input IDs?](../glossary#input-ids)
|
| 806 |
+
attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
|
| 807 |
+
Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
|
| 808 |
+
- 1 for tokens that are **not masked**,
|
| 809 |
+
- 0 for tokens that are **masked**.
|
| 810 |
+
[What are attention masks?](../glossary#attention-mask)
|
| 811 |
+
Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
|
| 812 |
+
[`PreTrainedTokenizer.__call__`] for details.
|
| 813 |
+
If `past_key_values` is used, optionally only the last `input_ids` have to be input (see
|
| 814 |
+
`past_key_values`).
|
| 815 |
+
If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`]
|
| 816 |
+
and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more
|
| 817 |
+
information on the default strategy.
|
| 818 |
+
- 1 indicates the head is **not masked**,
|
| 819 |
+
- 0 indicates the head is **masked**.
|
| 820 |
+
position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
|
| 821 |
+
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
|
| 822 |
+
config.n_positions - 1]`.
|
| 823 |
+
[What are position IDs?](../glossary#position-ids)
|
| 824 |
+
past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*):
|
| 825 |
+
Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
|
| 826 |
+
blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values`
|
| 827 |
+
returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`.
|
| 828 |
+
Two formats are allowed:
|
| 829 |
+
- a [`~cache_utils.Cache`] instance;
|
| 830 |
+
- Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of
|
| 831 |
+
shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy
|
| 832 |
+
cache format.
|
| 833 |
+
The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the
|
| 834 |
+
legacy cache format will be returned.
|
| 835 |
+
If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't
|
| 836 |
+
have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids`
|
| 837 |
+
of shape `(batch_size, sequence_length)`.
|
| 838 |
+
inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
|
| 839 |
+
Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
|
| 840 |
+
is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
|
| 841 |
+
model's internal embedding lookup matrix.
|
| 842 |
+
use_cache (`bool`, *optional*):
|
| 843 |
+
If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
|
| 844 |
+
`past_key_values`).
|
| 845 |
+
output_attentions (`bool`, *optional*):
|
| 846 |
+
Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
|
| 847 |
+
tensors for more detail.
|
| 848 |
+
output_hidden_states (`bool`, *optional*):
|
| 849 |
+
Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
|
| 850 |
+
more detail.
|
| 851 |
+
return_dict (`bool`, *optional*):
|
| 852 |
+
Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
|
| 853 |
+
"""
|
| 854 |
+
|
| 855 |
+
|
| 856 |
+
@add_start_docstrings(
|
| 857 |
+
"The bare Quasar Model outputting raw hidden-states without any specific head on top.",
|
| 858 |
+
QUASAR_START_DOCSTRING,
|
| 859 |
+
)
|
| 860 |
+
class QuasarModel(QuasarPreTrainedModel):
|
| 861 |
+
"""
|
| 862 |
+
Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`QuasarDecoderLayer`]
|
| 863 |
+
Args:
|
| 864 |
+
config: QuasarConfig
|
| 865 |
+
"""
|
| 866 |
+
|
| 867 |
+
def __init__(self, config: QuasarConfig):
|
| 868 |
+
super().__init__(config)
|
| 869 |
+
self.padding_idx = config.pad_token_id
|
| 870 |
+
self.vocab_size = config.vocab_size
|
| 871 |
+
|
| 872 |
+
self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
|
| 873 |
+
self.embed_dropout = nn.Dropout(config.embd_pdrop)
|
| 874 |
+
self.layers = nn.ModuleList(
|
| 875 |
+
[QuasarDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
|
| 876 |
+
)
|
| 877 |
+
|
| 878 |
+
self.final_layernorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
|
| 879 |
+
self._use_flash_attention_2 = config._attn_implementation == "flash_attention_2"
|
| 880 |
+
|
| 881 |
+
self.gradient_checkpointing = False
|
| 882 |
+
# Initialize weights and apply final processing
|
| 883 |
+
self.duplicate_grad = config.duplicate_grad
|
| 884 |
+
self.duplicate_trick = config.duplicate_trick
|
| 885 |
+
self.layer_ranges = [tuple(inner_list) for inner_list in config.layer_ranges]
|
| 886 |
+
if self.duplicate_trick and not self.layer_ranges:
|
| 887 |
+
raise ValueError("When using the duplicate trick, `layer_ranges` must be specified.")
|
| 888 |
+
|
| 889 |
+
self.post_init()
|
| 890 |
+
|
| 891 |
+
def get_input_embeddings(self):
|
| 892 |
+
return self.embed_tokens
|
| 893 |
+
|
| 894 |
+
def set_input_embeddings(self, value):
|
| 895 |
+
self.embed_tokens = value
|
| 896 |
+
|
| 897 |
+
def set_layer_idx(self, layer_idx, layer):
|
| 898 |
+
def set_layer_idx_fn(module):
|
| 899 |
+
if hasattr(module, 'self_attn'):
|
| 900 |
+
if hasattr(module.self_attn, 'layer_idx'):
|
| 901 |
+
module.self_attn.layer_idx = layer_idx
|
| 902 |
+
|
| 903 |
+
layer.apply(set_layer_idx_fn)
|
| 904 |
+
|
| 905 |
+
@add_start_docstrings_to_model_forward(QUASAR_INPUTS_DOCSTRING)
|
| 906 |
+
def forward(
|
| 907 |
+
self,
|
| 908 |
+
input_ids: torch.LongTensor = None,
|
| 909 |
+
attention_mask: Optional[torch.Tensor] = None,
|
| 910 |
+
position_ids: Optional[torch.LongTensor] = None,
|
| 911 |
+
past_key_values: Optional[List[torch.FloatTensor]] = None,
|
| 912 |
+
inputs_embeds: Optional[torch.FloatTensor] = None,
|
| 913 |
+
use_cache: Optional[bool] = None,
|
| 914 |
+
output_attentions: Optional[bool] = None,
|
| 915 |
+
output_hidden_states: Optional[bool] = None,
|
| 916 |
+
return_dict: Optional[bool] = None,
|
| 917 |
+
) -> Union[Tuple, BaseModelOutputWithPast]:
|
| 918 |
+
output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
|
| 919 |
+
output_hidden_states = (
|
| 920 |
+
output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
|
| 921 |
+
)
|
| 922 |
+
use_cache = use_cache if use_cache is not None else self.config.use_cache
|
| 923 |
+
|
| 924 |
+
return_dict = return_dict if return_dict is not None else self.config.use_return_dict
|
| 925 |
+
|
| 926 |
+
# retrieve input_ids and inputs_embeds
|
| 927 |
+
if input_ids is not None and inputs_embeds is not None:
|
| 928 |
+
raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
|
| 929 |
+
elif input_ids is not None:
|
| 930 |
+
batch_size, seq_length = input_ids.shape[:2]
|
| 931 |
+
elif inputs_embeds is not None:
|
| 932 |
+
batch_size, seq_length = inputs_embeds.shape[:2]
|
| 933 |
+
else:
|
| 934 |
+
raise ValueError("You have to specify either input_ids or inputs_embeds")
|
| 935 |
+
|
| 936 |
+
past_key_values_length = 0
|
| 937 |
+
|
| 938 |
+
if self.gradient_checkpointing and self.training:
|
| 939 |
+
if use_cache:
|
| 940 |
+
logger.warning_once(
|
| 941 |
+
"`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
|
| 942 |
+
)
|
| 943 |
+
use_cache = False
|
| 944 |
+
|
| 945 |
+
if use_cache:
|
| 946 |
+
use_legacy_cache = not isinstance(past_key_values, Cache)
|
| 947 |
+
if use_legacy_cache:
|
| 948 |
+
past_key_values = DynamicCache.from_legacy_cache(past_key_values)
|
| 949 |
+
past_key_values_length = past_key_values.get_usable_length(seq_length)
|
| 950 |
+
|
| 951 |
+
if position_ids is None:
|
| 952 |
+
device = input_ids.device if input_ids is not None else inputs_embeds.device
|
| 953 |
+
position_ids = torch.arange(
|
| 954 |
+
past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device
|
| 955 |
+
)
|
| 956 |
+
position_ids = position_ids.unsqueeze(0)
|
| 957 |
+
|
| 958 |
+
if inputs_embeds is None:
|
| 959 |
+
inputs_embeds = self.embed_tokens(input_ids)
|
| 960 |
+
|
| 961 |
+
inputs_embeds = self.embed_dropout(inputs_embeds)
|
| 962 |
+
|
| 963 |
+
# Attention mask.
|
| 964 |
+
if self._use_flash_attention_2:
|
| 965 |
+
# 2d mask is passed through the layers
|
| 966 |
+
attention_mask = attention_mask if (attention_mask is not None and 0 in attention_mask) else None
|
| 967 |
+
else:
|
| 968 |
+
# 4d mask is passed through the layers
|
| 969 |
+
attention_mask = _prepare_4d_causal_attention_mask(
|
| 970 |
+
attention_mask, (batch_size, seq_length), inputs_embeds, past_key_values_length,
|
| 971 |
+
sliding_window=self.config.sliding_window
|
| 972 |
+
)
|
| 973 |
+
|
| 974 |
+
hidden_states = inputs_embeds
|
| 975 |
+
|
| 976 |
+
# decoder layers
|
| 977 |
+
all_hidden_states = () if output_hidden_states else None
|
| 978 |
+
all_self_attns = () if output_attentions else None
|
| 979 |
+
next_decoder_cache = None
|
| 980 |
+
|
| 981 |
+
# Modify the forward method to use the duplicate trick
|
| 982 |
+
if self.duplicate_trick:
|
| 983 |
+
sequential_layer_idx = 0
|
| 984 |
+
# Iterate over each specified range of layers
|
| 985 |
+
for idx, (start_layer, end_layer) in enumerate(self.layer_ranges):
|
| 986 |
+
# if the layers are the original layers or self.duplicate_grad is False
|
| 987 |
+
duplicate_backprop = True
|
| 988 |
+
if idx % 2 != 0 and not self.duplicate_grad:
|
| 989 |
+
# if the layers are the duplicate layers and self.duplicate_grad is False
|
| 990 |
+
duplicate_backprop = False
|
| 991 |
+
|
| 992 |
+
for layer_idx in range(start_layer, end_layer + 1):
|
| 993 |
+
decoder_layer = self.layers[layer_idx] # Access the specific layer
|
| 994 |
+
if self.training and not duplicate_backprop:
|
| 995 |
+
for param in decoder_layer.parameters():
|
| 996 |
+
param.requires_grad = duplicate_backprop
|
| 997 |
+
self.set_layer_idx(sequential_layer_idx, decoder_layer)
|
| 998 |
+
# The rest remains the same as the original loop
|
| 999 |
+
if output_hidden_states:
|
| 1000 |
+
all_hidden_states += (hidden_states,)
|
| 1001 |
+
|
| 1002 |
+
if self.gradient_checkpointing and self.training:
|
| 1003 |
+
layer_outputs = self._gradient_checkpointing_func(
|
| 1004 |
+
decoder_layer.__call__,
|
| 1005 |
+
hidden_states,
|
| 1006 |
+
attention_mask,
|
| 1007 |
+
position_ids,
|
| 1008 |
+
past_key_values,
|
| 1009 |
+
output_attentions,
|
| 1010 |
+
)
|
| 1011 |
+
else:
|
| 1012 |
+
layer_outputs = decoder_layer(
|
| 1013 |
+
hidden_states,
|
| 1014 |
+
attention_mask=attention_mask,
|
| 1015 |
+
position_ids=position_ids,
|
| 1016 |
+
past_key_value=past_key_values,
|
| 1017 |
+
output_attentions=output_attentions,
|
| 1018 |
+
use_cache=use_cache,
|
| 1019 |
+
)
|
| 1020 |
+
|
| 1021 |
+
hidden_states = layer_outputs[0]
|
| 1022 |
+
|
| 1023 |
+
if use_cache:
|
| 1024 |
+
next_decoder_cache = layer_outputs[2 if output_attentions else 1]
|
| 1025 |
+
|
| 1026 |
+
if output_attentions:
|
| 1027 |
+
all_self_attns += (layer_outputs[1],)
|
| 1028 |
+
sequential_layer_idx += 1
|
| 1029 |
+
else:
|
| 1030 |
+
for decoder_layer in self.layers:
|
| 1031 |
+
if output_hidden_states:
|
| 1032 |
+
all_hidden_states += (hidden_states,)
|
| 1033 |
+
|
| 1034 |
+
if self.gradient_checkpointing and self.training:
|
| 1035 |
+
layer_outputs = self._gradient_checkpointing_func(
|
| 1036 |
+
decoder_layer.__call__,
|
| 1037 |
+
hidden_states,
|
| 1038 |
+
attention_mask,
|
| 1039 |
+
position_ids,
|
| 1040 |
+
past_key_values,
|
| 1041 |
+
output_attentions,
|
| 1042 |
+
)
|
| 1043 |
+
else:
|
| 1044 |
+
layer_outputs = decoder_layer(
|
| 1045 |
+
hidden_states,
|
| 1046 |
+
attention_mask=attention_mask,
|
| 1047 |
+
position_ids=position_ids,
|
| 1048 |
+
past_key_value=past_key_values,
|
| 1049 |
+
output_attentions=output_attentions,
|
| 1050 |
+
use_cache=use_cache,
|
| 1051 |
+
)
|
| 1052 |
+
|
| 1053 |
+
hidden_states = layer_outputs[0]
|
| 1054 |
+
|
| 1055 |
+
if use_cache:
|
| 1056 |
+
next_decoder_cache = layer_outputs[2 if output_attentions else 1]
|
| 1057 |
+
|
| 1058 |
+
if output_attentions:
|
| 1059 |
+
all_self_attns += (layer_outputs[1],)
|
| 1060 |
+
|
| 1061 |
+
hidden_states = self.final_layernorm(hidden_states)
|
| 1062 |
+
|
| 1063 |
+
# add hidden states from the last decoder layer
|
| 1064 |
+
if output_hidden_states:
|
| 1065 |
+
all_hidden_states += (hidden_states,)
|
| 1066 |
+
|
| 1067 |
+
next_cache = None
|
| 1068 |
+
if use_cache:
|
| 1069 |
+
next_cache = next_decoder_cache.to_legacy_cache() if use_legacy_cache else next_decoder_cache
|
| 1070 |
+
if not return_dict:
|
| 1071 |
+
return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)
|
| 1072 |
+
return BaseModelOutputWithPast(
|
| 1073 |
+
last_hidden_state=hidden_states,
|
| 1074 |
+
past_key_values=next_cache,
|
| 1075 |
+
hidden_states=all_hidden_states,
|
| 1076 |
+
attentions=all_self_attns,
|
| 1077 |
+
)
|
| 1078 |
+
|
| 1079 |
+
|
| 1080 |
+
class QuasarForCausalLM(QuasarPreTrainedModel):
|
| 1081 |
+
_tied_weights_keys = ["lm_head.weight"]
|
| 1082 |
+
|
| 1083 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaForCausalLM.__init__ with Llama->Quasar,bias=False->bias=True
|
| 1084 |
+
def __init__(self, config):
|
| 1085 |
+
super().__init__(config)
|
| 1086 |
+
self.model = QuasarModel(config)
|
| 1087 |
+
self.vocab_size = config.vocab_size
|
| 1088 |
+
self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=True)
|
| 1089 |
+
|
| 1090 |
+
# Initialize weights and apply final processing
|
| 1091 |
+
self.post_init()
|
| 1092 |
+
if config.remove_ff_bias:
|
| 1093 |
+
self.model = self.remove_ff_linear_bias()
|
| 1094 |
+
|
| 1095 |
+
def remove_ff_linear_bias(self):
|
| 1096 |
+
for layer in self.model.layers:
|
| 1097 |
+
if isinstance(layer, QuasarDecoderLayer):
|
| 1098 |
+
ff_layer = layer.mlp
|
| 1099 |
+
for module in ff_layer.modules():
|
| 1100 |
+
if isinstance(module, nn.Linear):
|
| 1101 |
+
module.bias = None
|
| 1102 |
+
return self.model
|
| 1103 |
+
|
| 1104 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaForCausalLM.get_input_embeddings
|
| 1105 |
+
def get_input_embeddings(self):
|
| 1106 |
+
return self.model.embed_tokens
|
| 1107 |
+
|
| 1108 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaForCausalLM.set_input_embeddings
|
| 1109 |
+
def set_input_embeddings(self, value):
|
| 1110 |
+
self.model.embed_tokens = value
|
| 1111 |
+
|
| 1112 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaForCausalLM.get_output_embeddings
|
| 1113 |
+
def get_output_embeddings(self):
|
| 1114 |
+
return self.lm_head
|
| 1115 |
+
|
| 1116 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaForCausalLM.set_output_embeddings
|
| 1117 |
+
def set_output_embeddings(self, new_embeddings):
|
| 1118 |
+
self.lm_head = new_embeddings
|
| 1119 |
+
|
| 1120 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaForCausalLM.set_decoder
|
| 1121 |
+
def set_decoder(self, decoder):
|
| 1122 |
+
self.model = decoder
|
| 1123 |
+
|
| 1124 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaForCausalLM.get_decoder
|
| 1125 |
+
def get_decoder(self):
|
| 1126 |
+
return self.model
|
| 1127 |
+
|
| 1128 |
+
@add_start_docstrings_to_model_forward(QUASAR_INPUTS_DOCSTRING)
|
| 1129 |
+
@replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)
|
| 1130 |
+
def forward(
|
| 1131 |
+
self,
|
| 1132 |
+
input_ids: torch.LongTensor = None,
|
| 1133 |
+
attention_mask: Optional[torch.Tensor] = None,
|
| 1134 |
+
position_ids: Optional[torch.LongTensor] = None,
|
| 1135 |
+
past_key_values: Optional[List[torch.FloatTensor]] = None,
|
| 1136 |
+
inputs_embeds: Optional[torch.FloatTensor] = None,
|
| 1137 |
+
labels: Optional[torch.LongTensor] = None,
|
| 1138 |
+
use_cache: Optional[bool] = None,
|
| 1139 |
+
output_attentions: Optional[bool] = None,
|
| 1140 |
+
output_hidden_states: Optional[bool] = None,
|
| 1141 |
+
return_dict: Optional[bool] = None,
|
| 1142 |
+
) -> Union[Tuple, CausalLMOutputWithPast]:
|
| 1143 |
+
r"""
|
| 1144 |
+
Args:
|
| 1145 |
+
labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
|
| 1146 |
+
Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
|
| 1147 |
+
config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
|
| 1148 |
+
(masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
|
| 1149 |
+
Returns:
|
| 1150 |
+
Example:
|
| 1151 |
+
```python
|
| 1152 |
+
>>> from transformers import AutoTokenizer, AutoModelForCausalLM
|
| 1153 |
+
>>> model = AutoModelForCausalLM.from_pretrained("AstraMindAI/AstraQuasar-4.5B", trust_remote_code=True)
|
| 1154 |
+
>>> tokenizer = AutoTokenizer.from_pretrained("AstraMindAI/AstraQuasar-4.5B")
|
| 1155 |
+
>>> # you can optionally disable the duplicate trick
|
| 1156 |
+
>>> # model.model.duplicate_trick = False
|
| 1157 |
+
>>> # You can also disable the duplicate gradient calculation
|
| 1158 |
+
>>> # model.model.duplicate_grad = False
|
| 1159 |
+
>>> # You can specify the layer ranges for the duplicate trick
|
| 1160 |
+
>>> # model.model.layer_ranges = [(0, 16),(8, 24),(17, 32),(25, 40),(33, 49),(40, 56)]
|
| 1161 |
+
>>> prompt = "This is an example script ."
|
| 1162 |
+
>>> inputs = tokenizer(prompt, return_tensors="pt")
|
| 1163 |
+
>>> # Generate
|
| 1164 |
+
>>> generate_ids = model.generate(inputs.input_ids, max_length=30)
|
| 1165 |
+
>>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
|
| 1166 |
+
'This is an example script .\n\n\n\nfrom typing import List\n\ndef find_most_common_letter(words: List[str'
|
| 1167 |
+
```"""
|
| 1168 |
+
|
| 1169 |
+
output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
|
| 1170 |
+
output_hidden_states = (
|
| 1171 |
+
output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
|
| 1172 |
+
)
|
| 1173 |
+
return_dict = return_dict if return_dict is not None else self.config.use_return_dict
|
| 1174 |
+
|
| 1175 |
+
# decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
|
| 1176 |
+
outputs = self.model(
|
| 1177 |
+
input_ids=input_ids,
|
| 1178 |
+
attention_mask=attention_mask,
|
| 1179 |
+
position_ids=position_ids,
|
| 1180 |
+
past_key_values=past_key_values,
|
| 1181 |
+
inputs_embeds=inputs_embeds,
|
| 1182 |
+
use_cache=use_cache,
|
| 1183 |
+
output_attentions=output_attentions,
|
| 1184 |
+
output_hidden_states=output_hidden_states,
|
| 1185 |
+
return_dict=return_dict,
|
| 1186 |
+
)
|
| 1187 |
+
|
| 1188 |
+
hidden_states = outputs[0]
|
| 1189 |
+
logits = self.lm_head(hidden_states)
|
| 1190 |
+
logits = logits.float()
|
| 1191 |
+
|
| 1192 |
+
loss = None
|
| 1193 |
+
if labels is not None:
|
| 1194 |
+
# Shift so that tokens < n predict n
|
| 1195 |
+
shift_logits = logits[..., :-1, :].contiguous()
|
| 1196 |
+
shift_labels = labels[..., 1:].contiguous()
|
| 1197 |
+
# Flatten the tokens
|
| 1198 |
+
loss_fct = CrossEntropyLoss()
|
| 1199 |
+
shift_logits = shift_logits.view(-1, self.config.vocab_size)
|
| 1200 |
+
shift_labels = shift_labels.view(-1)
|
| 1201 |
+
# Enable model parallelism
|
| 1202 |
+
shift_labels = shift_labels.to(shift_logits.device)
|
| 1203 |
+
loss = loss_fct(shift_logits, shift_labels)
|
| 1204 |
+
|
| 1205 |
+
if not return_dict:
|
| 1206 |
+
output = (logits,) + outputs[1:]
|
| 1207 |
+
return (loss,) + output if loss is not None else output
|
| 1208 |
+
|
| 1209 |
+
return CausalLMOutputWithPast(
|
| 1210 |
+
loss=loss,
|
| 1211 |
+
logits=logits,
|
| 1212 |
+
past_key_values=outputs.past_key_values,
|
| 1213 |
+
hidden_states=outputs.hidden_states,
|
| 1214 |
+
attentions=outputs.attentions,
|
| 1215 |
+
)
|
| 1216 |
+
|
| 1217 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaForCausalLM.prepare_inputs_for_generation
|
| 1218 |
+
def prepare_inputs_for_generation(
|
| 1219 |
+
self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs
|
| 1220 |
+
):
|
| 1221 |
+
if past_key_values is not None:
|
| 1222 |
+
if isinstance(past_key_values, Cache):
|
| 1223 |
+
cache_length = past_key_values.get_seq_length()
|
| 1224 |
+
past_length = past_key_values.seen_tokens
|
| 1225 |
+
max_cache_length = past_key_values.get_max_length()
|
| 1226 |
+
else:
|
| 1227 |
+
cache_length = past_length = past_key_values[0][0].shape[2]
|
| 1228 |
+
max_cache_length = None
|
| 1229 |
+
|
| 1230 |
+
# Keep only the unprocessed tokens:
|
| 1231 |
+
# 1 - If the length of the attention_mask exceeds the length of input_ids, then we are in a setting where
|
| 1232 |
+
# some of the inputs are exclusively passed as part of the cache (e.g. when passing input_embeds as
|
| 1233 |
+
# input)
|
| 1234 |
+
if attention_mask is not None and attention_mask.shape[1] > input_ids.shape[1]:
|
| 1235 |
+
input_ids = input_ids[:, -(attention_mask.shape[1] - past_length):]
|
| 1236 |
+
# 2 - If the past_length is smaller than input_ids', then input_ids holds all input tokens. We can discard
|
| 1237 |
+
# input_ids based on the past_length.
|
| 1238 |
+
elif past_length < input_ids.shape[1]:
|
| 1239 |
+
input_ids = input_ids[:, past_length:]
|
| 1240 |
+
# 3 - Otherwise (past_length >= input_ids.shape[1]), let's assume input_ids only has unprocessed tokens.
|
| 1241 |
+
|
| 1242 |
+
# If we are about to go beyond the maximum cache length, we need to crop the input attention mask.
|
| 1243 |
+
if (
|
| 1244 |
+
max_cache_length is not None
|
| 1245 |
+
and attention_mask is not None
|
| 1246 |
+
and cache_length + input_ids.shape[1] > max_cache_length
|
| 1247 |
+
):
|
| 1248 |
+
attention_mask = attention_mask[:, -max_cache_length:]
|
| 1249 |
+
|
| 1250 |
+
position_ids = kwargs.get("position_ids", None)
|
| 1251 |
+
if attention_mask is not None and position_ids is None:
|
| 1252 |
+
# create position_ids on the fly for batch generation
|
| 1253 |
+
position_ids = attention_mask.long().cumsum(-1) - 1
|
| 1254 |
+
position_ids.masked_fill_(attention_mask == 0, 1)
|
| 1255 |
+
if past_key_values:
|
| 1256 |
+
position_ids = position_ids[:, -input_ids.shape[1]:]
|
| 1257 |
+
|
| 1258 |
+
# if `inputs_embeds` are passed, we only want to use them in the 1st generation step
|
| 1259 |
+
if inputs_embeds is not None and past_key_values is None:
|
| 1260 |
+
model_inputs = {"inputs_embeds": inputs_embeds}
|
| 1261 |
+
else:
|
| 1262 |
+
model_inputs = {"input_ids": input_ids}
|
| 1263 |
+
|
| 1264 |
+
model_inputs.update(
|
| 1265 |
+
{
|
| 1266 |
+
"position_ids": position_ids,
|
| 1267 |
+
"past_key_values": past_key_values,
|
| 1268 |
+
"use_cache": kwargs.get("use_cache"),
|
| 1269 |
+
"attention_mask": attention_mask,
|
| 1270 |
+
}
|
| 1271 |
+
)
|
| 1272 |
+
return model_inputs
|
| 1273 |
+
|
| 1274 |
+
@staticmethod
|
| 1275 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaForCausalLM._reorder_cache
|
| 1276 |
+
def _reorder_cache(past_key_values, beam_idx):
|
| 1277 |
+
reordered_past = ()
|
| 1278 |
+
for layer_past in past_key_values:
|
| 1279 |
+
reordered_past += (
|
| 1280 |
+
tuple(past_state.index_select(0, beam_idx.to(past_state.device)) for past_state in layer_past),
|
| 1281 |
+
)
|
| 1282 |
+
return reordered_past
|
| 1283 |
+
|
| 1284 |
+
|
| 1285 |
+
@add_start_docstrings(
|
| 1286 |
+
"""
|
| 1287 |
+
The QuasarModel with a sequence classification head on top (linear layer).
|
| 1288 |
+
[`QuasarForSequenceClassification`] uses the last token in order to do the classification, as other causal models
|
| 1289 |
+
(e.g. GPT-2) do.
|
| 1290 |
+
Since it does classification on the last token, it requires to know the position of the last token. If a
|
| 1291 |
+
`pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If
|
| 1292 |
+
no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the
|
| 1293 |
+
padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in
|
| 1294 |
+
each row of the batch).
|
| 1295 |
+
""",
|
| 1296 |
+
QUASAR_START_DOCSTRING,
|
| 1297 |
+
)
|
| 1298 |
+
# Copied from transformers.models.llama.modeling_llama.LlamaForSequenceClassification with LLAMA->QUASAR,Llama->Quasar with self.transformer->self.model, transformer_outputs->model_outputs
|
| 1299 |
+
class QuasarForSequenceClassification(QuasarPreTrainedModel):
|
| 1300 |
+
def __init__(self, config):
|
| 1301 |
+
super().__init__(config)
|
| 1302 |
+
self.num_labels = config.num_labels
|
| 1303 |
+
self.model = QuasarModel(config)
|
| 1304 |
+
self.score = nn.Linear(config.hidden_size, self.num_labels, bias=False)
|
| 1305 |
+
|
| 1306 |
+
# Initialize weights and apply final processing
|
| 1307 |
+
self.post_init()
|
| 1308 |
+
|
| 1309 |
+
def get_input_embeddings(self):
|
| 1310 |
+
return self.model.embed_tokens
|
| 1311 |
+
|
| 1312 |
+
def set_input_embeddings(self, value):
|
| 1313 |
+
self.model.embed_tokens = value
|
| 1314 |
+
|
| 1315 |
+
@add_start_docstrings_to_model_forward(QUASAR_INPUTS_DOCSTRING)
|
| 1316 |
+
def forward(
|
| 1317 |
+
self,
|
| 1318 |
+
input_ids: torch.LongTensor = None,
|
| 1319 |
+
attention_mask: Optional[torch.Tensor] = None,
|
| 1320 |
+
position_ids: Optional[torch.LongTensor] = None,
|
| 1321 |
+
past_key_values: Optional[List[torch.FloatTensor]] = None,
|
| 1322 |
+
inputs_embeds: Optional[torch.FloatTensor] = None,
|
| 1323 |
+
labels: Optional[torch.LongTensor] = None,
|
| 1324 |
+
use_cache: Optional[bool] = None,
|
| 1325 |
+
output_attentions: Optional[bool] = None,
|
| 1326 |
+
output_hidden_states: Optional[bool] = None,
|
| 1327 |
+
return_dict: Optional[bool] = None,
|
| 1328 |
+
) -> Union[Tuple, SequenceClassifierOutputWithPast]:
|
| 1329 |
+
r"""
|
| 1330 |
+
labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
|
| 1331 |
+
Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
|
| 1332 |
+
config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
|
| 1333 |
+
`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
|
| 1334 |
+
"""
|
| 1335 |
+
return_dict = return_dict if return_dict is not None else self.config.use_return_dict
|
| 1336 |
+
|
| 1337 |
+
model_outputs = self.model(
|
| 1338 |
+
input_ids,
|
| 1339 |
+
attention_mask=attention_mask,
|
| 1340 |
+
position_ids=position_ids,
|
| 1341 |
+
past_key_values=past_key_values,
|
| 1342 |
+
inputs_embeds=inputs_embeds,
|
| 1343 |
+
use_cache=use_cache,
|
| 1344 |
+
output_attentions=output_attentions,
|
| 1345 |
+
output_hidden_states=output_hidden_states,
|
| 1346 |
+
return_dict=return_dict,
|
| 1347 |
+
)
|
| 1348 |
+
hidden_states = model_outputs[0]
|
| 1349 |
+
logits = self.score(hidden_states)
|
| 1350 |
+
|
| 1351 |
+
if input_ids is not None:
|
| 1352 |
+
batch_size = input_ids.shape[0]
|
| 1353 |
+
else:
|
| 1354 |
+
batch_size = inputs_embeds.shape[0]
|
| 1355 |
+
|
| 1356 |
+
if self.config.pad_token_id is None and batch_size != 1:
|
| 1357 |
+
raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.")
|
| 1358 |
+
if self.config.pad_token_id is None:
|
| 1359 |
+
sequence_lengths = -1
|
| 1360 |
+
else:
|
| 1361 |
+
if input_ids is not None:
|
| 1362 |
+
# if no pad token found, use modulo instead of reverse indexing for ONNX compatibility
|
| 1363 |
+
sequence_lengths = torch.eq(input_ids, self.config.pad_token_id).int().argmax(-1) - 1
|
| 1364 |
+
sequence_lengths = sequence_lengths % input_ids.shape[-1]
|
| 1365 |
+
sequence_lengths = sequence_lengths.to(logits.device)
|
| 1366 |
+
else:
|
| 1367 |
+
sequence_lengths = -1
|
| 1368 |
+
|
| 1369 |
+
pooled_logits = logits[torch.arange(batch_size, device=logits.device), sequence_lengths]
|
| 1370 |
+
|
| 1371 |
+
loss = None
|
| 1372 |
+
if labels is not None:
|
| 1373 |
+
labels = labels.to(logits.device)
|
| 1374 |
+
if self.config.problem_type is None:
|
| 1375 |
+
if self.num_labels == 1:
|
| 1376 |
+
self.config.problem_type = "regression"
|
| 1377 |
+
elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
|
| 1378 |
+
self.config.problem_type = "single_label_classification"
|
| 1379 |
+
else:
|
| 1380 |
+
self.config.problem_type = "multi_label_classification"
|
| 1381 |
+
|
| 1382 |
+
if self.config.problem_type == "regression":
|
| 1383 |
+
loss_fct = MSELoss()
|
| 1384 |
+
if self.num_labels == 1:
|
| 1385 |
+
loss = loss_fct(pooled_logits.squeeze(), labels.squeeze())
|
| 1386 |
+
else:
|
| 1387 |
+
loss = loss_fct(pooled_logits, labels)
|
| 1388 |
+
elif self.config.problem_type == "single_label_classification":
|
| 1389 |
+
loss_fct = CrossEntropyLoss()
|
| 1390 |
+
loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1))
|
| 1391 |
+
elif self.config.problem_type == "multi_label_classification":
|
| 1392 |
+
loss_fct = BCEWithLogitsLoss()
|
| 1393 |
+
loss = loss_fct(pooled_logits, labels)
|
| 1394 |
+
if not return_dict:
|
| 1395 |
+
output = (pooled_logits,) + model_outputs[1:]
|
| 1396 |
+
return ((loss,) + output) if loss is not None else output
|
| 1397 |
+
|
| 1398 |
+
return SequenceClassifierOutputWithPast(
|
| 1399 |
+
loss=loss,
|
| 1400 |
+
logits=pooled_logits,
|
| 1401 |
+
past_key_values=model_outputs.past_key_values,
|
| 1402 |
+
hidden_states=model_outputs.hidden_states,
|
| 1403 |
+
attentions=model_outputs.attentions,
|
| 1404 |
+
)
|
| 1405 |
+
|
| 1406 |
+
|
| 1407 |
+
@add_start_docstrings(
|
| 1408 |
+
"""
|
| 1409 |
+
QuasarModel with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for
|
| 1410 |
+
Named-Entity-Recognition (NER) tasks.
|
| 1411 |
+
""",
|
| 1412 |
+
QUASAR_START_DOCSTRING,
|
| 1413 |
+
)
|
| 1414 |
+
# Copied from transformers.models.mpt.modeling_mpt.MptForTokenClassification with MPT->QUASAR,Mpt->Quasar,self.transformer->self.model,transformer_outputs->model_outputs
|
| 1415 |
+
class QuasarForTokenClassification(QuasarPreTrainedModel):
|
| 1416 |
+
def __init__(self, config: QuasarConfig):
|
| 1417 |
+
super().__init__(config)
|
| 1418 |
+
self.num_labels = config.num_labels
|
| 1419 |
+
|
| 1420 |
+
self.model = QuasarModel(config)
|
| 1421 |
+
if hasattr(config, "classifier_dropout") and config.classifier_dropout is not None:
|
| 1422 |
+
classifier_dropout = config.classifier_dropout
|
| 1423 |
+
elif hasattr(config, "hidden_dropout") and config.hidden_dropout is not None:
|
| 1424 |
+
classifier_dropout = config.hidden_dropout
|
| 1425 |
+
else:
|
| 1426 |
+
classifier_dropout = 0.1
|
| 1427 |
+
self.dropout = nn.Dropout(classifier_dropout)
|
| 1428 |
+
self.classifier = nn.Linear(config.hidden_size, config.num_labels)
|
| 1429 |
+
|
| 1430 |
+
# Initialize weights and apply final processing
|
| 1431 |
+
self.post_init()
|
| 1432 |
+
|
| 1433 |
+
@add_start_docstrings_to_model_forward(QUASAR_INPUTS_DOCSTRING)
|
| 1434 |
+
@add_code_sample_docstrings(
|
| 1435 |
+
checkpoint=_CHECKPOINT_FOR_DOC,
|
| 1436 |
+
output_type=TokenClassifierOutput,
|
| 1437 |
+
config_class=_CONFIG_FOR_DOC,
|
| 1438 |
+
)
|
| 1439 |
+
def forward(
|
| 1440 |
+
self,
|
| 1441 |
+
input_ids: Optional[torch.LongTensor] = None,
|
| 1442 |
+
past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
|
| 1443 |
+
attention_mask: Optional[torch.Tensor] = None,
|
| 1444 |
+
inputs_embeds: Optional[torch.Tensor] = None,
|
| 1445 |
+
labels: Optional[torch.Tensor] = None,
|
| 1446 |
+
use_cache: Optional[bool] = None,
|
| 1447 |
+
output_attentions: Optional[bool] = None,
|
| 1448 |
+
output_hidden_states: Optional[bool] = None,
|
| 1449 |
+
return_dict: Optional[bool] = None,
|
| 1450 |
+
**deprecated_arguments,
|
| 1451 |
+
) -> Union[Tuple[torch.Tensor], TokenClassifierOutput]:
|
| 1452 |
+
r"""
|
| 1453 |
+
labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
|
| 1454 |
+
Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
|
| 1455 |
+
config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
|
| 1456 |
+
`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
|
| 1457 |
+
"""
|
| 1458 |
+
return_dict = return_dict if return_dict is not None else self.config.use_return_dict
|
| 1459 |
+
|
| 1460 |
+
model_outputs = self.model(
|
| 1461 |
+
input_ids,
|
| 1462 |
+
past_key_values=past_key_values,
|
| 1463 |
+
attention_mask=attention_mask,
|
| 1464 |
+
inputs_embeds=inputs_embeds,
|
| 1465 |
+
use_cache=use_cache,
|
| 1466 |
+
output_attentions=output_attentions,
|
| 1467 |
+
output_hidden_states=output_hidden_states,
|
| 1468 |
+
return_dict=return_dict,
|
| 1469 |
+
)
|
| 1470 |
+
|
| 1471 |
+
hidden_states = model_outputs[0]
|
| 1472 |
+
hidden_states = self.dropout(hidden_states)
|
| 1473 |
+
logits = self.classifier(hidden_states)
|
| 1474 |
+
|
| 1475 |
+
loss = None
|
| 1476 |
+
if labels is not None:
|
| 1477 |
+
# move labels to correct device to enable model parallelism
|
| 1478 |
+
labels = labels.to(logits.device)
|
| 1479 |
+
batch_size, seq_length = labels.shape
|
| 1480 |
+
loss_fct = CrossEntropyLoss()
|
| 1481 |
+
loss = loss_fct(
|
| 1482 |
+
logits.view(batch_size * seq_length, self.num_labels), labels.view(batch_size * seq_length)
|
| 1483 |
+
)
|
| 1484 |
+
|
| 1485 |
+
if not return_dict:
|
| 1486 |
+
output = (logits,) + model_outputs[2:]
|
| 1487 |
+
return ((loss,) + output) if loss is not None else output
|
| 1488 |
+
|
| 1489 |
+
return TokenClassifierOutput(
|
| 1490 |
+
loss=loss,
|
| 1491 |
+
logits=logits,
|
| 1492 |
+
hidden_states=model_outputs.hidden_states,
|
| 1493 |
+
attentions=model_outputs.attentions,
|
| 1494 |
+
)
|
special_tokens_map.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token": {
|
| 3 |
+
"content": "<|endoftext|>",
|
| 4 |
+
"lstrip": false,
|
| 5 |
+
"normalized": false,
|
| 6 |
+
"rstrip": false,
|
| 7 |
+
"single_word": false
|
| 8 |
+
},
|
| 9 |
+
"eos_token": {
|
| 10 |
+
"content": "<|endoftext|>",
|
| 11 |
+
"lstrip": false,
|
| 12 |
+
"normalized": false,
|
| 13 |
+
"rstrip": false,
|
| 14 |
+
"single_word": false
|
| 15 |
+
},
|
| 16 |
+
"unk_token": {
|
| 17 |
+
"content": "<|endoftext|>",
|
| 18 |
+
"lstrip": false,
|
| 19 |
+
"normalized": false,
|
| 20 |
+
"rstrip": false,
|
| 21 |
+
"single_word": false
|
| 22 |
+
}
|
| 23 |
+
}
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,323 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_prefix_space": false,
|
| 3 |
+
"added_tokens_decoder": {
|
| 4 |
+
"50256": {
|
| 5 |
+
"content": "<|endoftext|>",
|
| 6 |
+
"lstrip": false,
|
| 7 |
+
"normalized": false,
|
| 8 |
+
"rstrip": false,
|
| 9 |
+
"single_word": false,
|
| 10 |
+
"special": true
|
| 11 |
+
},
|
| 12 |
+
"50257": {
|
| 13 |
+
"content": " ",
|
| 14 |
+
"lstrip": false,
|
| 15 |
+
"normalized": true,
|
| 16 |
+
"rstrip": false,
|
| 17 |
+
"single_word": false,
|
| 18 |
+
"special": false
|
| 19 |
+
},
|
| 20 |
+
"50258": {
|
| 21 |
+
"content": " ",
|
| 22 |
+
"lstrip": false,
|
| 23 |
+
"normalized": true,
|
| 24 |
+
"rstrip": false,
|
| 25 |
+
"single_word": false,
|
| 26 |
+
"special": false
|
| 27 |
+
},
|
| 28 |
+
"50259": {
|
| 29 |
+
"content": " ",
|
| 30 |
+
"lstrip": false,
|
| 31 |
+
"normalized": true,
|
| 32 |
+
"rstrip": false,
|
| 33 |
+
"single_word": false,
|
| 34 |
+
"special": false
|
| 35 |
+
},
|
| 36 |
+
"50260": {
|
| 37 |
+
"content": " ",
|
| 38 |
+
"lstrip": false,
|
| 39 |
+
"normalized": true,
|
| 40 |
+
"rstrip": false,
|
| 41 |
+
"single_word": false,
|
| 42 |
+
"special": false
|
| 43 |
+
},
|
| 44 |
+
"50261": {
|
| 45 |
+
"content": " ",
|
| 46 |
+
"lstrip": false,
|
| 47 |
+
"normalized": true,
|
| 48 |
+
"rstrip": false,
|
| 49 |
+
"single_word": false,
|
| 50 |
+
"special": false
|
| 51 |
+
},
|
| 52 |
+
"50262": {
|
| 53 |
+
"content": " ",
|
| 54 |
+
"lstrip": false,
|
| 55 |
+
"normalized": true,
|
| 56 |
+
"rstrip": false,
|
| 57 |
+
"single_word": false,
|
| 58 |
+
"special": false
|
| 59 |
+
},
|
| 60 |
+
"50263": {
|
| 61 |
+
"content": " ",
|
| 62 |
+
"lstrip": false,
|
| 63 |
+
"normalized": true,
|
| 64 |
+
"rstrip": false,
|
| 65 |
+
"single_word": false,
|
| 66 |
+
"special": false
|
| 67 |
+
},
|
| 68 |
+
"50264": {
|
| 69 |
+
"content": " ",
|
| 70 |
+
"lstrip": false,
|
| 71 |
+
"normalized": true,
|
| 72 |
+
"rstrip": false,
|
| 73 |
+
"single_word": false,
|
| 74 |
+
"special": false
|
| 75 |
+
},
|
| 76 |
+
"50265": {
|
| 77 |
+
"content": " ",
|
| 78 |
+
"lstrip": false,
|
| 79 |
+
"normalized": true,
|
| 80 |
+
"rstrip": false,
|
| 81 |
+
"single_word": false,
|
| 82 |
+
"special": false
|
| 83 |
+
},
|
| 84 |
+
"50266": {
|
| 85 |
+
"content": " ",
|
| 86 |
+
"lstrip": false,
|
| 87 |
+
"normalized": true,
|
| 88 |
+
"rstrip": false,
|
| 89 |
+
"single_word": false,
|
| 90 |
+
"special": false
|
| 91 |
+
},
|
| 92 |
+
"50267": {
|
| 93 |
+
"content": " ",
|
| 94 |
+
"lstrip": false,
|
| 95 |
+
"normalized": true,
|
| 96 |
+
"rstrip": false,
|
| 97 |
+
"single_word": false,
|
| 98 |
+
"special": false
|
| 99 |
+
},
|
| 100 |
+
"50268": {
|
| 101 |
+
"content": " ",
|
| 102 |
+
"lstrip": false,
|
| 103 |
+
"normalized": true,
|
| 104 |
+
"rstrip": false,
|
| 105 |
+
"single_word": false,
|
| 106 |
+
"special": false
|
| 107 |
+
},
|
| 108 |
+
"50269": {
|
| 109 |
+
"content": " ",
|
| 110 |
+
"lstrip": false,
|
| 111 |
+
"normalized": true,
|
| 112 |
+
"rstrip": false,
|
| 113 |
+
"single_word": false,
|
| 114 |
+
"special": false
|
| 115 |
+
},
|
| 116 |
+
"50270": {
|
| 117 |
+
"content": " ",
|
| 118 |
+
"lstrip": false,
|
| 119 |
+
"normalized": true,
|
| 120 |
+
"rstrip": false,
|
| 121 |
+
"single_word": false,
|
| 122 |
+
"special": false
|
| 123 |
+
},
|
| 124 |
+
"50271": {
|
| 125 |
+
"content": " ",
|
| 126 |
+
"lstrip": false,
|
| 127 |
+
"normalized": true,
|
| 128 |
+
"rstrip": false,
|
| 129 |
+
"single_word": false,
|
| 130 |
+
"special": false
|
| 131 |
+
},
|
| 132 |
+
"50272": {
|
| 133 |
+
"content": " ",
|
| 134 |
+
"lstrip": false,
|
| 135 |
+
"normalized": true,
|
| 136 |
+
"rstrip": false,
|
| 137 |
+
"single_word": false,
|
| 138 |
+
"special": false
|
| 139 |
+
},
|
| 140 |
+
"50273": {
|
| 141 |
+
"content": " ",
|
| 142 |
+
"lstrip": false,
|
| 143 |
+
"normalized": true,
|
| 144 |
+
"rstrip": false,
|
| 145 |
+
"single_word": false,
|
| 146 |
+
"special": false
|
| 147 |
+
},
|
| 148 |
+
"50274": {
|
| 149 |
+
"content": " ",
|
| 150 |
+
"lstrip": false,
|
| 151 |
+
"normalized": true,
|
| 152 |
+
"rstrip": false,
|
| 153 |
+
"single_word": false,
|
| 154 |
+
"special": false
|
| 155 |
+
},
|
| 156 |
+
"50275": {
|
| 157 |
+
"content": " ",
|
| 158 |
+
"lstrip": false,
|
| 159 |
+
"normalized": true,
|
| 160 |
+
"rstrip": false,
|
| 161 |
+
"single_word": false,
|
| 162 |
+
"special": false
|
| 163 |
+
},
|
| 164 |
+
"50276": {
|
| 165 |
+
"content": " ",
|
| 166 |
+
"lstrip": false,
|
| 167 |
+
"normalized": true,
|
| 168 |
+
"rstrip": false,
|
| 169 |
+
"single_word": false,
|
| 170 |
+
"special": false
|
| 171 |
+
},
|
| 172 |
+
"50277": {
|
| 173 |
+
"content": " ",
|
| 174 |
+
"lstrip": false,
|
| 175 |
+
"normalized": true,
|
| 176 |
+
"rstrip": false,
|
| 177 |
+
"single_word": false,
|
| 178 |
+
"special": false
|
| 179 |
+
},
|
| 180 |
+
"50278": {
|
| 181 |
+
"content": " ",
|
| 182 |
+
"lstrip": false,
|
| 183 |
+
"normalized": true,
|
| 184 |
+
"rstrip": false,
|
| 185 |
+
"single_word": false,
|
| 186 |
+
"special": false
|
| 187 |
+
},
|
| 188 |
+
"50279": {
|
| 189 |
+
"content": " ",
|
| 190 |
+
"lstrip": false,
|
| 191 |
+
"normalized": true,
|
| 192 |
+
"rstrip": false,
|
| 193 |
+
"single_word": false,
|
| 194 |
+
"special": false
|
| 195 |
+
},
|
| 196 |
+
"50280": {
|
| 197 |
+
"content": " ",
|
| 198 |
+
"lstrip": false,
|
| 199 |
+
"normalized": true,
|
| 200 |
+
"rstrip": false,
|
| 201 |
+
"single_word": false,
|
| 202 |
+
"special": false
|
| 203 |
+
},
|
| 204 |
+
"50281": {
|
| 205 |
+
"content": " ",
|
| 206 |
+
"lstrip": false,
|
| 207 |
+
"normalized": true,
|
| 208 |
+
"rstrip": false,
|
| 209 |
+
"single_word": false,
|
| 210 |
+
"special": false
|
| 211 |
+
},
|
| 212 |
+
"50282": {
|
| 213 |
+
"content": " ",
|
| 214 |
+
"lstrip": false,
|
| 215 |
+
"normalized": true,
|
| 216 |
+
"rstrip": false,
|
| 217 |
+
"single_word": false,
|
| 218 |
+
"special": false
|
| 219 |
+
},
|
| 220 |
+
"50283": {
|
| 221 |
+
"content": " ",
|
| 222 |
+
"lstrip": false,
|
| 223 |
+
"normalized": true,
|
| 224 |
+
"rstrip": false,
|
| 225 |
+
"single_word": false,
|
| 226 |
+
"special": false
|
| 227 |
+
},
|
| 228 |
+
"50284": {
|
| 229 |
+
"content": " ",
|
| 230 |
+
"lstrip": false,
|
| 231 |
+
"normalized": true,
|
| 232 |
+
"rstrip": false,
|
| 233 |
+
"single_word": false,
|
| 234 |
+
"special": false
|
| 235 |
+
},
|
| 236 |
+
"50285": {
|
| 237 |
+
"content": " ",
|
| 238 |
+
"lstrip": false,
|
| 239 |
+
"normalized": true,
|
| 240 |
+
"rstrip": false,
|
| 241 |
+
"single_word": false,
|
| 242 |
+
"special": false
|
| 243 |
+
},
|
| 244 |
+
"50286": {
|
| 245 |
+
"content": " ",
|
| 246 |
+
"lstrip": false,
|
| 247 |
+
"normalized": true,
|
| 248 |
+
"rstrip": false,
|
| 249 |
+
"single_word": false,
|
| 250 |
+
"special": false
|
| 251 |
+
},
|
| 252 |
+
"50287": {
|
| 253 |
+
"content": "\t\t\t\t\t\t\t\t\t",
|
| 254 |
+
"lstrip": false,
|
| 255 |
+
"normalized": true,
|
| 256 |
+
"rstrip": false,
|
| 257 |
+
"single_word": false,
|
| 258 |
+
"special": false
|
| 259 |
+
},
|
| 260 |
+
"50288": {
|
| 261 |
+
"content": "\t\t\t\t\t\t\t\t",
|
| 262 |
+
"lstrip": false,
|
| 263 |
+
"normalized": true,
|
| 264 |
+
"rstrip": false,
|
| 265 |
+
"single_word": false,
|
| 266 |
+
"special": false
|
| 267 |
+
},
|
| 268 |
+
"50289": {
|
| 269 |
+
"content": "\t\t\t\t\t\t\t",
|
| 270 |
+
"lstrip": false,
|
| 271 |
+
"normalized": true,
|
| 272 |
+
"rstrip": false,
|
| 273 |
+
"single_word": false,
|
| 274 |
+
"special": false
|
| 275 |
+
},
|
| 276 |
+
"50290": {
|
| 277 |
+
"content": "\t\t\t\t\t\t",
|
| 278 |
+
"lstrip": false,
|
| 279 |
+
"normalized": true,
|
| 280 |
+
"rstrip": false,
|
| 281 |
+
"single_word": false,
|
| 282 |
+
"special": false
|
| 283 |
+
},
|
| 284 |
+
"50291": {
|
| 285 |
+
"content": "\t\t\t\t\t",
|
| 286 |
+
"lstrip": false,
|
| 287 |
+
"normalized": true,
|
| 288 |
+
"rstrip": false,
|
| 289 |
+
"single_word": false,
|
| 290 |
+
"special": false
|
| 291 |
+
},
|
| 292 |
+
"50292": {
|
| 293 |
+
"content": "\t\t\t\t",
|
| 294 |
+
"lstrip": false,
|
| 295 |
+
"normalized": true,
|
| 296 |
+
"rstrip": false,
|
| 297 |
+
"single_word": false,
|
| 298 |
+
"special": false
|
| 299 |
+
},
|
| 300 |
+
"50293": {
|
| 301 |
+
"content": "\t\t\t",
|
| 302 |
+
"lstrip": false,
|
| 303 |
+
"normalized": true,
|
| 304 |
+
"rstrip": false,
|
| 305 |
+
"single_word": false,
|
| 306 |
+
"special": false
|
| 307 |
+
},
|
| 308 |
+
"50294": {
|
| 309 |
+
"content": "\t\t",
|
| 310 |
+
"lstrip": false,
|
| 311 |
+
"normalized": true,
|
| 312 |
+
"rstrip": false,
|
| 313 |
+
"single_word": false,
|
| 314 |
+
"special": false
|
| 315 |
+
}
|
| 316 |
+
},
|
| 317 |
+
"bos_token": "<|endoftext|>",
|
| 318 |
+
"clean_up_tokenization_spaces": true,
|
| 319 |
+
"eos_token": "<|endoftext|>",
|
| 320 |
+
"model_max_length": 32768,
|
| 321 |
+
"tokenizer_class": "CodeGenTokenizer",
|
| 322 |
+
"unk_token": "<|endoftext|>"
|
| 323 |
+
}
|
vocab.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|