Meta

LICENSE

@@ -0,0 +1,373 @@
+Mozilla Public License Version 2.0
+==================================
+
+1. Definitions
+--------------
+
+1.1. "Contributor"
+    means each individual or legal entity that creates, contributes to
+    the creation of, or owns Covered Software.
+
+1.2. "Contributor Version"
+    means the combination of the Contributions of others (if any) used
+    by a Contributor and that particular Contributor's Contribution.
+
+1.3. "Contribution"
+    means Covered Software of a particular Contributor.
+
+1.4. "Covered Software"
+    means Source Code Form to which the initial Contributor has attached
+    the notice in Exhibit A, the Executable Form of such Source Code
+    Form, and Modifications of such Source Code Form, in each case
+    including portions thereof.
+
+1.5. "Incompatible With Secondary Licenses"
+    means
+
+    (a) that the initial Contributor has attached the notice described
+        in Exhibit B to the Covered Software; or
+
+    (b) that the Covered Software was made available under the terms of
+        version 1.1 or earlier of the License, but not also under the
+        terms of a Secondary License.
+
+1.6. "Executable Form"
+    means any form of the work other than Source Code Form.
+
+1.7. "Larger Work"
+    means a work that combines Covered Software with other material, in
+    a separate file or files, that is not Covered Software.
+
+1.8. "License"
+    means this document.
+
+1.9. "Licensable"
+    means having the right to grant, to the maximum extent possible,
+    whether at the time of the initial grant or subsequently, any and
+    all of the rights conveyed by this License.
+
+1.10. "Modifications"
+    means any of the following:
+
+    (a) any file in Source Code Form that results from an addition to,
+        deletion from, or modification of the contents of Covered
+        Software; or
+
+    (b) any new file in Source Code Form that contains any Covered
+        Software.
+
+1.11. "Patent Claims" of a Contributor
+    means any patent claim(s), including without limitation, method,
+    process, and apparatus claims, in any patent Licensable by such
+    Contributor that would be infringed, but for the grant of the
+    License, by the making, using, selling, offering for sale, having
+    made, import, or transfer of either its Contributions or its
+    Contributor Version.
+
+1.12. "Secondary License"
+    means either the GNU General Public License, Version 2.0, the GNU
+    Lesser General Public License, Version 2.1, the GNU Affero General
+    Public License, Version 3.0, or any later versions of those
+    licenses.
+
+1.13. "Source Code Form"
+    means the form of the work preferred for making modifications.
+
+1.14. "You" (or "Your")
+    means an individual or a legal entity exercising rights under this
+    License. For legal entities, "You" includes any entity that
+    controls, is controlled by, or is under common control with You. For
+    purposes of this definition, "control" means (a) the power, direct
+    or indirect, to cause the direction or management of such entity,
+    whether by contract or otherwise, or (b) ownership of more than
+    fifty percent (50%) of the outstanding shares or beneficial
+    ownership of such entity.
+
+2. License Grants and Conditions
+--------------------------------
+
+2.1. Grants
+
+Each Contributor hereby grants You a world-wide, royalty-free,
+non-exclusive license:
+
+(a) under intellectual property rights (other than patent or trademark)
+    Licensable by such Contributor to use, reproduce, make available,
+    modify, display, perform, distribute, and otherwise exploit its
+    Contributions, either on an unmodified basis, with Modifications, or
+    as part of a Larger Work; and
+
+(b) under Patent Claims of such Contributor to make, use, sell, offer
+    for sale, have made, import, and otherwise transfer either its
+    Contributions or its Contributor Version.
+
+2.2. Effective Date
+
+The licenses granted in Section 2.1 with respect to any Contribution
+become effective for each Contribution on the date the Contributor first
+distributes such Contribution.
+
+2.3. Limitations on Grant Scope
+
+The licenses granted in this Section 2 are the only rights granted under
+this License. No additional rights or licenses will be implied from the
+distribution or licensing of Covered Software under this License.
+Notwithstanding Section 2.1(b) above, no patent license is granted by a
+Contributor:
+
+(a) for any code that a Contributor has removed from Covered Software;
+    or
+
+(b) for infringements caused by: (i) Your and any other third party's
+    modifications of Covered Software, or (ii) the combination of its
+    Contributions with other software (except as part of its Contributor
+    Version); or
+
+(c) under Patent Claims infringed by Covered Software in the absence of
+    its Contributions.
+
+This License does not grant any rights in the trademarks, service marks,
+or logos of any Contributor (except as may be necessary to comply with
+the notice requirements in Section 3.4).
+
+2.4. Subsequent Licenses
+
+No Contributor makes additional grants as a result of Your choice to
+distribute the Covered Software under a subsequent version of this
+License (see Section 10.2) or under the terms of a Secondary License (if
+permitted under the terms of Section 3.3).
+
+2.5. Representation
+
+Each Contributor represents that the Contributor believes its
+Contributions are its original creation(s) or it has sufficient rights
+to grant the rights to its Contributions conveyed by this License.
+
+2.6. Fair Use
+
+This License is not intended to limit any rights You have under
+applicable copyright doctrines of fair use, fair dealing, or other
+equivalents.
+
+2.7. Conditions
+
+Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
+in Section 2.1.
+
+3. Responsibilities
+-------------------
+
+3.1. Distribution of Source Form
+
+All distribution of Covered Software in Source Code Form, including any
+Modifications that You create or to which You contribute, must be under
+the terms of this License. You must inform recipients that the Source
+Code Form of the Covered Software is governed by the terms of this
+License, and how they can obtain a copy of this License. You may not
+attempt to alter or restrict the recipients' rights in the Source Code
+Form.
+
+3.2. Distribution of Executable Form
+
+If You distribute Covered Software in Executable Form then:
+
+(a) such Covered Software must also be made available in Source Code
+    Form, as described in Section 3.1, and You must inform recipients of
+    the Executable Form how they can obtain a copy of such Source Code
+    Form by reasonable means in a timely manner, at a charge no more
+    than the cost of distribution to the recipient; and
+
+(b) You may distribute such Executable Form under the terms of this
+    License, or sublicense it under different terms, provided that the
+    license for the Executable Form does not attempt to limit or alter
+    the recipients' rights in the Source Code Form under this License.
+
+3.3. Distribution of a Larger Work
+
+You may create and distribute a Larger Work under terms of Your choice,
+provided that You also comply with the requirements of this License for
+the Covered Software. If the Larger Work is a combination of Covered
+Software with a work governed by one or more Secondary Licenses, and the
+Covered Software is not Incompatible With Secondary Licenses, this
+License permits You to additionally distribute such Covered Software
+under the terms of such Secondary License(s), so that the recipient of
+the Larger Work may, at their option, further distribute the Covered
+Software under the terms of either this License or such Secondary
+License(s).
+
+3.4. Notices
+
+You may not remove or alter the substance of any license notices
+(including copyright notices, patent notices, disclaimers of warranty,
+or limitations of liability) contained within the Source Code Form of
+the Covered Software, except that You may alter any license notices to
+the extent required to remedy known factual inaccuracies.
+
+3.5. Application of Additional Terms
+
+You may choose to offer, and to charge a fee for, warranty, support,
+indemnity or liability obligations to one or more recipients of Covered
+Software. However, You may do so only on Your own behalf, and not on
+behalf of any Contributor. You must make it absolutely clear that any
+such warranty, support, indemnity, or liability obligation is offered by
+You alone, and You hereby agree to indemnify every Contributor for any
+liability incurred by such Contributor as a result of warranty, support,
+indemnity or liability terms You offer. You may include additional
+disclaimers of warranty and limitations of liability specific to any
+jurisdiction.
+
+4. Inability to Comply Due to Statute or Regulation
+---------------------------------------------------
+
+If it is impossible for You to comply with any of the terms of this
+License with respect to some or all of the Covered Software due to
+statute, judicial order, or regulation then You must: (a) comply with
+the terms of this License to the maximum extent possible; and (b)
+describe the limitations and the code they affect. Such description must
+be placed in a text file included with all distributions of the Covered
+Software under this License. Except to the extent prohibited by statute
+or regulation, such description must be sufficiently detailed for a
+recipient of ordinary skill to be able to understand it.
+
+5. Termination
+--------------
+
+5.1. The rights granted under this License will terminate automatically
+if You fail to comply with any of its terms. However, if You become
+compliant, then the rights granted under this License from a particular
+Contributor are reinstated (a) provisionally, unless and until such
+Contributor explicitly and finally terminates Your grants, and (b) on an
+ongoing basis, if such Contributor fails to notify You of the
+non-compliance by some reasonable means prior to 60 days after You have
+come back into compliance. Moreover, Your grants from a particular
+Contributor are reinstated on an ongoing basis if such Contributor
+notifies You of the non-compliance by some reasonable means, this is the
+first time You have received notice of non-compliance with this License
+from such Contributor, and You become compliant prior to 30 days after
+Your receipt of the notice.
+
+5.2. If You initiate litigation against any entity by asserting a patent
+infringement claim (excluding declaratory judgment actions,
+counter-claims, and cross-claims) alleging that a Contributor Version
+directly or indirectly infringes any patent, then the rights granted to
+You by any and all Contributors for the Covered Software under Section
+2.1 of this License shall terminate.
+
+5.3. In the event of termination under Sections 5.1 or 5.2 above, all
+end user license agreements (excluding distributors and resellers) which
+have been validly granted by You or Your distributors under this License
+prior to termination shall survive termination.
+
+************************************************************************
+*                                                                      *
+*  6. Disclaimer of Warranty                                           *
+*  -------------------------                                           *
+*                                                                      *
+*  Covered Software is provided under this License on an "as is"       *
+*  basis, without warranty of any kind, either expressed, implied, or  *
+*  statutory, including, without limitation, warranties that the       *
+*  Covered Software is free of defects, merchantable, fit for a        *
+*  particular purpose or non-infringing. The entire risk as to the     *
+*  quality and performance of the Covered Software is with You.        *
+*  Should any Covered Software prove defective in any respect, You     *
+*  (not any Contributor) assume the cost of any necessary servicing,   *
+*  repair, or correction. This disclaimer of warranty constitutes an   *
+*  essential part of this License. No use of any Covered Software is   *
+*  authorized under this License except under this disclaimer.         *
+*                                                                      *
+************************************************************************
+
+************************************************************************
+*                                                                      *
+*  7. Limitation of Liability                                          *
+*  --------------------------                                          *
+*                                                                      *
+*  Under no circumstances and under no legal theory, whether tort      *
+*  (including negligence), contract, or otherwise, shall any           *
+*  Contributor, or anyone who distributes Covered Software as          *
+*  permitted above, be liable to You for any direct, indirect,         *
+*  special, incidental, or consequential damages of any character      *
+*  including, without limitation, damages for lost profits, loss of    *
+*  goodwill, work stoppage, computer failure or malfunction, or any    *
+*  and all other commercial damages or losses, even if such party      *
+*  shall have been informed of the possibility of such damages. This   *
+*  limitation of liability shall not apply to liability for death or   *
+*  personal injury resulting from such party's negligence to the       *
+*  extent applicable law prohibits such limitation. Some               *
+*  jurisdictions do not allow the exclusion or limitation of           *
+*  incidental or consequential damages, so this exclusion and          *
+*  limitation may not apply to You.                                    *
+*                                                                      *
+************************************************************************
+
+8. Litigation
+-------------
+
+Any litigation relating to this License may be brought only in the
+courts of a jurisdiction where the defendant maintains its principal
+place of business and such litigation shall be governed by laws of that
+jurisdiction, without reference to its conflict-of-law provisions.
+Nothing in this Section shall prevent a party's ability to bring
+cross-claims or counter-claims.
+
+9. Miscellaneous
+----------------
+
+This License represents the complete agreement concerning the subject
+matter hereof. If any provision of this License is held to be
+unenforceable, such provision shall be reformed only to the extent
+necessary to make it enforceable. Any law or regulation which provides
+that the language of a contract shall be construed against the drafter
+shall not be used to construe this License against a Contributor.
+
+10. Versions of the License
+---------------------------
+
+10.1. New Versions
+
+William Held, Diyi Yang, Georgia Tech, and Stanford University are the license stewards. Except as provided in Section
+10.3, no one other than the license steward has the right to modify or
+publish new versions of this License. Each version will be given a
+distinguishing version number.
+
+10.2. Effect of New Versions
+
+You may distribute the Covered Software under the terms of the version
+of the License under which You originally received the Covered Software,
+or under the terms of any subsequent version published by the license
+steward.
+
+10.3. Modified Versions
+
+If you create software not governed by this License, and you want to
+create a new license for such software, you may create and use a
+modified version of this License if you rename the license and remove
+any references to the name of the license steward (except to note that
+such modified license differs from this License).
+
+10.4. Distributing Source Code Form that is Incompatible With Secondary
+Licenses
+
+If You choose to distribute Source Code Form that is Incompatible With
+Secondary Licenses under the terms of this version of the License, the
+notice described in Exhibit B of this License must be attached.
+
+Exhibit A - Source Code Form License Notice
+-------------------------------------------
+
+  This Source Code Form is subject to the terms of the Mozilla Public
+  License, v. 2.0. If a copy of the MPL was not distributed with this
+  file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+If it is not possible or desirable to put the notice in a particular
+file, then You may include the notice in a location (such as a LICENSE
+file in a relevant directory) where a recipient would be likely to look
+for such a notice.
+
+You may add additional accurate notices of copyright ownership.
+
+Exhibit B - "Incompatible With Secondary Licenses" Notice
+---------------------------------------------------------
+
+  This Source Code Form is "Incompatible With Secondary Licenses", as
+  defined by the Mozilla Public License, v. 2.0.

README.md

@@ -0,0 +1,76 @@
+# Model Card for Diva Llama 3
+
+<!-- Provide a quick summary of what the model is/does. [Optional] -->
+This is an end-to-end Voice Assistant Model which can handle speech and text as inputs. It is trained using distillation loss. More details will be in a paper [COMING SOON]!
+
+See the model in action compared to SALMONN and Qwen-Audio at [diva-audio.github.io](https://diva-audio.github.io).
+## Citation
+No Publication As of Yet, But If You Use Please Cite the Below
+**BibTeX:**
+
+```
+	@misc{held2024diva,
+	  author="Held, Will and Zhang, Yanzhe and Ryan, Michael and Shi, Weiyan and Li, Ella and Yang, Diyi",
+	  title="Distilling an End-to-End Voice Assistant from Speech Recognition Data",
+	  year="2024",
+	  publisher="HuggingFace",
+	}
+    
+```
+
+##  Table of Contents
+
+- [Model Card for DiVA Llama 3](#model-card-for-DiVA-Llama-3)
+- [Citation](#citation)
+- [Table of Contents](#table-of-contents)
+- [Training Details](#training-details)
+  - [Training Data](#training-data)
+  - [Training Procedure](#training-procedure)
+- [Environmental Impact](#environmental-impact)
+- [Technical Specifications [optional]](#technical-specifications-optional)
+  - [Model Architecture and Objective](#model-architecture-and-objective)
+  - [Compute Infrastructure](#compute-infrastructure)
+    - [Hardware](#hardware)
+    - [Software](#software)
+- [Model Card Contact](#model-card-contact)
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+This model was trained on the [CommonVoice](https://huggingface.co/datasets/mozilla-foundation/common_voice_16_1) corpus.
+
+
+### Training Procedure
+
+This model was trained for 7k gradient steps with a batch size of 512 Recordings and a linearly decaying learning rate from 5e-5 to zero, with a linear warmup of 70 steps.
+
+### Environmental Impact
+
+- **Hardware Type:** V4-32 TPU
+- **Hours used:** 8 Hours
+- **Cloud Provider:** Google Cloud.
+- **Compute Region:** US Central C
+
+
+### Hardware
+
+This model was trained on at V4 TPU on Google Cloud.
+
+### Software
+
+This model was trained with [Levanter](https://github.com/stanford-crfm/levanter) 
+
+
+## Model Card Authors [optional]
+
+<!-- This section provides another layer of transparency and accountability. Whose views is this model card representing? How many voices were included in its construction? Etc. -->
+
+Will Held
+
+## Model Card Contact
+
+[email protected]
+

config.json

@@ -0,0 +1,138 @@
+{
+  "model_type": "diva",
+  "architectures": [ "DiVAModel" ],
+  "auto_map": {
+    "AutoConfig": "configuring_diva.DiVAConfig",
+    "AutoModel": "modeling_diva.DiVAModel"
+  },
+  "vocab_size": 128256,
+  "decoder": {
+    "architectures": [
+      "LlamaForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0,
+    "bos_token_id": 128000,
+    "eos_token_id": 128001,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "initializer_range": 0.02,
+    "intermediate_size": 14336,
+    "max_position_embeddings": 8192,
+    "model_type": "llama",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 8,
+    "pretraining_tp": 1,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": null,
+    "rope_theta": 500000,
+    "tie_word_embeddings": false,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.40.0.dev0",
+    "use_cache": true,
+    "vocab_size": 128256
+  },
+  "encoder": {
+    "_name_or_path": "openai/whisper-large-v3",
+    "activation_dropout": 0,
+    "activation_function": "gelu",
+    "add_cross_attention": false,
+    "apply_spec_augment": false,
+    "architectures": [
+      "WhisperForConditionalGeneration"
+    ],
+    "attention_dropout": 0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": [
+      220,
+      50257
+    ],
+    "bos_token_id": 50257,
+    "chunk_size_feed_forward": 0,
+    "classifier_proj_size": 256,
+    "cross_attention_hidden_size": null,
+    "d_model": 1280,
+    "decoder_attention_heads": 20,
+    "decoder_ffn_dim": 5120,
+    "decoder_layerdrop": 0,
+    "decoder_layers": 32,
+    "decoder_start_token_id": 50258,
+    "diversity_penalty": 0,
+    "do_sample": false,
+    "dropout": 0,
+    "early_stopping": false,
+    "encoder_attention_heads": 20,
+    "encoder_ffn_dim": 5120,
+    "encoder_layerdrop": 0,
+    "encoder_layers": 32,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 50257,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "init_std": 0.02,
+    "is_decoder": false,
+    "is_encoder_decoder": true,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1,
+    "mask_feature_length": 10,
+    "mask_feature_min_masks": 0,
+    "mask_feature_prob": 0,
+    "mask_time_length": 10,
+    "mask_time_min_masks": 2,
+    "mask_time_prob": 0.05,
+    "max_length": 448,
+    "max_source_positions": 1500,
+    "max_target_positions": 448,
+    "median_filter_width": 7,
+    "min_length": 0,
+    "model_type": "whisper",
+    "no_repeat_ngram_size": 0,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 32,
+    "num_mel_bins": 128,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 50256,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "scale_embedding": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1,
+    "torch_dtype": "float16",
+    "torchscript": false,
+    "transformers_version": "4.38.2",
+    "typical_p": 1,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "use_weighted_layer_sum": false,
+    "vocab_size": 51866
+  },
+  "time_dialation": 4,
+  "transformers_version": "4.38.2"
+}

configuring_diva.py

@@ -0,0 +1,11 @@
+from transformers import PretrainedConfig
+
+
+class DiVAConfig(PretrainedConfig):
+    model_type = "diva"
+
+    def __init__(
+        self,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)

model.safetensors.index.json

@@ -0,0 +1,782 @@
+{
+  "metadata": {
+    "total_size": 38310608896
+  },
+  "weight_map": {
+    "query_tokens": "model-00001-of-00004.safetensors",
+    "projection.weight": "model-00001-of-00004.safetensors",
+    "projection.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.self_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.self_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.self_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.self_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.encoder_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.encoder_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.encoder_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.encoder_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.encoder_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.encoder_attn.out_proj.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.encoder_attn.out_proj.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.encoder_attn_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.encoder_attn_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.fc1.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.fc1.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.fc2.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.fc2.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.0.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.1.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.2.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.3.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.4.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.5.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.6.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.7.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.8.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.9.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.10.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.11.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.12.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.13.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.14.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.15.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.16.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.17.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.18.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.19.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.20.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.21.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.22.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.23.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.24.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.25.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.26.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.27.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.28.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.29.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.30.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.31.final_layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layers.0.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.1.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.2.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.3.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.4.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.5.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.6.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.7.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.8.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.9.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.10.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.11.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.12.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.13.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.14.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.15.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.16.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.17.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.18.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.19.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.20.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.21.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.22.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.23.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.24.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.25.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.26.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.27.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.28.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.29.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.30.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layers.31.final_layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.layer_norm.weight": "model-00001-of-00004.safetensors",
+    "connector.layer_norm.bias": "model-00001-of-00004.safetensors",
+    "connector.embed_tokens.weight": "model-00001-of-00004.safetensors",
+    "connector.embed_positions.weight": "model-00001-of-00004.safetensors"
+  }
+}

modeling_diva.py

@@ -0,0 +1,308 @@
+import copy
+import json
+import os
+from typing import Optional, Union
+
+import librosa
+import numpy as np
+import torch
+import torch.nn.functional as F
+from datasets import Audio
+from safetensors.torch import load, load_model
+from torch import nn
+from .configuring_diva import DiVAConfig
+from transformers import (
+    AutoProcessor,
+    AutoTokenizer,
+    LlamaForCausalLM,
+    PreTrainedModel,
+    WhisperForConditionalGeneration,
+)
+
+
+class WhisperConnector(nn.Module):
+    def __init__(
+        self,
+    ):
+        super().__init__()
+        self.decoder = None
+        self.projection = nn.Linear(1280, 4096)
+        self.query_tokens = nn.Parameter(torch.randn(448, 1280))
+
+    def forward(self, x, output_device="cuda:1"):
+        bsz = x.shape[0]
+        query_tokens = self.query_tokens[None, :, :].expand(bsz, -1, -1)
+        virt_whisper_tokens = self.decoder(
+            inputs_embeds=query_tokens, encoder_hidden_states=x
+        )
+        if self.projection.weight.shape[-1] == 5120:
+            virtual_tokens = self.projection(virt_whisper_tokens[0].reshape(112, 5120))
+        else:
+            virtual_tokens = self.projection(virt_whisper_tokens[0])
+        return virtual_tokens.to(output_device)
+
+
+class DiVAModel(PreTrainedModel):
+    config_class = DiVAConfig
+
+    def __init__(
+        self, via_path=None, config_dict={}, device_map=None, speech_encoder_device=None
+    ):
+        super().__init__(DiVAConfig.from_dict(config_dict))
+        if speech_encoder_device is None:
+            speech_encoder_device = "cuda:0"
+        whisper = WhisperForConditionalGeneration.from_pretrained(
+            "openai/whisper-large-v3"
+        )
+        connector = WhisperConnector()
+        connector.decoder = copy.deepcopy(whisper.model.decoder)
+        if via_path is not None:
+            with open(via_path, "rb") as f:
+                sd = load(f.read())
+
+            with torch.no_grad():
+                connector.query_tokens = nn.Parameter(sd["query_tokens"])
+                connector.projection.weight = nn.Parameter(sd["projection.weight"].T)
+                connector.projection.bias = nn.Parameter(sd["projection.bias"])
+                wsd = {
+                    key.replace("connector.", ""): sd[key]
+                    for key in sd
+                    if key.startswith("connector.")
+                }
+                connector.decoder.load_state_dict(wsd)
+
+        if device_map == None:
+            num_layers = 32
+            num_gpus = 2
+            device_map = dict(
+                **{"model.embed_tokens": 1, "model.norm": 1, "lm_head": 2},
+                **{
+                    "model.layers." + str(i): 1 + (i // (num_layers // num_gpus))
+                    for i in range(num_layers)
+                },
+            )
+
+        self.connector = connector.to(speech_encoder_device)
+        self.whisper_encoder = whisper.model.encoder.to(speech_encoder_device)
+        self.llama_decoder = LlamaForCausalLM.from_pretrained(
+            "meta-llama/Meta-Llama-3-8B-Instruct",
+            device_map=device_map,
+            torch_dtype=torch.float16,
+        )
+        self.processor = AutoProcessor.from_pretrained("openai/whisper-large-v3")
+        self.tokenizer = AutoTokenizer.from_pretrained("WillHeld/via-llama")
+        self.prefix = torch.tensor([128000, 128006, 882, 128007, 271]).to(
+            self.llama_decoder.model.embed_tokens.weight.device
+        )
+
+        self.pre_user_suffix = torch.tensor(
+            self.tokenizer.encode(
+                "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n"
+            )
+        ).to(self.llama_decoder.model.embed_tokens.weight.device)
+        self.final_header = torch.tensor([128009, 128006, 78191, 128007, 271]).to(
+            self.llama_decoder.model.embed_tokens.weight.device
+        )
+        self.speech_encoder_device = speech_encoder_device
+
+    @classmethod
+    def from_pretrained(
+        cls,
+        pretrained_model_name_or_path: Optional[Union[str, os.PathLike]],
+        *model_args,
+        config=None,
+        cache_dir=None,
+        **kwargs,
+    ):
+        if os.path.isdir(pretrained_model_name_or_path):
+            via_path = (
+                pretrained_model_name_or_path + "/model-00001-of-00004.safetensors"
+            )
+            config_path = pretrained_model_name_or_path + "/config.json"
+        else:
+            # Loading from huggingface repo
+            from huggingface_hub import hf_hub_download
+
+            hf_hub_download(
+                repo_id=pretrained_model_name_or_path,
+                filename="model-00001-of-00004.safetensors",
+                token=kwargs.get("token", None),
+                local_dir=os.path.dirname(__file__),
+            )
+            hf_hub_download(
+                repo_id=pretrained_model_name_or_path,
+                filename="config.json",
+                token=kwargs.get("token", None),
+                local_dir=os.path.dirname(__file__),
+            )
+            via_path = os.path.dirname(__file__) + "/model-00001-of-00004.safetensors"
+            config_path = os.path.dirname(__file__) + "/config.json"
+        with open(config_path, "r") as f:
+            config_dict = json.loads(f.read())
+        return cls(
+            via_path,
+            config_dict,
+            kwargs["device_map"] if "device_map" in kwargs else "auto",
+            (
+                kwargs["speech_encoder_device"]
+                if "speech_encoder_device" in kwargs
+                else None
+            ),
+        )
+
+    def forward(self, audio, prefix_text_tokens, suffix_text_tokens):
+        inputs = self.processor(audio, return_tensors="pt", sampling_rate=16_000)
+        input_features = inputs.input_features.to(self.speech_encoder_device)
+        hidden_states = self.whisper_encoder(input_features=input_features)[
+            "last_hidden_state"
+        ]
+        virt_tokens = self.connector(
+            hidden_states,
+            output_device=self.llama_decoder.model.embed_tokens.weight.device,
+        ).squeeze()
+
+        prefix_embed = self.llama_decoder.model.embed_tokens(prefix_text_tokens)
+        suffix_embed = self.llama_decoder.model.embed_tokens(suffix_text_tokens)
+        inputs_embeds = torch.cat(
+            [prefix_embed, virt_tokens, suffix_embed], axis=0
+        ).unsqueeze(0)
+
+        outputs = self.llama_decoder(
+            inputs_embeds=inputs_embeds.to(
+                self.llama_decoder.model.embed_tokens.weight.device
+            ).half(),
+            return_dict=True,
+            output_hidden_states=True,
+            past_key_values=past_key_values,
+        )
+
+        return outputs
+
+    def generate(
+        self, audio, text_prompt, do_sample=False, logits_processor=None, max_new_tokens=128
+    ):
+        inputs = self.processor(audio, return_tensors="pt", sampling_rate=16_000)
+        input_features = inputs.input_features.to(self.speech_encoder_device)
+        hidden_states = self.whisper_encoder(input_features=input_features)[
+            "last_hidden_state"
+        ]
+        virt_tokens = self.connector(
+            hidden_states,
+            output_device=self.llama_decoder.model.embed_tokens.weight.device,
+        ).squeeze()
+
+        if text_prompt != None and text_prompt != "":
+            user_prompt_text = torch.tensor(
+                self.tokenizer(text_prompt, add_special_tokens=False)["input_ids"],
+                device=self.pre_user_suffix.device,
+            )
+            prefix = torch.cat(
+                [self.pre_user_suffix, user_prompt_text, self.prefix], axis=0
+            )
+        else:
+            prefix = self.prefix
+        prefix_embed = self.llama_decoder.model.embed_tokens(prefix)
+        suffix = self.final_header
+        suffix_embed = self.llama_decoder.model.embed_tokens(suffix)
+        inputs_embeds = torch.cat(
+            [prefix_embed, virt_tokens, suffix_embed], axis=0
+        ).unsqueeze(0)
+        outs = []
+        outputs = None
+        greedy = 1
+        i = 0
+        while greedy != 128009 and len(outs) < max_new_tokens:
+            past_key_values = outputs.past_key_values if outputs else None
+            outputs = self.llama_decoder(
+                inputs_embeds=inputs_embeds.to(
+                    self.llama_decoder.model.embed_tokens.weight.device
+                ).half(),
+                return_dict=True,
+                output_hidden_states=True,
+                past_key_values=past_key_values,
+            )
+            next_token_logits = outputs.logits[-1, -1, :]
+
+            if logits_processor:
+                local_outs = torch.tensor(outs) if outs != [] else suffix
+                local_outs = local_outs.reshape(1, -1)
+                next_token_logits = logits_processor(
+                    local_outs,
+                    next_token_logits.reshape(1, -1),
+                )
+                next_token_logits = next_token_logits.flatten()
+            if do_sample:
+                logits = next_token_logits / temperature
+                probs = F.softmax(logits, dim=-1)
+                greedy = torch.multinomial(probs, num_samples=1)[0]
+            else:
+                greedy = next_token_logits.argmax()
+            outs.append(greedy)
+            next_embed = self.llama_decoder.model.embed_tokens(greedy.reshape(1, 1))
+            inputs_embeds = next_embed
+        return self.tokenizer.decode(outs, skip_special_tokens=True).replace(
+            "<|eot_id|>", ""
+        )
+
+    def generate_stream(
+        self, audio, text_prompt, do_sample=False, logits_processor=None, max_new_tokens=128
+    ):
+        inputs = self.processor(audio, return_tensors="pt", sampling_rate=16_000)
+        input_features = inputs.input_features
+        hidden_states = self.whisper_encoder(input_features=input_features)[
+            "last_hidden_state"
+        ]
+        virt_tokens = self.connector(
+            hidden_states,
+            output_device=self.llama_decoder.model.embed_tokens.weight.device,
+        ).squeeze()
+
+        if text_prompt != None and text_prompt != "":
+            user_prompt_text = torch.tensor(
+                self.tokenizer(text_prompt, add_special_tokens=False)["input_ids"],
+                device=self.pre_user_suffix.device,
+            )
+            prefix = torch.cat(
+                [self.pre_user_suffix, user_prompt_text, self.prefix], axis=0
+            )
+        else:
+            prefix = self.prefix
+        prefix_embed = self.llama_decoder.model.embed_tokens(prefix)
+        suffix = self.final_header
+        suffix_embed = self.llama_decoder.model.embed_tokens(suffix)
+        inputs_embeds = torch.cat(
+            [prefix_embed, virt_tokens, suffix_embed], axis=0
+        ).unsqueeze(0)
+        outs = []
+        outputs = None
+        greedy = 1
+        i = 0
+        while greedy != 128009 and len(outs) < max_new_tokens:
+            past_key_values = outputs.past_key_values if outputs else None
+            outputs = self.llama_decoder(
+                inputs_embeds=inputs_embeds,
+                return_dict=True,
+                output_hidden_states=True,
+                past_key_values=past_key_values,
+            )
+            next_token_logits = outputs.logits[-1, -1, :]
+
+            if logits_processor:
+                local_outs = torch.tensor(outs) if outs != [] else suffix
+                local_outs = local_outs.reshape(1, -1)
+                next_token_logits = logits_processor(
+                    local_outs,
+                    next_token_logits.reshape(1, -1),
+                )
+                next_token_logits = next_token_logits.flatten()
+            if do_sample:
+                logits = next_token_logits / temperature
+                probs = F.softmax(logits, dim=-1)
+                greedy = torch.multinomial(probs, num_samples=1)[0]
+            else:
+                greedy = next_token_logits.argmax()
+            outs.append(greedy)
+            next_embed = self.llama_decoder.model.embed_tokens(greedy.reshape(1, 1))
+            inputs_embeds = next_embed
+            yield self.tokenizer.decode(outs, skip_special_tokens=True).replace("<|eot_id|>", "")
+        return self.tokenizer.decode(outs, skip_special_tokens=True).replace("<|eot_id|>", "")

preprocessor_config.json

@@ -0,0 +1,14 @@
+{
+  "chunk_length": 30,
+  "feature_extractor_type": "WhisperFeatureExtractor",
+  "feature_size": 128,
+  "hop_length": 160,
+  "n_fft": 400,
+  "n_samples": 480000,
+  "nb_max_frames": 3000,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "processor_class": "WhisperProcessor",
+  "return_attention_mask": false,
+  "sampling_rate": 16000
+}

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|reserved_special_token_0|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,2064 @@
+{
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_248|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_249|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_250|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|begin_of_text|>",
+  "chat_template": "{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|reserved_special_token_0|>",
+  "padding_side": "right",
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}