Hugging Face's logo

kloodia
/
m8x7b-final-vision-adaptee

PEFT
Safetensors
mistral-lmm
finetuned
multimodal
Model card Files Community
kloodia commited on
Commit
b71fd7e
•
1 Parent(s): 4192aad

Upload folder using huggingface_hub

Browse files
Files changed (7) hide show
  1. README.md +173 -0
  2. adapter_config.json +34 -0
  3. adapter_model.safetensors +3 -0
  4. config.json +35 -0
  5. model_named_parameters.txt +743 -0
  6. non_lora_trainables.bin +3 -0
  7. trainer_state.json +0 -0
README.md ADDED
@@ -0,0 +1,173 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: peft
4
+ tags:
5
+ - finetuned
6
+ - multimodal
7
+ base_model: mistralai/Mixtral-8x7B-Instruct-v0.1
8
+ dataset: ./out
9
+ inference: false
10
+ ---
11
+
12
+ These are weights for a version of `mistralai/Mixtral-8x7B-Instruct-v0.1` finetuned for multimodal applications.
13
+
14
+ ### Modalities
15
+
16
+ * CLIPVisionModality (use `<image>` in text and provide `images`, encoded as 576 tokens)
17
+
18
+ ### Usage
19
+
20
+ GitHub: https://github.com/sshh12/multi_token (includes training scripts and basic inference server)
21
+
22
+ ### Dataset
23
+
24
+ ./out (558128 examples)
25
+
26
+ ```
27
+ {'id': '004539375', 'images': ['/data/llava_pretrain_data/images/00453/004539375.jpg'], 'messages': [{'content': 'Render a clear and concise summary of the photo.\n<image>', 'role': 'user'}, {'content': 'select luxury furniture 3 - inch gel memory foam mattress topper', 'role': 'assistant'}]}
28
+ ```
29
+
30
+ ### Training Device(s)
31
+
32
+ ```
33
+ name, pci.bus_id, vbios_version
34
+ NVIDIA A100 80GB PCIe, 00000000:61:00.0, 92.00.90.00.0F
35
+ ```
36
+
37
+
38
+ ### Model
39
+
40
+ ```
41
+ MistralLMMForCausalLM.model =
42
+
43
+ PeftModelForCausalLM(
44
+ (base_model): LoraModel(
45
+ (model): MistralLMMForCausalLM(
46
+ (model): MistralLMMModel(
47
+ (embed_tokens): Embedding(32000, 4096)
48
+ (layers): ModuleList(
49
+ (0-31): 32 x MistralDecoderLayer(
50
+ (self_attn): MistralAttention(
51
+ (q_proj): lora.Linear(
52
+ (base_layer): Linear(in_features=4096, out_features=4096, bias=False)
53
+ (lora_dropout): ModuleDict(
54
+ (default): Dropout(p=0.05, inplace=False)
55
+ )
56
+ (lora_A): ModuleDict(
57
+ (default): Linear(in_features=4096, out_features=64, bias=False)
58
+ )
59
+ (lora_B): ModuleDict(
60
+ (default): Linear(in_features=64, out_features=4096, bias=False)
61
+ )
62
+ (lora_embedding_A): ParameterDict()
63
+ (lora_embedding_B): ParameterDict()
64
+ )
65
+ (k_proj): lora.Linear(
66
+ (base_layer): Linear(in_features=4096, out_features=1024, bias=False)
67
+ (lora_dropout): ModuleDict(
68
+ (default): Dropout(p=0.05, inplace=False)
69
+ )
70
+ (lora_A): ModuleDict(
71
+ (default): Linear(in_features=4096, out_features=64, bias=False)
72
+ )
73
+ (lora_B): ModuleDict(
74
+ (default): Linear(in_features=64, out_features=1024, bias=False)
75
+ )
76
+ (lora_embedding_A): ParameterDict()
77
+ (lora_embedding_B): ParameterDict()
78
+ )
79
+ (v_proj): lora.Linear(
80
+ (base_layer): Linear(in_features=4096, out_features=1024, bias=False)
81
+ (lora_dropout): ModuleDict(
82
+ (default): Dropout(p=0.05, inplace=False)
83
+ )
84
+ (lora_A): ModuleDict(
85
+ (default): Linear(in_features=4096, out_features=64, bias=False)
86
+ )
87
+ (lora_B): ModuleDict(
88
+ (default): Linear(in_features=64, out_features=1024, bias=False)
89
+ )
90
+ (lora_embedding_A): ParameterDict()
91
+ (lora_embedding_B): ParameterDict()
92
+ )
93
+ (o_proj): lora.Linear(
94
+ (base_layer): Linear(in_features=4096, out_features=4096, bias=False)
95
+ (lora_dropout): ModuleDict(
96
+ (default): Dropout(p=0.05, inplace=False)
97
+ )
98
+ (lora_A): ModuleDict(
99
+ (default): Linear(in_features=4096, out_features=64, bias=False)
100
+ )
101
+ (lora_B): ModuleDict(
102
+ (default): Linear(in_features=64, out_features=4096, bias=False)
103
+ )
104
+ (lora_embedding_A): ParameterDict()
105
+ (lora_embedding_B): ParameterDict()
106
+ )
107
+ (rotary_emb): MistralRotaryEmbedding()
108
+ )
109
+ (mlp): MistralMLP(
110
+ (gate_proj): lora.Linear(
111
+ (base_layer): Linear(in_features=4096, out_features=14336, bias=False)
112
+ (lora_dropout): ModuleDict(
113
+ (default): Dropout(p=0.05, inplace=False)
114
+ )
115
+ (lora_A): ModuleDict(
116
+ (default): Linear(in_features=4096, out_features=64, bias=False)
117
+ )
118
+ (lora_B): ModuleDict(
119
+ (default): Linear(in_features=64, out_features=14336, bias=False)
120
+ )
121
+ (lora_embedding_A): ParameterDict()
122
+ (lora_embedding_B): ParameterDict()
123
+ )
124
+ (up_proj): lora.Linear(
125
+ (base_layer): Linear(in_features=4096, out_features=14336, bias=False)
126
+ (lora_dropout): ModuleDict(
127
+ (default): Dropout(p=0.05, inplace=False)
128
+ )
129
+ (lora_A): ModuleDict(
130
+ (default): Linear(in_features=4096, out_features=64, bias=False)
131
+ )
132
+ (lora_B): ModuleDict(
133
+ (default): Linear(in_features=64, out_features=14336, bias=False)
134
+ )
135
+ (lora_embedding_A): ParameterDict()
136
+ (lora_embedding_B): ParameterDict()
137
+ )
138
+ (down_proj): lora.Linear(
139
+ (base_layer): Linear(in_features=14336, out_features=4096, bias=False)
140
+ (lora_dropout): ModuleDict(
141
+ (default): Dropout(p=0.05, inplace=False)
142
+ )
143
+ (lora_A): ModuleDict(
144
+ (default): Linear(in_features=14336, out_features=64, bias=False)
145
+ )
146
+ (lora_B): ModuleDict(
147
+ (default): Linear(in_features=64, out_features=4096, bias=False)
148
+ )
149
+ (lora_embedding_A): ParameterDict()
150
+ (lora_embedding_B): ParameterDict()
151
+ )
152
+ (act_fn): SiLU()
153
+ )
154
+ (input_layernorm): MistralRMSNorm()
155
+ (post_attention_layernorm): MistralRMSNorm()
156
+ )
157
+ )
158
+ (norm): MistralRMSNorm()
159
+ (vision_clip_lmm_projector): Sequential(
160
+ (0): Linear(in_features=1024, out_features=4096, bias=True)
161
+ (1): GELU(approximate='none')
162
+ (2): Linear(in_features=4096, out_features=4096, bias=True)
163
+ )
164
+ )
165
+ (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
166
+ )
167
+ )
168
+ )
169
+ ```
170
+
171
+ ### Framework versions
172
+
173
+ - PEFT 0.10.0
adapter_config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": null,
4
+ "base_model_name_or_path": "mistralai/Mixtral-8x7B-Instruct-v0.1",
5
+ "bias": "none",
6
+ "fan_in_fan_out": false,
7
+ "inference_mode": true,
8
+ "init_lora_weights": true,
9
+ "layer_replication": null,
10
+ "layers_pattern": null,
11
+ "layers_to_transform": null,
12
+ "loftq_config": {},
13
+ "lora_alpha": 16,
14
+ "lora_dropout": 0.05,
15
+ "megatron_config": null,
16
+ "megatron_core": "megatron.core",
17
+ "modules_to_save": null,
18
+ "peft_type": "LORA",
19
+ "r": 64,
20
+ "rank_pattern": {},
21
+ "revision": null,
22
+ "target_modules": [
23
+ "o_proj",
24
+ "up_proj",
25
+ "v_proj",
26
+ "q_proj",
27
+ "k_proj",
28
+ "gate_proj",
29
+ "down_proj"
30
+ ],
31
+ "task_type": "CAUSAL_LM",
32
+ "use_dora": false,
33
+ "use_rslora": false
34
+ }
adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:041e1bfd360e47521472b48f4766b73303d54e6a9912b07fe3763db0d22c14af
3
+ size 335605144
config.json ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "mistralai/Mixtral-8x7B-Instruct-v0.1",
3
+ "architectures": [
4
+ "MixtralForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 4096,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 14336,
13
+ "max_position_embeddings": 32768,
14
+ "modalities": [
15
+ "vision_clip"
16
+ ],
17
+ "modality_builder": "vision_clip",
18
+ "model_cls": "MistralLMMForCausalLM",
19
+ "model_type": "mistral-lmm",
20
+ "num_attention_heads": 32,
21
+ "num_experts_per_tok": 2,
22
+ "num_hidden_layers": 32,
23
+ "num_key_value_heads": 8,
24
+ "num_local_experts": 8,
25
+ "output_router_logits": false,
26
+ "rms_norm_eps": 1e-05,
27
+ "rope_theta": 1000000.0,
28
+ "router_aux_loss_coef": 0.02,
29
+ "sliding_window": null,
30
+ "tie_word_embeddings": false,
31
+ "torch_dtype": "bfloat16",
32
+ "transformers_version": "4.40.2",
33
+ "use_cache": true,
34
+ "vocab_size": 32000
35
+ }
model_named_parameters.txt ADDED
@@ -0,0 +1,743 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ base_model.model.model.embed_tokens.weight torch.Size([32000, 4096]) False
2
+ base_model.model.model.layers.0.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
3
+ base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
4
+ base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
5
+ base_model.model.model.layers.0.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
6
+ base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
7
+ base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
8
+ base_model.model.model.layers.0.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
9
+ base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
10
+ base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
11
+ base_model.model.model.layers.0.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
12
+ base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
13
+ base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
14
+ base_model.model.model.layers.0.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
15
+ base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
16
+ base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
17
+ base_model.model.model.layers.0.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
18
+ base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
19
+ base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
20
+ base_model.model.model.layers.0.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
21
+ base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
22
+ base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
23
+ base_model.model.model.layers.0.input_layernorm.weight torch.Size([4096]) False
24
+ base_model.model.model.layers.0.post_attention_layernorm.weight torch.Size([4096]) False
25
+ base_model.model.model.layers.1.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
26
+ base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
27
+ base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
28
+ base_model.model.model.layers.1.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
29
+ base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
30
+ base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
31
+ base_model.model.model.layers.1.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
32
+ base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
33
+ base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
34
+ base_model.model.model.layers.1.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
35
+ base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
36
+ base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
37
+ base_model.model.model.layers.1.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
38
+ base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
39
+ base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
40
+ base_model.model.model.layers.1.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
41
+ base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
42
+ base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
43
+ base_model.model.model.layers.1.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
44
+ base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
45
+ base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
46
+ base_model.model.model.layers.1.input_layernorm.weight torch.Size([4096]) False
47
+ base_model.model.model.layers.1.post_attention_layernorm.weight torch.Size([4096]) False
48
+ base_model.model.model.layers.2.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
49
+ base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
50
+ base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
51
+ base_model.model.model.layers.2.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
52
+ base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
53
+ base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
54
+ base_model.model.model.layers.2.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
55
+ base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
56
+ base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
57
+ base_model.model.model.layers.2.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
58
+ base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
59
+ base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
60
+ base_model.model.model.layers.2.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
61
+ base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
62
+ base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
63
+ base_model.model.model.layers.2.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
64
+ base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
65
+ base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
66
+ base_model.model.model.layers.2.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
67
+ base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
68
+ base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
69
+ base_model.model.model.layers.2.input_layernorm.weight torch.Size([4096]) False
70
+ base_model.model.model.layers.2.post_attention_layernorm.weight torch.Size([4096]) False
71
+ base_model.model.model.layers.3.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
72
+ base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
73
+ base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
74
+ base_model.model.model.layers.3.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
75
+ base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
76
+ base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
77
+ base_model.model.model.layers.3.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
78
+ base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
79
+ base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
80
+ base_model.model.model.layers.3.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
81
+ base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
82
+ base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
83
+ base_model.model.model.layers.3.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
84
+ base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
85
+ base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
86
+ base_model.model.model.layers.3.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
87
+ base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
88
+ base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
89
+ base_model.model.model.layers.3.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
90
+ base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
91
+ base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
92
+ base_model.model.model.layers.3.input_layernorm.weight torch.Size([4096]) False
93
+ base_model.model.model.layers.3.post_attention_layernorm.weight torch.Size([4096]) False
94
+ base_model.model.model.layers.4.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
95
+ base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
96
+ base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
97
+ base_model.model.model.layers.4.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
98
+ base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
99
+ base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
100
+ base_model.model.model.layers.4.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
101
+ base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
102
+ base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
103
+ base_model.model.model.layers.4.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
104
+ base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
105
+ base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
106
+ base_model.model.model.layers.4.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
107
+ base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
108
+ base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
109
+ base_model.model.model.layers.4.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
110
+ base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
111
+ base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
112
+ base_model.model.model.layers.4.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
113
+ base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
114
+ base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
115
+ base_model.model.model.layers.4.input_layernorm.weight torch.Size([4096]) False
116
+ base_model.model.model.layers.4.post_attention_layernorm.weight torch.Size([4096]) False
117
+ base_model.model.model.layers.5.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
118
+ base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
119
+ base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
120
+ base_model.model.model.layers.5.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
121
+ base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
122
+ base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
123
+ base_model.model.model.layers.5.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
124
+ base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
125
+ base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
126
+ base_model.model.model.layers.5.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
127
+ base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
128
+ base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
129
+ base_model.model.model.layers.5.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
130
+ base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
131
+ base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
132
+ base_model.model.model.layers.5.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
133
+ base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
134
+ base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
135
+ base_model.model.model.layers.5.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
136
+ base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
137
+ base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
138
+ base_model.model.model.layers.5.input_layernorm.weight torch.Size([4096]) False
139
+ base_model.model.model.layers.5.post_attention_layernorm.weight torch.Size([4096]) False
140
+ base_model.model.model.layers.6.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
141
+ base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
142
+ base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
143
+ base_model.model.model.layers.6.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
144
+ base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
145
+ base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
146
+ base_model.model.model.layers.6.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
147
+ base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
148
+ base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
149
+ base_model.model.model.layers.6.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
150
+ base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
151
+ base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
152
+ base_model.model.model.layers.6.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
153
+ base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
154
+ base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
155
+ base_model.model.model.layers.6.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
156
+ base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
157
+ base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
158
+ base_model.model.model.layers.6.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
159
+ base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
160
+ base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
161
+ base_model.model.model.layers.6.input_layernorm.weight torch.Size([4096]) False
162
+ base_model.model.model.layers.6.post_attention_layernorm.weight torch.Size([4096]) False
163
+ base_model.model.model.layers.7.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
164
+ base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
165
+ base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
166
+ base_model.model.model.layers.7.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
167
+ base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
168
+ base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
169
+ base_model.model.model.layers.7.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
170
+ base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
171
+ base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
172
+ base_model.model.model.layers.7.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
173
+ base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
174
+ base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
175
+ base_model.model.model.layers.7.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
176
+ base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
177
+ base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
178
+ base_model.model.model.layers.7.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
179
+ base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
180
+ base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
181
+ base_model.model.model.layers.7.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
182
+ base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
183
+ base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
184
+ base_model.model.model.layers.7.input_layernorm.weight torch.Size([4096]) False
185
+ base_model.model.model.layers.7.post_attention_layernorm.weight torch.Size([4096]) False
186
+ base_model.model.model.layers.8.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
187
+ base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
188
+ base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
189
+ base_model.model.model.layers.8.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
190
+ base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
191
+ base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
192
+ base_model.model.model.layers.8.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
193
+ base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
194
+ base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
195
+ base_model.model.model.layers.8.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
196
+ base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
197
+ base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
198
+ base_model.model.model.layers.8.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
199
+ base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
200
+ base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
201
+ base_model.model.model.layers.8.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
202
+ base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
203
+ base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
204
+ base_model.model.model.layers.8.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
205
+ base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
206
+ base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
207
+ base_model.model.model.layers.8.input_layernorm.weight torch.Size([4096]) False
208
+ base_model.model.model.layers.8.post_attention_layernorm.weight torch.Size([4096]) False
209
+ base_model.model.model.layers.9.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
210
+ base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
211
+ base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
212
+ base_model.model.model.layers.9.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
213
+ base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
214
+ base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
215
+ base_model.model.model.layers.9.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
216
+ base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
217
+ base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
218
+ base_model.model.model.layers.9.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
219
+ base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
220
+ base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
221
+ base_model.model.model.layers.9.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
222
+ base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
223
+ base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
224
+ base_model.model.model.layers.9.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
225
+ base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
226
+ base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
227
+ base_model.model.model.layers.9.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
228
+ base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
229
+ base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
230
+ base_model.model.model.layers.9.input_layernorm.weight torch.Size([4096]) False
231
+ base_model.model.model.layers.9.post_attention_layernorm.weight torch.Size([4096]) False
232
+ base_model.model.model.layers.10.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
233
+ base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
234
+ base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
235
+ base_model.model.model.layers.10.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
236
+ base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
237
+ base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
238
+ base_model.model.model.layers.10.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
239
+ base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
240
+ base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
241
+ base_model.model.model.layers.10.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
242
+ base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
243
+ base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
244
+ base_model.model.model.layers.10.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
245
+ base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
246
+ base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
247
+ base_model.model.model.layers.10.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
248
+ base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
249
+ base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
250
+ base_model.model.model.layers.10.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
251
+ base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
252
+ base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
253
+ base_model.model.model.layers.10.input_layernorm.weight torch.Size([4096]) False
254
+ base_model.model.model.layers.10.post_attention_layernorm.weight torch.Size([4096]) False
255
+ base_model.model.model.layers.11.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
256
+ base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
257
+ base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
258
+ base_model.model.model.layers.11.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
259
+ base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
260
+ base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
261
+ base_model.model.model.layers.11.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
262
+ base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
263
+ base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
264
+ base_model.model.model.layers.11.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
265
+ base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
266
+ base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
267
+ base_model.model.model.layers.11.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
268
+ base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
269
+ base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
270
+ base_model.model.model.layers.11.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
271
+ base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
272
+ base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
273
+ base_model.model.model.layers.11.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
274
+ base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
275
+ base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
276
+ base_model.model.model.layers.11.input_layernorm.weight torch.Size([4096]) False
277
+ base_model.model.model.layers.11.post_attention_layernorm.weight torch.Size([4096]) False
278
+ base_model.model.model.layers.12.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
279
+ base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
280
+ base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
281
+ base_model.model.model.layers.12.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
282
+ base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
283
+ base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
284
+ base_model.model.model.layers.12.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
285
+ base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
286
+ base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
287
+ base_model.model.model.layers.12.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
288
+ base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
289
+ base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
290
+ base_model.model.model.layers.12.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
291
+ base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
292
+ base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
293
+ base_model.model.model.layers.12.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
294
+ base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
295
+ base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
296
+ base_model.model.model.layers.12.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
297
+ base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
298
+ base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
299
+ base_model.model.model.layers.12.input_layernorm.weight torch.Size([4096]) False
300
+ base_model.model.model.layers.12.post_attention_layernorm.weight torch.Size([4096]) False
301
+ base_model.model.model.layers.13.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
302
+ base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
303
+ base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
304
+ base_model.model.model.layers.13.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
305
+ base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
306
+ base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
307
+ base_model.model.model.layers.13.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
308
+ base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
309
+ base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
310
+ base_model.model.model.layers.13.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
311
+ base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
312
+ base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
313
+ base_model.model.model.layers.13.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
314
+ base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
315
+ base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
316
+ base_model.model.model.layers.13.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
317
+ base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
318
+ base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
319
+ base_model.model.model.layers.13.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
320
+ base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
321
+ base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
322
+ base_model.model.model.layers.13.input_layernorm.weight torch.Size([4096]) False
323
+ base_model.model.model.layers.13.post_attention_layernorm.weight torch.Size([4096]) False
324
+ base_model.model.model.layers.14.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
325
+ base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
326
+ base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
327
+ base_model.model.model.layers.14.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
328
+ base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
329
+ base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
330
+ base_model.model.model.layers.14.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
331
+ base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
332
+ base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
333
+ base_model.model.model.layers.14.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
334
+ base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
335
+ base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
336
+ base_model.model.model.layers.14.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
337
+ base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
338
+ base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
339
+ base_model.model.model.layers.14.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
340
+ base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
341
+ base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
342
+ base_model.model.model.layers.14.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
343
+ base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
344
+ base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
345
+ base_model.model.model.layers.14.input_layernorm.weight torch.Size([4096]) False
346
+ base_model.model.model.layers.14.post_attention_layernorm.weight torch.Size([4096]) False
347
+ base_model.model.model.layers.15.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
348
+ base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
349
+ base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
350
+ base_model.model.model.layers.15.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
351
+ base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
352
+ base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
353
+ base_model.model.model.layers.15.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
354
+ base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
355
+ base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
356
+ base_model.model.model.layers.15.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
357
+ base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
358
+ base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
359
+ base_model.model.model.layers.15.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
360
+ base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
361
+ base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
362
+ base_model.model.model.layers.15.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
363
+ base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
364
+ base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
365
+ base_model.model.model.layers.15.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
366
+ base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
367
+ base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
368
+ base_model.model.model.layers.15.input_layernorm.weight torch.Size([4096]) False
369
+ base_model.model.model.layers.15.post_attention_layernorm.weight torch.Size([4096]) False
370
+ base_model.model.model.layers.16.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
371
+ base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
372
+ base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
373
+ base_model.model.model.layers.16.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
374
+ base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
375
+ base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
376
+ base_model.model.model.layers.16.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
377
+ base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
378
+ base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
379
+ base_model.model.model.layers.16.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
380
+ base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
381
+ base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
382
+ base_model.model.model.layers.16.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
383
+ base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
384
+ base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
385
+ base_model.model.model.layers.16.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
386
+ base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
387
+ base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
388
+ base_model.model.model.layers.16.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
389
+ base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
390
+ base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
391
+ base_model.model.model.layers.16.input_layernorm.weight torch.Size([4096]) False
392
+ base_model.model.model.layers.16.post_attention_layernorm.weight torch.Size([4096]) False
393
+ base_model.model.model.layers.17.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
394
+ base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
395
+ base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
396
+ base_model.model.model.layers.17.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
397
+ base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
398
+ base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
399
+ base_model.model.model.layers.17.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
400
+ base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
401
+ base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
402
+ base_model.model.model.layers.17.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
403
+ base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
404
+ base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
405
+ base_model.model.model.layers.17.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
406
+ base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
407
+ base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
408
+ base_model.model.model.layers.17.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
409
+ base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
410
+ base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
411
+ base_model.model.model.layers.17.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
412
+ base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
413
+ base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
414
+ base_model.model.model.layers.17.input_layernorm.weight torch.Size([4096]) False
415
+ base_model.model.model.layers.17.post_attention_layernorm.weight torch.Size([4096]) False
416
+ base_model.model.model.layers.18.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
417
+ base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
418
+ base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
419
+ base_model.model.model.layers.18.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
420
+ base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
421
+ base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
422
+ base_model.model.model.layers.18.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
423
+ base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
424
+ base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
425
+ base_model.model.model.layers.18.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
426
+ base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
427
+ base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
428
+ base_model.model.model.layers.18.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
429
+ base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
430
+ base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
431
+ base_model.model.model.layers.18.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
432
+ base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
433
+ base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
434
+ base_model.model.model.layers.18.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
435
+ base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
436
+ base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
437
+ base_model.model.model.layers.18.input_layernorm.weight torch.Size([4096]) False
438
+ base_model.model.model.layers.18.post_attention_layernorm.weight torch.Size([4096]) False
439
+ base_model.model.model.layers.19.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
440
+ base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
441
+ base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
442
+ base_model.model.model.layers.19.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
443
+ base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
444
+ base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
445
+ base_model.model.model.layers.19.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
446
+ base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
447
+ base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
448
+ base_model.model.model.layers.19.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
449
+ base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
450
+ base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
451
+ base_model.model.model.layers.19.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
452
+ base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
453
+ base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
454
+ base_model.model.model.layers.19.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
455
+ base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
456
+ base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
457
+ base_model.model.model.layers.19.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
458
+ base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
459
+ base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
460
+ base_model.model.model.layers.19.input_layernorm.weight torch.Size([4096]) False
461
+ base_model.model.model.layers.19.post_attention_layernorm.weight torch.Size([4096]) False
462
+ base_model.model.model.layers.20.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
463
+ base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
464
+ base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
465
+ base_model.model.model.layers.20.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
466
+ base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
467
+ base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
468
+ base_model.model.model.layers.20.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
469
+ base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
470
+ base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
471
+ base_model.model.model.layers.20.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
472
+ base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
473
+ base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
474
+ base_model.model.model.layers.20.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
475
+ base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
476
+ base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
477
+ base_model.model.model.layers.20.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
478
+ base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
479
+ base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
480
+ base_model.model.model.layers.20.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
481
+ base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
482
+ base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
483
+ base_model.model.model.layers.20.input_layernorm.weight torch.Size([4096]) False
484
+ base_model.model.model.layers.20.post_attention_layernorm.weight torch.Size([4096]) False
485
+ base_model.model.model.layers.21.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
486
+ base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
487
+ base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
488
+ base_model.model.model.layers.21.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
489
+ base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
490
+ base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
491
+ base_model.model.model.layers.21.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
492
+ base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
493
+ base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
494
+ base_model.model.model.layers.21.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
495
+ base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
496
+ base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
497
+ base_model.model.model.layers.21.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
498
+ base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
499
+ base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
500
+ base_model.model.model.layers.21.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
501
+ base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
502
+ base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
503
+ base_model.model.model.layers.21.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
504
+ base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
505
+ base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
506
+ base_model.model.model.layers.21.input_layernorm.weight torch.Size([4096]) False
507
+ base_model.model.model.layers.21.post_attention_layernorm.weight torch.Size([4096]) False
508
+ base_model.model.model.layers.22.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
509
+ base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
510
+ base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
511
+ base_model.model.model.layers.22.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
512
+ base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
513
+ base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
514
+ base_model.model.model.layers.22.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
515
+ base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
516
+ base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
517
+ base_model.model.model.layers.22.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
518
+ base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
519
+ base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
520
+ base_model.model.model.layers.22.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
521
+ base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
522
+ base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
523
+ base_model.model.model.layers.22.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
524
+ base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
525
+ base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
526
+ base_model.model.model.layers.22.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
527
+ base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
528
+ base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
529
+ base_model.model.model.layers.22.input_layernorm.weight torch.Size([4096]) False
530
+ base_model.model.model.layers.22.post_attention_layernorm.weight torch.Size([4096]) False
531
+ base_model.model.model.layers.23.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
532
+ base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
533
+ base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
534
+ base_model.model.model.layers.23.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
535
+ base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
536
+ base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
537
+ base_model.model.model.layers.23.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
538
+ base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
539
+ base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
540
+ base_model.model.model.layers.23.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
541
+ base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
542
+ base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
543
+ base_model.model.model.layers.23.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
544
+ base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
545
+ base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
546
+ base_model.model.model.layers.23.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
547
+ base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
548
+ base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
549
+ base_model.model.model.layers.23.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
550
+ base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
551
+ base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
552
+ base_model.model.model.layers.23.input_layernorm.weight torch.Size([4096]) False
553
+ base_model.model.model.layers.23.post_attention_layernorm.weight torch.Size([4096]) False
554
+ base_model.model.model.layers.24.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
555
+ base_model.model.model.layers.24.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
556
+ base_model.model.model.layers.24.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
557
+ base_model.model.model.layers.24.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
558
+ base_model.model.model.layers.24.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
559
+ base_model.model.model.layers.24.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
560
+ base_model.model.model.layers.24.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
561
+ base_model.model.model.layers.24.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
562
+ base_model.model.model.layers.24.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
563
+ base_model.model.model.layers.24.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
564
+ base_model.model.model.layers.24.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
565
+ base_model.model.model.layers.24.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
566
+ base_model.model.model.layers.24.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
567
+ base_model.model.model.layers.24.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
568
+ base_model.model.model.layers.24.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
569
+ base_model.model.model.layers.24.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
570
+ base_model.model.model.layers.24.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
571
+ base_model.model.model.layers.24.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
572
+ base_model.model.model.layers.24.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
573
+ base_model.model.model.layers.24.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
574
+ base_model.model.model.layers.24.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
575
+ base_model.model.model.layers.24.input_layernorm.weight torch.Size([4096]) False
576
+ base_model.model.model.layers.24.post_attention_layernorm.weight torch.Size([4096]) False
577
+ base_model.model.model.layers.25.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
578
+ base_model.model.model.layers.25.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
579
+ base_model.model.model.layers.25.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
580
+ base_model.model.model.layers.25.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
581
+ base_model.model.model.layers.25.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
582
+ base_model.model.model.layers.25.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
583
+ base_model.model.model.layers.25.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
584
+ base_model.model.model.layers.25.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
585
+ base_model.model.model.layers.25.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
586
+ base_model.model.model.layers.25.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
587
+ base_model.model.model.layers.25.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
588
+ base_model.model.model.layers.25.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
589
+ base_model.model.model.layers.25.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
590
+ base_model.model.model.layers.25.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
591
+ base_model.model.model.layers.25.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
592
+ base_model.model.model.layers.25.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
593
+ base_model.model.model.layers.25.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
594
+ base_model.model.model.layers.25.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
595
+ base_model.model.model.layers.25.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
596
+ base_model.model.model.layers.25.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
597
+ base_model.model.model.layers.25.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
598
+ base_model.model.model.layers.25.input_layernorm.weight torch.Size([4096]) False
599
+ base_model.model.model.layers.25.post_attention_layernorm.weight torch.Size([4096]) False
600
+ base_model.model.model.layers.26.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
601
+ base_model.model.model.layers.26.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
602
+ base_model.model.model.layers.26.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
603
+ base_model.model.model.layers.26.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
604
+ base_model.model.model.layers.26.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
605
+ base_model.model.model.layers.26.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
606
+ base_model.model.model.layers.26.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
607
+ base_model.model.model.layers.26.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
608
+ base_model.model.model.layers.26.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
609
+ base_model.model.model.layers.26.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
610
+ base_model.model.model.layers.26.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
611
+ base_model.model.model.layers.26.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
612
+ base_model.model.model.layers.26.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
613
+ base_model.model.model.layers.26.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
614
+ base_model.model.model.layers.26.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
615
+ base_model.model.model.layers.26.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
616
+ base_model.model.model.layers.26.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
617
+ base_model.model.model.layers.26.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
618
+ base_model.model.model.layers.26.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
619
+ base_model.model.model.layers.26.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
620
+ base_model.model.model.layers.26.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
621
+ base_model.model.model.layers.26.input_layernorm.weight torch.Size([4096]) False
622
+ base_model.model.model.layers.26.post_attention_layernorm.weight torch.Size([4096]) False
623
+ base_model.model.model.layers.27.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
624
+ base_model.model.model.layers.27.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
625
+ base_model.model.model.layers.27.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
626
+ base_model.model.model.layers.27.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
627
+ base_model.model.model.layers.27.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
628
+ base_model.model.model.layers.27.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
629
+ base_model.model.model.layers.27.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
630
+ base_model.model.model.layers.27.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
631
+ base_model.model.model.layers.27.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
632
+ base_model.model.model.layers.27.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
633
+ base_model.model.model.layers.27.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
634
+ base_model.model.model.layers.27.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
635
+ base_model.model.model.layers.27.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
636
+ base_model.model.model.layers.27.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
637
+ base_model.model.model.layers.27.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
638
+ base_model.model.model.layers.27.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
639
+ base_model.model.model.layers.27.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
640
+ base_model.model.model.layers.27.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
641
+ base_model.model.model.layers.27.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
642
+ base_model.model.model.layers.27.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
643
+ base_model.model.model.layers.27.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
644
+ base_model.model.model.layers.27.input_layernorm.weight torch.Size([4096]) False
645
+ base_model.model.model.layers.27.post_attention_layernorm.weight torch.Size([4096]) False
646
+ base_model.model.model.layers.28.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
647
+ base_model.model.model.layers.28.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
648
+ base_model.model.model.layers.28.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
649
+ base_model.model.model.layers.28.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
650
+ base_model.model.model.layers.28.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
651
+ base_model.model.model.layers.28.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
652
+ base_model.model.model.layers.28.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
653
+ base_model.model.model.layers.28.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
654
+ base_model.model.model.layers.28.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
655
+ base_model.model.model.layers.28.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
656
+ base_model.model.model.layers.28.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
657
+ base_model.model.model.layers.28.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
658
+ base_model.model.model.layers.28.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
659
+ base_model.model.model.layers.28.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
660
+ base_model.model.model.layers.28.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
661
+ base_model.model.model.layers.28.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
662
+ base_model.model.model.layers.28.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
663
+ base_model.model.model.layers.28.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
664
+ base_model.model.model.layers.28.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
665
+ base_model.model.model.layers.28.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
666
+ base_model.model.model.layers.28.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
667
+ base_model.model.model.layers.28.input_layernorm.weight torch.Size([4096]) False
668
+ base_model.model.model.layers.28.post_attention_layernorm.weight torch.Size([4096]) False
669
+ base_model.model.model.layers.29.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
670
+ base_model.model.model.layers.29.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
671
+ base_model.model.model.layers.29.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
672
+ base_model.model.model.layers.29.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
673
+ base_model.model.model.layers.29.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
674
+ base_model.model.model.layers.29.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
675
+ base_model.model.model.layers.29.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
676
+ base_model.model.model.layers.29.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
677
+ base_model.model.model.layers.29.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
678
+ base_model.model.model.layers.29.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
679
+ base_model.model.model.layers.29.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
680
+ base_model.model.model.layers.29.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
681
+ base_model.model.model.layers.29.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
682
+ base_model.model.model.layers.29.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
683
+ base_model.model.model.layers.29.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
684
+ base_model.model.model.layers.29.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
685
+ base_model.model.model.layers.29.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
686
+ base_model.model.model.layers.29.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
687
+ base_model.model.model.layers.29.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
688
+ base_model.model.model.layers.29.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
689
+ base_model.model.model.layers.29.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
690
+ base_model.model.model.layers.29.input_layernorm.weight torch.Size([4096]) False
691
+ base_model.model.model.layers.29.post_attention_layernorm.weight torch.Size([4096]) False
692
+ base_model.model.model.layers.30.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
693
+ base_model.model.model.layers.30.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
694
+ base_model.model.model.layers.30.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
695
+ base_model.model.model.layers.30.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
696
+ base_model.model.model.layers.30.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
697
+ base_model.model.model.layers.30.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
698
+ base_model.model.model.layers.30.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
699
+ base_model.model.model.layers.30.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
700
+ base_model.model.model.layers.30.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
701
+ base_model.model.model.layers.30.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
702
+ base_model.model.model.layers.30.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
703
+ base_model.model.model.layers.30.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
704
+ base_model.model.model.layers.30.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
705
+ base_model.model.model.layers.30.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
706
+ base_model.model.model.layers.30.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
707
+ base_model.model.model.layers.30.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
708
+ base_model.model.model.layers.30.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
709
+ base_model.model.model.layers.30.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
710
+ base_model.model.model.layers.30.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
711
+ base_model.model.model.layers.30.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
712
+ base_model.model.model.layers.30.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
713
+ base_model.model.model.layers.30.input_layernorm.weight torch.Size([4096]) False
714
+ base_model.model.model.layers.30.post_attention_layernorm.weight torch.Size([4096]) False
715
+ base_model.model.model.layers.31.self_attn.q_proj.base_layer.weight torch.Size([4096, 4096]) False
716
+ base_model.model.model.layers.31.self_attn.q_proj.lora_A.default.weight torch.Size([64, 4096]) False
717
+ base_model.model.model.layers.31.self_attn.q_proj.lora_B.default.weight torch.Size([4096, 64]) False
718
+ base_model.model.model.layers.31.self_attn.k_proj.base_layer.weight torch.Size([1024, 4096]) False
719
+ base_model.model.model.layers.31.self_attn.k_proj.lora_A.default.weight torch.Size([64, 4096]) False
720
+ base_model.model.model.layers.31.self_attn.k_proj.lora_B.default.weight torch.Size([1024, 64]) False
721
+ base_model.model.model.layers.31.self_attn.v_proj.base_layer.weight torch.Size([1024, 4096]) False
722
+ base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight torch.Size([64, 4096]) False
723
+ base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 64]) False
724
+ base_model.model.model.layers.31.self_attn.o_proj.base_layer.weight torch.Size([4096, 4096]) False
725
+ base_model.model.model.layers.31.self_attn.o_proj.lora_A.default.weight torch.Size([64, 4096]) False
726
+ base_model.model.model.layers.31.self_attn.o_proj.lora_B.default.weight torch.Size([4096, 64]) False
727
+ base_model.model.model.layers.31.mlp.gate_proj.base_layer.weight torch.Size([14336, 4096]) False
728
+ base_model.model.model.layers.31.mlp.gate_proj.lora_A.default.weight torch.Size([64, 4096]) False
729
+ base_model.model.model.layers.31.mlp.gate_proj.lora_B.default.weight torch.Size([14336, 64]) False
730
+ base_model.model.model.layers.31.mlp.up_proj.base_layer.weight torch.Size([14336, 4096]) False
731
+ base_model.model.model.layers.31.mlp.up_proj.lora_A.default.weight torch.Size([64, 4096]) False
732
+ base_model.model.model.layers.31.mlp.up_proj.lora_B.default.weight torch.Size([14336, 64]) False
733
+ base_model.model.model.layers.31.mlp.down_proj.base_layer.weight torch.Size([4096, 14336]) False
734
+ base_model.model.model.layers.31.mlp.down_proj.lora_A.default.weight torch.Size([64, 14336]) False
735
+ base_model.model.model.layers.31.mlp.down_proj.lora_B.default.weight torch.Size([4096, 64]) False
736
+ base_model.model.model.layers.31.input_layernorm.weight torch.Size([4096]) False
737
+ base_model.model.model.layers.31.post_attention_layernorm.weight torch.Size([4096]) False
738
+ base_model.model.model.norm.weight torch.Size([4096]) False
739
+ base_model.model.model.vision_clip_lmm_projector.0.weight torch.Size([4096, 1024]) True
740
+ base_model.model.model.vision_clip_lmm_projector.0.bias torch.Size([4096]) True
741
+ base_model.model.model.vision_clip_lmm_projector.2.weight torch.Size([4096, 4096]) True
742
+ base_model.model.model.vision_clip_lmm_projector.2.bias torch.Size([4096]) True
743
+ base_model.model.lm_head.weight torch.Size([32000, 4096]) False
non_lora_trainables.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:54decdfde1b64530e2b993dff729c3cf1c2d88ebe4e3445b20537c4573581c53
3
+ size 83920679
trainer_state.json ADDED
The diff for this file is too large to render. See raw diff