Hugging Face's logo

markury
/
sd35-spirited-away-lokr

Text-to-Image
Diffusers
sd3
sd3-diffusers
simpletuner
safe-for-work
lora
template:sd-lora
lycoris
Model card Files Community
markury commited on
Commit
f612435
•
1 Parent(s): e3113b9

Model card auto-generated by SimpleTuner

Browse files
Files changed (1) hide show
  1. README.md +1075 -14
README.md CHANGED
@@ -79,9 +79,9 @@ You may reuse the base model text encoder for inference.
79
 
80
  ## Training settings
81
 
82
- - Training epochs: 4
83
- - Training steps: 900
84
- - Learning rate: 5e-06
85
  - Max grad norm: 0.01
86
  - Effective batch size: 4
87
  - Micro-batch size: 4
@@ -89,29 +89,1090 @@ You may reuse the base model text encoder for inference.
89
  - Number of GPUs: 1
90
  - Prediction type: flow-matching
91
  - Rescaled betas zero SNR: False
92
- - Optimizer: optimi-stableadamw
93
  - Precision: Pure BF16
94
- - Quantised: No
95
  - Xformers: Not used
96
  - LyCORIS Config:
97
  ```json
98
  {
99
- "bypass_mode": true,
100
  "algo": "lokr",
101
  "multiplier": 1.0,
102
- "full_matrix": true,
103
- "linear_dim": 10000,
104
  "linear_alpha": 1,
105
- "factor": 12,
 
106
  "apply_preset": {
107
  "target_module": [
108
- "Attention"
109
  ],
110
- "module_algo_map": {
111
- "Attention": {
112
- "factor": 6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
113
  }
114
- }
 
115
  }
116
  }
117
  ```
 
79
 
80
  ## Training settings
81
 
82
+ - Training epochs: 0
83
+ - Training steps: 100
84
+ - Learning rate: 2e-06
85
  - Max grad norm: 0.01
86
  - Effective batch size: 4
87
  - Micro-batch size: 4
 
89
  - Number of GPUs: 1
90
  - Prediction type: flow-matching
91
  - Rescaled betas zero SNR: False
92
+ - Optimizer: adamw_bf16
93
  - Precision: Pure BF16
94
+ - Quantised: Yes: int8-quanto
95
  - Xformers: Not used
96
  - LyCORIS Config:
97
  ```json
98
  {
 
99
  "algo": "lokr",
100
  "multiplier": 1.0,
101
+ "linear_dim": 1000000,
 
102
  "linear_alpha": 1,
103
+ "factor": 1,
104
+ "full_matrix": true,
105
  "apply_preset": {
106
  "target_module": [
107
+ "JointTransformerBlock"
108
  ],
109
+ "name_algo_map": {
110
+ "transformer_blocks.0.norm1*": {
111
+ "algo": "lokr",
112
+ "factor": 1,
113
+ "linear_dim": 1000000,
114
+ "linear_alpha": 1,
115
+ "full_matrix": true
116
+ },
117
+ "transformer_blocks.0.norm1_context*": {
118
+ "algo": "lokr",
119
+ "factor": 1,
120
+ "linear_dim": 1000000,
121
+ "linear_alpha": 1,
122
+ "full_matrix": true
123
+ },
124
+ "transformer_blocks.0.ff*": {
125
+ "algo": "lokr",
126
+ "factor": 1,
127
+ "linear_dim": 1000000,
128
+ "linear_alpha": 1,
129
+ "full_matrix": true
130
+ },
131
+ "transformer_blocks.0.*": {
132
+ "algo": "lokr",
133
+ "factor": 2,
134
+ "linear_dim": 1000000,
135
+ "linear_alpha": 1,
136
+ "full_matrix": true
137
+ },
138
+ "transformer_blocks.1.norm1*": {
139
+ "algo": "lokr",
140
+ "factor": 3,
141
+ "linear_dim": 1000000,
142
+ "linear_alpha": 1,
143
+ "full_matrix": true
144
+ },
145
+ "transformer_blocks.1.norm1_context*": {
146
+ "algo": "lokr",
147
+ "factor": 3,
148
+ "linear_dim": 1000000,
149
+ "linear_alpha": 1,
150
+ "full_matrix": true
151
+ },
152
+ "transformer_blocks.1.ff*": {
153
+ "algo": "lokr",
154
+ "factor": 3,
155
+ "linear_dim": 1000000,
156
+ "linear_alpha": 1,
157
+ "full_matrix": true
158
+ },
159
+ "transformer_blocks.1.*": {
160
+ "algo": "lokr",
161
+ "factor": 6,
162
+ "linear_dim": 1000000,
163
+ "linear_alpha": 1,
164
+ "full_matrix": true
165
+ },
166
+ "transformer_blocks.2.norm1*": {
167
+ "algo": "lokr",
168
+ "factor": 4,
169
+ "linear_dim": 1000000,
170
+ "linear_alpha": 1,
171
+ "full_matrix": true
172
+ },
173
+ "transformer_blocks.2.norm1_context*": {
174
+ "algo": "lokr",
175
+ "factor": 4,
176
+ "linear_dim": 1000000,
177
+ "linear_alpha": 1,
178
+ "full_matrix": true
179
+ },
180
+ "transformer_blocks.2.ff*": {
181
+ "algo": "lokr",
182
+ "factor": 4,
183
+ "linear_dim": 1000000,
184
+ "linear_alpha": 1,
185
+ "full_matrix": true
186
+ },
187
+ "transformer_blocks.2.*": {
188
+ "algo": "lokr",
189
+ "factor": 8,
190
+ "linear_dim": 1000000,
191
+ "linear_alpha": 1,
192
+ "full_matrix": true
193
+ },
194
+ "transformer_blocks.3.norm1*": {
195
+ "algo": "lokr",
196
+ "factor": 7,
197
+ "linear_dim": 1000000,
198
+ "linear_alpha": 1,
199
+ "full_matrix": true
200
+ },
201
+ "transformer_blocks.3.norm1_context*": {
202
+ "algo": "lokr",
203
+ "factor": 7,
204
+ "linear_dim": 1000000,
205
+ "linear_alpha": 1,
206
+ "full_matrix": true
207
+ },
208
+ "transformer_blocks.3.ff*": {
209
+ "algo": "lokr",
210
+ "factor": 7,
211
+ "linear_dim": 1000000,
212
+ "linear_alpha": 1,
213
+ "full_matrix": true
214
+ },
215
+ "transformer_blocks.3.*": {
216
+ "algo": "lokr",
217
+ "factor": 14,
218
+ "linear_dim": 1000000,
219
+ "linear_alpha": 1,
220
+ "full_matrix": true
221
+ },
222
+ "transformer_blocks.4.norm1*": {
223
+ "algo": "lokr",
224
+ "factor": 6,
225
+ "linear_dim": 1000000,
226
+ "linear_alpha": 1,
227
+ "full_matrix": true
228
+ },
229
+ "transformer_blocks.4.norm1_context*": {
230
+ "algo": "lokr",
231
+ "factor": 6,
232
+ "linear_dim": 1000000,
233
+ "linear_alpha": 1,
234
+ "full_matrix": true
235
+ },
236
+ "transformer_blocks.4.ff*": {
237
+ "algo": "lokr",
238
+ "factor": 6,
239
+ "linear_dim": 1000000,
240
+ "linear_alpha": 1,
241
+ "full_matrix": true
242
+ },
243
+ "transformer_blocks.4.*": {
244
+ "algo": "lokr",
245
+ "factor": 12,
246
+ "linear_dim": 1000000,
247
+ "linear_alpha": 1,
248
+ "full_matrix": true
249
+ },
250
+ "transformer_blocks.5.norm1*": {
251
+ "algo": "lokr",
252
+ "factor": 7,
253
+ "linear_dim": 1000000,
254
+ "linear_alpha": 1,
255
+ "full_matrix": true
256
+ },
257
+ "transformer_blocks.5.norm1_context*": {
258
+ "algo": "lokr",
259
+ "factor": 7,
260
+ "linear_dim": 1000000,
261
+ "linear_alpha": 1,
262
+ "full_matrix": true
263
+ },
264
+ "transformer_blocks.5.ff*": {
265
+ "algo": "lokr",
266
+ "factor": 7,
267
+ "linear_dim": 1000000,
268
+ "linear_alpha": 1,
269
+ "full_matrix": true
270
+ },
271
+ "transformer_blocks.5.*": {
272
+ "algo": "lokr",
273
+ "factor": 14,
274
+ "linear_dim": 1000000,
275
+ "linear_alpha": 1,
276
+ "full_matrix": true
277
+ },
278
+ "transformer_blocks.6.norm1*": {
279
+ "algo": "lokr",
280
+ "factor": 7,
281
+ "linear_dim": 1000000,
282
+ "linear_alpha": 1,
283
+ "full_matrix": true
284
+ },
285
+ "transformer_blocks.6.norm1_context*": {
286
+ "algo": "lokr",
287
+ "factor": 7,
288
+ "linear_dim": 1000000,
289
+ "linear_alpha": 1,
290
+ "full_matrix": true
291
+ },
292
+ "transformer_blocks.6.ff*": {
293
+ "algo": "lokr",
294
+ "factor": 7,
295
+ "linear_dim": 1000000,
296
+ "linear_alpha": 1,
297
+ "full_matrix": true
298
+ },
299
+ "transformer_blocks.6.*": {
300
+ "algo": "lokr",
301
+ "factor": 14,
302
+ "linear_dim": 1000000,
303
+ "linear_alpha": 1,
304
+ "full_matrix": true
305
+ },
306
+ "transformer_blocks.7.norm1*": {
307
+ "algo": "lokr",
308
+ "factor": 7,
309
+ "linear_dim": 1000000,
310
+ "linear_alpha": 1,
311
+ "full_matrix": true
312
+ },
313
+ "transformer_blocks.7.norm1_context*": {
314
+ "algo": "lokr",
315
+ "factor": 7,
316
+ "linear_dim": 1000000,
317
+ "linear_alpha": 1,
318
+ "full_matrix": true
319
+ },
320
+ "transformer_blocks.7.ff*": {
321
+ "algo": "lokr",
322
+ "factor": 7,
323
+ "linear_dim": 1000000,
324
+ "linear_alpha": 1,
325
+ "full_matrix": true
326
+ },
327
+ "transformer_blocks.7.*": {
328
+ "algo": "lokr",
329
+ "factor": 14,
330
+ "linear_dim": 1000000,
331
+ "linear_alpha": 1,
332
+ "full_matrix": true
333
+ },
334
+ "transformer_blocks.8.norm1*": {
335
+ "algo": "lokr",
336
+ "factor": 8,
337
+ "linear_dim": 1000000,
338
+ "linear_alpha": 1,
339
+ "full_matrix": true
340
+ },
341
+ "transformer_blocks.8.norm1_context*": {
342
+ "algo": "lokr",
343
+ "factor": 8,
344
+ "linear_dim": 1000000,
345
+ "linear_alpha": 1,
346
+ "full_matrix": true
347
+ },
348
+ "transformer_blocks.8.ff*": {
349
+ "algo": "lokr",
350
+ "factor": 8,
351
+ "linear_dim": 1000000,
352
+ "linear_alpha": 1,
353
+ "full_matrix": true
354
+ },
355
+ "transformer_blocks.8.*": {
356
+ "algo": "lokr",
357
+ "factor": 16,
358
+ "linear_dim": 1000000,
359
+ "linear_alpha": 1,
360
+ "full_matrix": true
361
+ },
362
+ "transformer_blocks.9.norm1*": {
363
+ "algo": "lokr",
364
+ "factor": 8,
365
+ "linear_dim": 1000000,
366
+ "linear_alpha": 1,
367
+ "full_matrix": true
368
+ },
369
+ "transformer_blocks.9.norm1_context*": {
370
+ "algo": "lokr",
371
+ "factor": 8,
372
+ "linear_dim": 1000000,
373
+ "linear_alpha": 1,
374
+ "full_matrix": true
375
+ },
376
+ "transformer_blocks.9.ff*": {
377
+ "algo": "lokr",
378
+ "factor": 8,
379
+ "linear_dim": 1000000,
380
+ "linear_alpha": 1,
381
+ "full_matrix": true
382
+ },
383
+ "transformer_blocks.9.*": {
384
+ "algo": "lokr",
385
+ "factor": 16,
386
+ "linear_dim": 1000000,
387
+ "linear_alpha": 1,
388
+ "full_matrix": true
389
+ },
390
+ "transformer_blocks.10.norm1*": {
391
+ "algo": "lokr",
392
+ "factor": 7,
393
+ "linear_dim": 1000000,
394
+ "linear_alpha": 1,
395
+ "full_matrix": true
396
+ },
397
+ "transformer_blocks.10.norm1_context*": {
398
+ "algo": "lokr",
399
+ "factor": 7,
400
+ "linear_dim": 1000000,
401
+ "linear_alpha": 1,
402
+ "full_matrix": true
403
+ },
404
+ "transformer_blocks.10.ff*": {
405
+ "algo": "lokr",
406
+ "factor": 7,
407
+ "linear_dim": 1000000,
408
+ "linear_alpha": 1,
409
+ "full_matrix": true
410
+ },
411
+ "transformer_blocks.10.*": {
412
+ "algo": "lokr",
413
+ "factor": 14,
414
+ "linear_dim": 1000000,
415
+ "linear_alpha": 1,
416
+ "full_matrix": true
417
+ },
418
+ "transformer_blocks.11.norm1*": {
419
+ "algo": "lokr",
420
+ "factor": 7,
421
+ "linear_dim": 1000000,
422
+ "linear_alpha": 1,
423
+ "full_matrix": true
424
+ },
425
+ "transformer_blocks.11.norm1_context*": {
426
+ "algo": "lokr",
427
+ "factor": 7,
428
+ "linear_dim": 1000000,
429
+ "linear_alpha": 1,
430
+ "full_matrix": true
431
+ },
432
+ "transformer_blocks.11.ff*": {
433
+ "algo": "lokr",
434
+ "factor": 7,
435
+ "linear_dim": 1000000,
436
+ "linear_alpha": 1,
437
+ "full_matrix": true
438
+ },
439
+ "transformer_blocks.11.*": {
440
+ "algo": "lokr",
441
+ "factor": 14,
442
+ "linear_dim": 1000000,
443
+ "linear_alpha": 1,
444
+ "full_matrix": true
445
+ },
446
+ "transformer_blocks.12.norm1*": {
447
+ "algo": "lokr",
448
+ "factor": 7,
449
+ "linear_dim": 1000000,
450
+ "linear_alpha": 1,
451
+ "full_matrix": true
452
+ },
453
+ "transformer_blocks.12.norm1_context*": {
454
+ "algo": "lokr",
455
+ "factor": 7,
456
+ "linear_dim": 1000000,
457
+ "linear_alpha": 1,
458
+ "full_matrix": true
459
+ },
460
+ "transformer_blocks.12.ff*": {
461
+ "algo": "lokr",
462
+ "factor": 7,
463
+ "linear_dim": 1000000,
464
+ "linear_alpha": 1,
465
+ "full_matrix": true
466
+ },
467
+ "transformer_blocks.12.*": {
468
+ "algo": "lokr",
469
+ "factor": 14,
470
+ "linear_dim": 1000000,
471
+ "linear_alpha": 1,
472
+ "full_matrix": true
473
+ },
474
+ "transformer_blocks.13.norm1*": {
475
+ "algo": "lokr",
476
+ "factor": 8,
477
+ "linear_dim": 1000000,
478
+ "linear_alpha": 1,
479
+ "full_matrix": true
480
+ },
481
+ "transformer_blocks.13.norm1_context*": {
482
+ "algo": "lokr",
483
+ "factor": 8,
484
+ "linear_dim": 1000000,
485
+ "linear_alpha": 1,
486
+ "full_matrix": true
487
+ },
488
+ "transformer_blocks.13.ff*": {
489
+ "algo": "lokr",
490
+ "factor": 8,
491
+ "linear_dim": 1000000,
492
+ "linear_alpha": 1,
493
+ "full_matrix": true
494
+ },
495
+ "transformer_blocks.13.*": {
496
+ "algo": "lokr",
497
+ "factor": 16,
498
+ "linear_dim": 1000000,
499
+ "linear_alpha": 1,
500
+ "full_matrix": true
501
+ },
502
+ "transformer_blocks.14.norm1*": {
503
+ "algo": "lokr",
504
+ "factor": 7,
505
+ "linear_dim": 1000000,
506
+ "linear_alpha": 1,
507
+ "full_matrix": true
508
+ },
509
+ "transformer_blocks.14.norm1_context*": {
510
+ "algo": "lokr",
511
+ "factor": 7,
512
+ "linear_dim": 1000000,
513
+ "linear_alpha": 1,
514
+ "full_matrix": true
515
+ },
516
+ "transformer_blocks.14.ff*": {
517
+ "algo": "lokr",
518
+ "factor": 7,
519
+ "linear_dim": 1000000,
520
+ "linear_alpha": 1,
521
+ "full_matrix": true
522
+ },
523
+ "transformer_blocks.14.*": {
524
+ "algo": "lokr",
525
+ "factor": 14,
526
+ "linear_dim": 1000000,
527
+ "linear_alpha": 1,
528
+ "full_matrix": true
529
+ },
530
+ "transformer_blocks.15.norm1*": {
531
+ "algo": "lokr",
532
+ "factor": 6,
533
+ "linear_dim": 1000000,
534
+ "linear_alpha": 1,
535
+ "full_matrix": true
536
+ },
537
+ "transformer_blocks.15.norm1_context*": {
538
+ "algo": "lokr",
539
+ "factor": 6,
540
+ "linear_dim": 1000000,
541
+ "linear_alpha": 1,
542
+ "full_matrix": true
543
+ },
544
+ "transformer_blocks.15.ff*": {
545
+ "algo": "lokr",
546
+ "factor": 6,
547
+ "linear_dim": 1000000,
548
+ "linear_alpha": 1,
549
+ "full_matrix": true
550
+ },
551
+ "transformer_blocks.15.*": {
552
+ "algo": "lokr",
553
+ "factor": 12,
554
+ "linear_dim": 1000000,
555
+ "linear_alpha": 1,
556
+ "full_matrix": true
557
+ },
558
+ "transformer_blocks.16.norm1*": {
559
+ "algo": "lokr",
560
+ "factor": 6,
561
+ "linear_dim": 1000000,
562
+ "linear_alpha": 1,
563
+ "full_matrix": true
564
+ },
565
+ "transformer_blocks.16.norm1_context*": {
566
+ "algo": "lokr",
567
+ "factor": 6,
568
+ "linear_dim": 1000000,
569
+ "linear_alpha": 1,
570
+ "full_matrix": true
571
+ },
572
+ "transformer_blocks.16.ff*": {
573
+ "algo": "lokr",
574
+ "factor": 6,
575
+ "linear_dim": 1000000,
576
+ "linear_alpha": 1,
577
+ "full_matrix": true
578
+ },
579
+ "transformer_blocks.16.*": {
580
+ "algo": "lokr",
581
+ "factor": 12,
582
+ "linear_dim": 1000000,
583
+ "linear_alpha": 1,
584
+ "full_matrix": true
585
+ },
586
+ "transformer_blocks.17.norm1*": {
587
+ "algo": "lokr",
588
+ "factor": 6,
589
+ "linear_dim": 1000000,
590
+ "linear_alpha": 1,
591
+ "full_matrix": true
592
+ },
593
+ "transformer_blocks.17.norm1_context*": {
594
+ "algo": "lokr",
595
+ "factor": 6,
596
+ "linear_dim": 1000000,
597
+ "linear_alpha": 1,
598
+ "full_matrix": true
599
+ },
600
+ "transformer_blocks.17.ff*": {
601
+ "algo": "lokr",
602
+ "factor": 6,
603
+ "linear_dim": 1000000,
604
+ "linear_alpha": 1,
605
+ "full_matrix": true
606
+ },
607
+ "transformer_blocks.17.*": {
608
+ "algo": "lokr",
609
+ "factor": 12,
610
+ "linear_dim": 1000000,
611
+ "linear_alpha": 1,
612
+ "full_matrix": true
613
+ },
614
+ "transformer_blocks.18.norm1*": {
615
+ "algo": "lokr",
616
+ "factor": 6,
617
+ "linear_dim": 1000000,
618
+ "linear_alpha": 1,
619
+ "full_matrix": true
620
+ },
621
+ "transformer_blocks.18.norm1_context*": {
622
+ "algo": "lokr",
623
+ "factor": 6,
624
+ "linear_dim": 1000000,
625
+ "linear_alpha": 1,
626
+ "full_matrix": true
627
+ },
628
+ "transformer_blocks.18.ff*": {
629
+ "algo": "lokr",
630
+ "factor": 6,
631
+ "linear_dim": 1000000,
632
+ "linear_alpha": 1,
633
+ "full_matrix": true
634
+ },
635
+ "transformer_blocks.18.*": {
636
+ "algo": "lokr",
637
+ "factor": 12,
638
+ "linear_dim": 1000000,
639
+ "linear_alpha": 1,
640
+ "full_matrix": true
641
+ },
642
+ "transformer_blocks.19.norm1*": {
643
+ "algo": "lokr",
644
+ "factor": 4,
645
+ "linear_dim": 1000000,
646
+ "linear_alpha": 1,
647
+ "full_matrix": true
648
+ },
649
+ "transformer_blocks.19.norm1_context*": {
650
+ "algo": "lokr",
651
+ "factor": 4,
652
+ "linear_dim": 1000000,
653
+ "linear_alpha": 1,
654
+ "full_matrix": true
655
+ },
656
+ "transformer_blocks.19.ff*": {
657
+ "algo": "lokr",
658
+ "factor": 4,
659
+ "linear_dim": 1000000,
660
+ "linear_alpha": 1,
661
+ "full_matrix": true
662
+ },
663
+ "transformer_blocks.19.*": {
664
+ "algo": "lokr",
665
+ "factor": 8,
666
+ "linear_dim": 1000000,
667
+ "linear_alpha": 1,
668
+ "full_matrix": true
669
+ },
670
+ "transformer_blocks.20.norm1*": {
671
+ "algo": "lokr",
672
+ "factor": 5,
673
+ "linear_dim": 1000000,
674
+ "linear_alpha": 1,
675
+ "full_matrix": true
676
+ },
677
+ "transformer_blocks.20.norm1_context*": {
678
+ "algo": "lokr",
679
+ "factor": 5,
680
+ "linear_dim": 1000000,
681
+ "linear_alpha": 1,
682
+ "full_matrix": true
683
+ },
684
+ "transformer_blocks.20.ff*": {
685
+ "algo": "lokr",
686
+ "factor": 5,
687
+ "linear_dim": 1000000,
688
+ "linear_alpha": 1,
689
+ "full_matrix": true
690
+ },
691
+ "transformer_blocks.20.*": {
692
+ "algo": "lokr",
693
+ "factor": 10,
694
+ "linear_dim": 1000000,
695
+ "linear_alpha": 1,
696
+ "full_matrix": true
697
+ },
698
+ "transformer_blocks.21.norm1*": {
699
+ "algo": "lokr",
700
+ "factor": 4,
701
+ "linear_dim": 1000000,
702
+ "linear_alpha": 1,
703
+ "full_matrix": true
704
+ },
705
+ "transformer_blocks.21.norm1_context*": {
706
+ "algo": "lokr",
707
+ "factor": 4,
708
+ "linear_dim": 1000000,
709
+ "linear_alpha": 1,
710
+ "full_matrix": true
711
+ },
712
+ "transformer_blocks.21.ff*": {
713
+ "algo": "lokr",
714
+ "factor": 4,
715
+ "linear_dim": 1000000,
716
+ "linear_alpha": 1,
717
+ "full_matrix": true
718
+ },
719
+ "transformer_blocks.21.*": {
720
+ "algo": "lokr",
721
+ "factor": 8,
722
+ "linear_dim": 1000000,
723
+ "linear_alpha": 1,
724
+ "full_matrix": true
725
+ },
726
+ "transformer_blocks.22.norm1*": {
727
+ "algo": "lokr",
728
+ "factor": 5,
729
+ "linear_dim": 1000000,
730
+ "linear_alpha": 1,
731
+ "full_matrix": true
732
+ },
733
+ "transformer_blocks.22.norm1_context*": {
734
+ "algo": "lokr",
735
+ "factor": 5,
736
+ "linear_dim": 1000000,
737
+ "linear_alpha": 1,
738
+ "full_matrix": true
739
+ },
740
+ "transformer_blocks.22.ff*": {
741
+ "algo": "lokr",
742
+ "factor": 5,
743
+ "linear_dim": 1000000,
744
+ "linear_alpha": 1,
745
+ "full_matrix": true
746
+ },
747
+ "transformer_blocks.22.*": {
748
+ "algo": "lokr",
749
+ "factor": 10,
750
+ "linear_dim": 1000000,
751
+ "linear_alpha": 1,
752
+ "full_matrix": true
753
+ },
754
+ "transformer_blocks.23.norm1*": {
755
+ "algo": "lokr",
756
+ "factor": 4,
757
+ "linear_dim": 1000000,
758
+ "linear_alpha": 1,
759
+ "full_matrix": true
760
+ },
761
+ "transformer_blocks.23.norm1_context*": {
762
+ "algo": "lokr",
763
+ "factor": 4,
764
+ "linear_dim": 1000000,
765
+ "linear_alpha": 1,
766
+ "full_matrix": true
767
+ },
768
+ "transformer_blocks.23.ff*": {
769
+ "algo": "lokr",
770
+ "factor": 4,
771
+ "linear_dim": 1000000,
772
+ "linear_alpha": 1,
773
+ "full_matrix": true
774
+ },
775
+ "transformer_blocks.23.*": {
776
+ "algo": "lokr",
777
+ "factor": 8,
778
+ "linear_dim": 1000000,
779
+ "linear_alpha": 1,
780
+ "full_matrix": true
781
+ },
782
+ "transformer_blocks.24.norm1*": {
783
+ "algo": "lokr",
784
+ "factor": 4,
785
+ "linear_dim": 1000000,
786
+ "linear_alpha": 1,
787
+ "full_matrix": true
788
+ },
789
+ "transformer_blocks.24.norm1_context*": {
790
+ "algo": "lokr",
791
+ "factor": 4,
792
+ "linear_dim": 1000000,
793
+ "linear_alpha": 1,
794
+ "full_matrix": true
795
+ },
796
+ "transformer_blocks.24.ff*": {
797
+ "algo": "lokr",
798
+ "factor": 4,
799
+ "linear_dim": 1000000,
800
+ "linear_alpha": 1,
801
+ "full_matrix": true
802
+ },
803
+ "transformer_blocks.24.*": {
804
+ "algo": "lokr",
805
+ "factor": 8,
806
+ "linear_dim": 1000000,
807
+ "linear_alpha": 1,
808
+ "full_matrix": true
809
+ },
810
+ "transformer_blocks.25.norm1*": {
811
+ "algo": "lokr",
812
+ "factor": 4,
813
+ "linear_dim": 1000000,
814
+ "linear_alpha": 1,
815
+ "full_matrix": true
816
+ },
817
+ "transformer_blocks.25.norm1_context*": {
818
+ "algo": "lokr",
819
+ "factor": 4,
820
+ "linear_dim": 1000000,
821
+ "linear_alpha": 1,
822
+ "full_matrix": true
823
+ },
824
+ "transformer_blocks.25.ff*": {
825
+ "algo": "lokr",
826
+ "factor": 4,
827
+ "linear_dim": 1000000,
828
+ "linear_alpha": 1,
829
+ "full_matrix": true
830
+ },
831
+ "transformer_blocks.25.*": {
832
+ "algo": "lokr",
833
+ "factor": 8,
834
+ "linear_dim": 1000000,
835
+ "linear_alpha": 1,
836
+ "full_matrix": true
837
+ },
838
+ "transformer_blocks.26.norm1*": {
839
+ "algo": "lokr",
840
+ "factor": 4,
841
+ "linear_dim": 1000000,
842
+ "linear_alpha": 1,
843
+ "full_matrix": true
844
+ },
845
+ "transformer_blocks.26.norm1_context*": {
846
+ "algo": "lokr",
847
+ "factor": 4,
848
+ "linear_dim": 1000000,
849
+ "linear_alpha": 1,
850
+ "full_matrix": true
851
+ },
852
+ "transformer_blocks.26.ff*": {
853
+ "algo": "lokr",
854
+ "factor": 4,
855
+ "linear_dim": 1000000,
856
+ "linear_alpha": 1,
857
+ "full_matrix": true
858
+ },
859
+ "transformer_blocks.26.*": {
860
+ "algo": "lokr",
861
+ "factor": 8,
862
+ "linear_dim": 1000000,
863
+ "linear_alpha": 1,
864
+ "full_matrix": true
865
+ },
866
+ "transformer_blocks.27.norm1*": {
867
+ "algo": "lokr",
868
+ "factor": 4,
869
+ "linear_dim": 1000000,
870
+ "linear_alpha": 1,
871
+ "full_matrix": true
872
+ },
873
+ "transformer_blocks.27.norm1_context*": {
874
+ "algo": "lokr",
875
+ "factor": 4,
876
+ "linear_dim": 1000000,
877
+ "linear_alpha": 1,
878
+ "full_matrix": true
879
+ },
880
+ "transformer_blocks.27.ff*": {
881
+ "algo": "lokr",
882
+ "factor": 4,
883
+ "linear_dim": 1000000,
884
+ "linear_alpha": 1,
885
+ "full_matrix": true
886
+ },
887
+ "transformer_blocks.27.*": {
888
+ "algo": "lokr",
889
+ "factor": 8,
890
+ "linear_dim": 1000000,
891
+ "linear_alpha": 1,
892
+ "full_matrix": true
893
+ },
894
+ "transformer_blocks.28.norm1*": {
895
+ "algo": "lokr",
896
+ "factor": 4,
897
+ "linear_dim": 1000000,
898
+ "linear_alpha": 1,
899
+ "full_matrix": true
900
+ },
901
+ "transformer_blocks.28.norm1_context*": {
902
+ "algo": "lokr",
903
+ "factor": 4,
904
+ "linear_dim": 1000000,
905
+ "linear_alpha": 1,
906
+ "full_matrix": true
907
+ },
908
+ "transformer_blocks.28.ff*": {
909
+ "algo": "lokr",
910
+ "factor": 4,
911
+ "linear_dim": 1000000,
912
+ "linear_alpha": 1,
913
+ "full_matrix": true
914
+ },
915
+ "transformer_blocks.28.*": {
916
+ "algo": "lokr",
917
+ "factor": 8,
918
+ "linear_dim": 1000000,
919
+ "linear_alpha": 1,
920
+ "full_matrix": true
921
+ },
922
+ "transformer_blocks.29.norm1*": {
923
+ "algo": "lokr",
924
+ "factor": 4,
925
+ "linear_dim": 1000000,
926
+ "linear_alpha": 1,
927
+ "full_matrix": true
928
+ },
929
+ "transformer_blocks.29.norm1_context*": {
930
+ "algo": "lokr",
931
+ "factor": 4,
932
+ "linear_dim": 1000000,
933
+ "linear_alpha": 1,
934
+ "full_matrix": true
935
+ },
936
+ "transformer_blocks.29.ff*": {
937
+ "algo": "lokr",
938
+ "factor": 4,
939
+ "linear_dim": 1000000,
940
+ "linear_alpha": 1,
941
+ "full_matrix": true
942
+ },
943
+ "transformer_blocks.29.*": {
944
+ "algo": "lokr",
945
+ "factor": 8,
946
+ "linear_dim": 1000000,
947
+ "linear_alpha": 1,
948
+ "full_matrix": true
949
+ },
950
+ "transformer_blocks.30.norm1*": {
951
+ "algo": "lokr",
952
+ "factor": 3,
953
+ "linear_dim": 1000000,
954
+ "linear_alpha": 1,
955
+ "full_matrix": true
956
+ },
957
+ "transformer_blocks.30.norm1_context*": {
958
+ "algo": "lokr",
959
+ "factor": 3,
960
+ "linear_dim": 1000000,
961
+ "linear_alpha": 1,
962
+ "full_matrix": true
963
+ },
964
+ "transformer_blocks.30.ff*": {
965
+ "algo": "lokr",
966
+ "factor": 3,
967
+ "linear_dim": 1000000,
968
+ "linear_alpha": 1,
969
+ "full_matrix": true
970
+ },
971
+ "transformer_blocks.30.*": {
972
+ "algo": "lokr",
973
+ "factor": 6,
974
+ "linear_dim": 1000000,
975
+ "linear_alpha": 1,
976
+ "full_matrix": true
977
+ },
978
+ "transformer_blocks.31.norm1*": {
979
+ "algo": "lokr",
980
+ "factor": 1,
981
+ "linear_dim": 1000000,
982
+ "linear_alpha": 1,
983
+ "full_matrix": true
984
+ },
985
+ "transformer_blocks.31.norm1_context*": {
986
+ "algo": "lokr",
987
+ "factor": 1,
988
+ "linear_dim": 1000000,
989
+ "linear_alpha": 1,
990
+ "full_matrix": true
991
+ },
992
+ "transformer_blocks.31.ff*": {
993
+ "algo": "lokr",
994
+ "factor": 1,
995
+ "linear_dim": 1000000,
996
+ "linear_alpha": 1,
997
+ "full_matrix": true
998
+ },
999
+ "transformer_blocks.31.*": {
1000
+ "algo": "lokr",
1001
+ "factor": 2,
1002
+ "linear_dim": 1000000,
1003
+ "linear_alpha": 1,
1004
+ "full_matrix": true
1005
+ },
1006
+ "transformer_blocks.32.norm1*": {
1007
+ "algo": "lokr",
1008
+ "factor": 4,
1009
+ "linear_dim": 1000000,
1010
+ "linear_alpha": 1,
1011
+ "full_matrix": true
1012
+ },
1013
+ "transformer_blocks.32.norm1_context*": {
1014
+ "algo": "lokr",
1015
+ "factor": 4,
1016
+ "linear_dim": 1000000,
1017
+ "linear_alpha": 1,
1018
+ "full_matrix": true
1019
+ },
1020
+ "transformer_blocks.32.ff*": {
1021
+ "algo": "lokr",
1022
+ "factor": 4,
1023
+ "linear_dim": 1000000,
1024
+ "linear_alpha": 1,
1025
+ "full_matrix": true
1026
+ },
1027
+ "transformer_blocks.32.*": {
1028
+ "algo": "lokr",
1029
+ "factor": 8,
1030
+ "linear_dim": 1000000,
1031
+ "linear_alpha": 1,
1032
+ "full_matrix": true
1033
+ },
1034
+ "transformer_blocks.33.norm1*": {
1035
+ "algo": "lokr",
1036
+ "factor": 3,
1037
+ "linear_dim": 1000000,
1038
+ "linear_alpha": 1,
1039
+ "full_matrix": true
1040
+ },
1041
+ "transformer_blocks.33.norm1_context*": {
1042
+ "algo": "lokr",
1043
+ "factor": 3,
1044
+ "linear_dim": 1000000,
1045
+ "linear_alpha": 1,
1046
+ "full_matrix": true
1047
+ },
1048
+ "transformer_blocks.33.ff*": {
1049
+ "algo": "lokr",
1050
+ "factor": 3,
1051
+ "linear_dim": 1000000,
1052
+ "linear_alpha": 1,
1053
+ "full_matrix": true
1054
+ },
1055
+ "transformer_blocks.33.*": {
1056
+ "algo": "lokr",
1057
+ "factor": 6,
1058
+ "linear_dim": 1000000,
1059
+ "linear_alpha": 1,
1060
+ "full_matrix": true
1061
+ },
1062
+ "transformer_blocks.34.norm1*": {
1063
+ "algo": "lokr",
1064
+ "factor": 3,
1065
+ "linear_dim": 1000000,
1066
+ "linear_alpha": 1,
1067
+ "full_matrix": true
1068
+ },
1069
+ "transformer_blocks.34.norm1_context*": {
1070
+ "algo": "lokr",
1071
+ "factor": 3,
1072
+ "linear_dim": 1000000,
1073
+ "linear_alpha": 1,
1074
+ "full_matrix": true
1075
+ },
1076
+ "transformer_blocks.34.ff*": {
1077
+ "algo": "lokr",
1078
+ "factor": 3,
1079
+ "linear_dim": 1000000,
1080
+ "linear_alpha": 1,
1081
+ "full_matrix": true
1082
+ },
1083
+ "transformer_blocks.34.*": {
1084
+ "algo": "lokr",
1085
+ "factor": 6,
1086
+ "linear_dim": 1000000,
1087
+ "linear_alpha": 1,
1088
+ "full_matrix": true
1089
+ },
1090
+ "transformer_blocks.35.norm1*": {
1091
+ "algo": "lokr",
1092
+ "factor": 1,
1093
+ "linear_dim": 1000000,
1094
+ "linear_alpha": 1,
1095
+ "full_matrix": true
1096
+ },
1097
+ "transformer_blocks.35.norm1_context*": {
1098
+ "algo": "lokr",
1099
+ "factor": 1,
1100
+ "linear_dim": 1000000,
1101
+ "linear_alpha": 1,
1102
+ "full_matrix": true
1103
+ },
1104
+ "transformer_blocks.35.ff*": {
1105
+ "algo": "lokr",
1106
+ "factor": 1,
1107
+ "linear_dim": 1000000,
1108
+ "linear_alpha": 1,
1109
+ "full_matrix": true
1110
+ },
1111
+ "transformer_blocks.35.*": {
1112
+ "algo": "lokr",
1113
+ "factor": 2,
1114
+ "linear_dim": 1000000,
1115
+ "linear_alpha": 1,
1116
+ "full_matrix": true
1117
+ },
1118
+ "transformer_blocks.36.norm1*": {
1119
+ "algo": "lokr",
1120
+ "factor": 3,
1121
+ "linear_dim": 1000000,
1122
+ "linear_alpha": 1,
1123
+ "full_matrix": true
1124
+ },
1125
+ "transformer_blocks.36.norm1_context*": {
1126
+ "algo": "lokr",
1127
+ "factor": 3,
1128
+ "linear_dim": 1000000,
1129
+ "linear_alpha": 1,
1130
+ "full_matrix": true
1131
+ },
1132
+ "transformer_blocks.36.ff*": {
1133
+ "algo": "lokr",
1134
+ "factor": 3,
1135
+ "linear_dim": 1000000,
1136
+ "linear_alpha": 1,
1137
+ "full_matrix": true
1138
+ },
1139
+ "transformer_blocks.36.*": {
1140
+ "algo": "lokr",
1141
+ "factor": 6,
1142
+ "linear_dim": 1000000,
1143
+ "linear_alpha": 1,
1144
+ "full_matrix": true
1145
+ },
1146
+ "transformer_blocks.37.norm1*": {
1147
+ "algo": "lokr",
1148
+ "factor": 3,
1149
+ "linear_dim": 1000000,
1150
+ "linear_alpha": 1,
1151
+ "full_matrix": true
1152
+ },
1153
+ "transformer_blocks.37.norm1_context*": {
1154
+ "algo": "lokr",
1155
+ "factor": 3,
1156
+ "linear_dim": 1000000,
1157
+ "linear_alpha": 1,
1158
+ "full_matrix": true
1159
+ },
1160
+ "transformer_blocks.37.ff*": {
1161
+ "algo": "lokr",
1162
+ "factor": 3,
1163
+ "linear_dim": 1000000,
1164
+ "linear_alpha": 1,
1165
+ "full_matrix": true
1166
+ },
1167
+ "transformer_blocks.37.*": {
1168
+ "algo": "lokr",
1169
+ "factor": 6,
1170
+ "linear_dim": 1000000,
1171
+ "linear_alpha": 1,
1172
+ "full_matrix": true
1173
  }
1174
+ },
1175
+ "use_fnmatch": true
1176
  }
1177
  }
1178
  ```