Commit
•
619d7e8
1
Parent(s):
536522b
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,5 +1,8 @@
|
|
| 1 |
-
|
| 2 |
|
|
|
|
|
|
|
|
|
|
| 3 |
```yaml
|
| 4 |
merge_method: passthrough
|
| 5 |
slices:
|
|
@@ -34,7 +37,7 @@ slices:
|
|
| 34 |
- Layers 0 to 18 are original
|
| 35 |
- Layers 19 to 41 are duplicated, zero'd out, and put in the middle twice
|
| 36 |
- Layers 42 to 54 are original
|
| 37 |
-
- down_proj and o_proj layers for the duplicated part have been nulled and will require healing to 'unignore' the added layers
|
| 38 |
|
| 39 |
```
|
| 40 |
[ Unique ][ Duplicated ][ Unique ]
|
|
@@ -42,7 +45,7 @@ slices:
|
|
| 42 |
34.5% 41.8% 23.7%
|
| 43 |
```
|
| 44 |
|
| 45 |
-
# Weight
|
| 46 |
- Nemo x Rocinante
|
| 47 |
- Small x Cydonia
|
| 48 |
- 39B Upscale x Tunguska 1 Epoch
|
|
|
|
| 1 |
+
# Upscaled Tuning Experiment
|
| 2 |
|
| 3 |
+
## What is the 39B Upscale?
|
| 4 |
+
|
| 5 |
+
https://huggingface.co/TheSkullery/BA-Zephyria-39b
|
| 6 |
```yaml
|
| 7 |
merge_method: passthrough
|
| 8 |
slices:
|
|
|
|
| 37 |
- Layers 0 to 18 are original
|
| 38 |
- Layers 19 to 41 are duplicated, zero'd out, and put in the middle twice
|
| 39 |
- Layers 42 to 54 are original
|
| 40 |
+
- **down_proj** and **o_proj** layers for the duplicated part have been nulled and will require healing to 'unignore' the added layers
|
| 41 |
|
| 42 |
```
|
| 43 |
[ Unique ][ Duplicated ][ Unique ]
|
|
|
|
| 45 |
34.5% 41.8% 23.7%
|
| 46 |
```
|
| 47 |
|
| 48 |
+
# Weight Difference Visualization
|
| 49 |
- Nemo x Rocinante
|
| 50 |
- Small x Cydonia
|
| 51 |
- 39B Upscale x Tunguska 1 Epoch
|