Initial release

.gitattributes

@@ -4,6 +4,7 @@
 *.bz2 filter=lfs diff=lfs merge=lfs -text
 *.ckpt filter=lfs diff=lfs merge=lfs -text
 *.ftz filter=lfs diff=lfs merge=lfs -text
+*.gguf filter=lfs diff=lfs merge=lfs -text
 *.gz filter=lfs diff=lfs merge=lfs -text
 *.h5 filter=lfs diff=lfs merge=lfs -text
 *.joblib filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,48 @@
 ---
+base_model:
+- grimjim/kukulemon-32K-7B
+- grimjim/rogue-enchantress-32k-7B
+library_name: transformers
+tags:
+- mergekit
+- merge
 license: cc-by-nc-4.0
+pipeline_tag: text-generation
 ---
+# kukulemon-v3-soul_mix-32k-7B
+
+This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
+
+We explore merger at extremely low weight as an alternative to fine-tuning. The additional model was applied at a weight of 10e-5, which was selected to be comparable to a few epochs of training. The low weight also amounts to the additional model being flattened, though technically not sparsified.
+
+- [Full weights](https://huggingface.co/grimjim/kukulemon-v3-soul_mix-32k-7B)
+- [GGUF quants](https://huggingface.co/grimjim/kukulemon-v3-soul_mix-32k-7B-GGUF)
+
+## Merge Details
+### Merge Method
+
+This model was merged using the [task arithmetic](https://arxiv.org/abs/2212.04089) merge method using [grimjim/kukulemon-32K-7B](https://huggingface.co/grimjim/kukulemon-32K-7B) as a base.
+
+### Models Merged
+
+The following model was included in the merge:
+* [grimjim/rogue-enchantress-32k-7B](https://huggingface.co/grimjim/rogue-enchantress-32k-7B)
+
+### Configuration
+
+The following YAML configuration was used to produce this model:
+
+```yaml
+base_model: grimjim/kukulemon-32K-7B
+dtype: bfloat16
+merge_method: task_arithmetic
+slices:
+- sources:
+  - layer_range: [0, 32]
+    model: grimjim/kukulemon-32K-7B
+  - layer_range: [0, 32]
+    model: grimjim/rogue-enchantress-32k-7B
+    parameters:
+      weight: 10e-5
+
+```

kukulemon-v3-soul_mix-32k-7B.Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0e2dc06a0f84ae5fe4ffe5a524df1d94a237f9f772b22a68408990e06e90d38
+size 4368439072

kukulemon-v3-soul_mix-32k-7B.Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6728b94bd5df709d990b6f9222fac1d92b1873e39e4c9a2e925892497e428e0
+size 5131409184

kukulemon-v3-soul_mix-32k-7B.Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfe71c6c9e0d208b5c276764d66e6f4c412706afff7a4ff611664b069fe571dd
+size 5942064928

kukulemon-v3-soul_mix-32k-7B.Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:657ec991b22bc90bdae3b676f336ee9b2be66fef202b8efcffb6f397d6fa4ec0
+size 7695857440