uploaded readme

README.md

@@ -0,0 +1,176 @@
+Quantization made by Richard Erkhov.
+
+[Github](https://github.com/RichardErkhov)
+
+[Discord](https://discord.gg/pvy7H8DZMG)
+
+[Request more models](https://github.com/RichardErkhov/quant_request)
+
+
+DolphinStar-12.5B - GGUF
+- Model creator: https://huggingface.co/Noodlz/
+- Original model: https://huggingface.co/Noodlz/DolphinStar-12.5B/
+
+
+| Name | Quant method | Size |
+| ---- | ---- | ---- |
+| [DolphinStar-12.5B.Q2_K.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q2_K.gguf) | Q2_K | 4.33GB |
+| [DolphinStar-12.5B.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.IQ3_XS.gguf) | IQ3_XS | 4.81GB |
+| [DolphinStar-12.5B.IQ3_S.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.IQ3_S.gguf) | IQ3_S | 5.07GB |
+| [DolphinStar-12.5B.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q3_K_S.gguf) | Q3_K_S | 5.04GB |
+| [DolphinStar-12.5B.IQ3_M.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.IQ3_M.gguf) | IQ3_M | 5.24GB |
+| [DolphinStar-12.5B.Q3_K.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q3_K.gguf) | Q3_K | 5.62GB |
+| [DolphinStar-12.5B.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q3_K_M.gguf) | Q3_K_M | 5.62GB |
+| [DolphinStar-12.5B.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q3_K_L.gguf) | Q3_K_L | 6.11GB |
+| [DolphinStar-12.5B.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.IQ4_XS.gguf) | IQ4_XS | 6.3GB |
+| [DolphinStar-12.5B.Q4_0.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q4_0.gguf) | Q4_0 | 6.57GB |
+| [DolphinStar-12.5B.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.IQ4_NL.gguf) | IQ4_NL | 6.64GB |
+| [DolphinStar-12.5B.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q4_K_S.gguf) | Q4_K_S | 6.62GB |
+| [DolphinStar-12.5B.Q4_K.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q4_K.gguf) | Q4_K | 6.99GB |
+| [DolphinStar-12.5B.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q4_K_M.gguf) | Q4_K_M | 6.99GB |
+| [DolphinStar-12.5B.Q4_1.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q4_1.gguf) | Q4_1 | 7.29GB |
+| [DolphinStar-12.5B.Q5_0.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q5_0.gguf) | Q5_0 | 8.01GB |
+| [DolphinStar-12.5B.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q5_K_S.gguf) | Q5_K_S | 8.01GB |
+| [DolphinStar-12.5B.Q5_K.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q5_K.gguf) | Q5_K | 8.22GB |
+| [DolphinStar-12.5B.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q5_K_M.gguf) | Q5_K_M | 8.22GB |
+| [DolphinStar-12.5B.Q5_1.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q5_1.gguf) | Q5_1 | 8.73GB |
+| [DolphinStar-12.5B.Q6_K.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q6_K.gguf) | Q6_K | 9.53GB |
+| [DolphinStar-12.5B.Q8_0.gguf](https://huggingface.co/RichardErkhov/Noodlz_-_DolphinStar-12.5B-gguf/blob/main/DolphinStar-12.5B.Q8_0.gguf) | Q8_0 | 12.35GB |
+
+
+
+
+Original model description:
+---
+license: apache-2.0
+---
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/63cf23cffbd0cc580bc65c73/QDvxvuS3M7oHv7JI5d1ke.png)
+
+Custom Model "Dolphin2Star1" Merged by Noodlz.
+12.5B linear merged from the uncensored mistral 7B v0.2 as the base, with the fine tunes of StarlingLM 7B Beta that's originally mistral 7B v0.1
+
+have fun =)
+
+
+
+[EDIT] - preset wise it seems like it likes the "ChatML" format.
+[EDIT 2] - Usage Notes - model is sorta picky with the batch size and prompt preset/template. (maybe because merge of ChatML and OpenChat models)
+
+My current recommended setting & findings
+- Using LM Studio - use the default preset. GPU acceleration to max. prompt eval size to 1024, context length to 32768. this yields me decent, coherant results. ChatML works too but occasionall spits up odd texts after a couple of turns.
+- Using Oobabooga (Windows PC) - runs well using run-in-4bit along with use_flash_attention_2. default presets and everything works just fine.
+- Using OobaBooga (Mac) - [investigating]
+
+
+
+
+## Instructions Template:
+```
+{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{{ '<s>' }}{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}
+```
+
+
+## Chat Template:
+```
+{%- for message in messages %}
+    {%- if message['role'] == 'system' -%}
+        {%- if message['content'] -%}
+            {{- message['content'] + '\n\n' -}}
+        {%- endif -%}
+        {%- if user_bio -%}
+            {{- user_bio + '\n\n' -}}
+        {%- endif -%}
+    {%- else -%}
+        {%- if message['role'] == 'user' -%}
+            {{- name1 + ': ' + message['content'] + '\n'-}}
+        {%- else -%}
+            {{- name2 + ': ' + message['content'] + '\n' -}}
+        {%- endif -%}
+    {%- endif -%}
+{%- endfor -%}
+```
+
+
+
+
+---
+license: apache-2.0
+---
+
+
+---
+base_model:
+- cognitivecomputations/dolphin-2.8-mistral-7b-v02
+- NexusFlow/Starling-LM-7B-beta
+library_name: transformers
+tags:
+- mergekit
+- merge
+
+---
+# output_folder
+
+This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
+
+## Merge Details
+### Merge Method
+
+This model was merged using the [linear](https://arxiv.org/abs/2203.05482) merge method.
+
+### Models Merged
+
+The following models were included in the merge:
+* [cognitivecomputations/dolphin-2.8-mistral-7b-v02](https://huggingface.co/cognitivecomputations/dolphin-2.8-mistral-7b-v02)
+* [NexusFlow/Starling-LM-7B-beta](https://huggingface.co/NexusFlow/Starling-LM-7B-beta)
+
+### Configuration
+
+The following YAML configuration was used to produce this model:
+
+```yaml
+merge_method: linear
+parameters:
+  weight: 1.0
+slices:
+  - sources:
+      - model: cognitivecomputations/dolphin-2.8-mistral-7b-v02
+        layer_range: [0,1]
+      - model: NexusFlow/Starling-LM-7B-beta
+        layer_range: [0,1]
+        parameters: 
+          weight: 0
+  - sources:
+      - model: cognitivecomputations/dolphin-2.8-mistral-7b-v02
+        layer_range: [1,8]        
+  - sources:
+      - model: NexusFlow/Starling-LM-7B-beta
+        layer_range: [4,12]
+  - sources:
+      - model: cognitivecomputations/dolphin-2.8-mistral-7b-v02
+        layer_range: [8,16]        
+  - sources:
+      - model: NexusFlow/Starling-LM-7B-beta
+        layer_range: [12,20]  
+  - sources:
+      - model: cognitivecomputations/dolphin-2.8-mistral-7b-v02
+        layer_range: [16,24]        
+  - sources:
+      - model: NexusFlow/Starling-LM-7B-beta
+        layer_range: [20,28]
+  - sources:
+      - model: cognitivecomputations/dolphin-2.8-mistral-7b-v02
+        layer_range: [24,31]        
+  - sources:
+      - model: cognitivecomputations/dolphin-2.8-mistral-7b-v02
+        layer_range: [31,32]
+      - model: NexusFlow/Starling-LM-7B-beta
+        layer_range: [31,32]
+        parameters: 
+          weight: 0          
+dtype: float16
+tokenizer_source: model:cognitivecomputations/dolphin-2.8-mistral-7b-v02
+```
+