File size: 6,363 Bytes
eaf84a6 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 |
---
license: apache-2.0
library_name: transformers
tags:
- general-purpose
- roleplay
- storywriting
- merge
- finetune
base_model: mistralai/Mistral-Nemo-Base-2407
model-index:
- name: Chronos-Gold-12B-1.0
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 31.66
name: strict accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 35.91
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 4.38
name: exact match
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 9.06
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 19.42
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 27.98
name: accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=elinas/Chronos-Gold-12B-1.0
name: Open LLM Leaderboard
---
# Chronos Gold 12B-1.0
Chronos Gold 12B 1.0 is a very unique model that applies to domain areas such as
general chatbot functionatliy, *roleplay*, and storywriting. The model has been observed to write up to 2250 tokens in a single sequence. The model was trained at a
sequence length of 16384 (16k) and will still retain the *apparent* 128k context length from Mistral-Nemo, though it deteriorates over time like regular Nemo does based on
the [RULER Test](https://github.com/hsiehjackson/RULER?tab=readme-ov-file#-ruler-whats-the-real-context-size-of-your-long-context-language-models)
As a result, is recommended to keep your sequence length max at 16384, or you will experience performance degredation.
The base model is `mistralai/Mistral-Nemo-Base-2407` which was heavily modified to produce a more coherent model, comparable to much larger models.
**Chronos Gold 12B-1.0** re-creates the uniqueness of the original Chronos with significiantly enhanced prompt adherence (following), coherence, a modern dataset, as well as supporting a majority of "character card" formats in applications like SillyTavern.
It went through an iterative and objective merge process as my previous models and was further finetuned on a dataset curated for it.
The specifics of the model will not be disclosed at the time due to dataset ownership.
## Instruct Template
This model uses `ChatML` - below is an example. It is a preset in many frontends.
```
<|im_start|>system
A system prompt describing how you'd like your bot to act.<|im_end|>
<|im_start|>user
Hello there!<|im_end|>
<|im_start|>assistant
I can assist you or we can discuss other things?<|im_end|>
<|im_start|>user
I was wondering how transformers work?<|im_end|>
<|im_start|>assistant
```
## Quantization
#### LlamaCPP
[GGUFs from @bartowski](https://huggingface.co/bartowski/Chronos-Gold-12B-1.0-GGUF)
#### Exllama2
[4.5bpw by @Pyroserenus](https://huggingface.co/Pyroserenus/Chronos-Gold-12B-1.0-4.5bpw-h6-exl2)
[5.5bpw by @Pyroserenus](https://huggingface.co/Pyroserenus/Chronos-Gold-12B-1.0-5.5bpw-h6-exl2)
[6.5bpw by @Pyroserenus](https://huggingface.co/Pyroserenus/Chronos-Gold-12B-1.0-6.5bpw-h6-exl2)
[8.0bpw by @Pyroserenus](https://huggingface.co/Pyroserenus/Chronos-Gold-12B-1.0-8.0bpw-h8-exl2)
#### FP8
[FP8 Quant by @Pyroserenus](https://huggingface.co/Pyroserenus/Chronos-Gold-12B-1.0-FP8)
## Sampling Settings
Nemo is a bit sensitive to high temperatures, so I use lower. Here are my settings:
```
Temp - 0.7 (0.9 max)
Presence Penalty - 1.0
Repetition Penalty range - 2800
Min P - 0.10
```
## Additional Details
This model was created by **elinas** on discord. Thank you to [@kalomaze](https://huggingface.co/kalomaze) for providing a model that made this merge possible.
This is one of multiple models to come out in the series by size and model architecture, so look forward to it!
Contact me on Discord for inquiries.
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_elinas__Chronos-Gold-12B-1.0)
| Metric |Value|
|-------------------|----:|
|Avg. |21.40|
|IFEval (0-Shot) |31.66|
|BBH (3-Shot) |35.91|
|MATH Lvl 5 (4-Shot)| 4.38|
|GPQA (0-shot) | 9.06|
|MuSR (0-shot) |19.42|
|MMLU-PRO (5-shot) |27.98| |