metadata
library_name: transformers
tags:
- not-for-all-audiences
model-index:
- name: OpenCrystal-12B-L3
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 40.71
name: strict accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Darkknight535/OpenCrystal-12B-L3
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 31.84
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Darkknight535/OpenCrystal-12B-L3
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 7.93
name: exact match
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Darkknight535/OpenCrystal-12B-L3
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 7.49
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Darkknight535/OpenCrystal-12B-L3
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 5.74
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Darkknight535/OpenCrystal-12B-L3
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 29.34
name: accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Darkknight535/OpenCrystal-12B-L3
name: Open LLM Leaderboard
OpenCrystal-12B-L3
This is a finetuned language model. (I recommend using this one v2 and v2.1 are not good enough)
128K??
Instruct Template
Default llama3 instruct and context preset, but here is the one i use. Instruct Context
Samplers
Creative
Temp : 1.23
Min P : 0.05
Repetition Penalty : 1.05
[And everything else neutral]
Normal
Temp : 0.6 - 0.8
Min P : 0.1
Repetition Penalty : 1.1
[And everything else neutral]
Pro Tip
- You can uncheck Include Names option in sillytavern, to force it to speak as others dynamically. Not Recommended
Features
- Can speak as other npc automatically.
- Creative (Swipes are crazy.)
- Coherent (Sometime gets horny)
- Output feels like you're using Character.ai
- Follows prompt better
- Likes higher context length. (12K easily tested)
- can summarize and generate image prompts well [The Above image's prompt is generated in a roleplay by this model] (Possible : Due to llama-3-instruct as base)
Instruct Prompt
You're {{char}}, follow {{char}} personality and plot of the story, Don't impersonate as {{user}}, Speak as others NPC except {{user}} when needed. Be Creative, Create various interesting events and situations during the story.
FeedBack
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 20.51 |
| IFEval (0-Shot) | 40.71 |
| BBH (3-Shot) | 31.84 |
| MATH Lvl 5 (4-Shot) | 7.93 |
| GPQA (0-shot) | 7.49 |
| MuSR (0-shot) | 5.74 |
| MMLU-PRO (5-shot) | 29.34 |