💫 Draco-8x7B

This is the model for Draco-8x7B. I used this repo to make this MOE model.

This model's experts are not using any merged models.

📚 Other branches (Number of Experts Per Token)

Other branches that this repository contains differ only slightly (from a git diff perspective) in terms of the number of experts per token.

Usually, a higher value for the number of experts per token will result in better performance, but it may also lead to increased inference time.

Number of experts per token	Link of the branch
2	Main
3	3-experts-per-token
4	4-experts-per-token
6	6-experts-per-token
8	8-experts-per-token

💬 Prompt Template(s):

This model includes many models, so providing only one prompt template is not enough. You can use and try these prompt templates and decide which works best for you.

Note: The current chat template in the tokenizer config is set to openchat-3.5-0106's chat template.

Note 2: It is also important to note that jondurbin/bagel-dpo-7b-v0.1 is using many prompt templates other than I provided. You can visit jondurbin/bagel-dpo-7b-v0.1 to learn more about this templates.

GPT4 Correct

Used in openchat/openchat-3.5-0106, beowolx/CodeNinja-1.0-OpenChat-7B

GPT4 Correct User: {user}<|end_of_turn|>GPT4 Correct Assistant: {asistant}<|end_of_turn|>

ChatML:

Used in teknium/OpenHermes-2.5-Mistral-7B, jondurbin/bagel-dpo-7b-v0.1, cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser, senseable/WestLake-7B-v2

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{asistant}<|im_end|>

Math Alpaca

Used in meta-math/MetaMath-Mistral-7B

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response: Let's think step by step.

🛠️ Yaml Config

See config

base_model: openchat/openchat-3.5-0106
gate_mode: hidden
dtype: bfloat16

experts:
  - source_model: openchat/openchat-3.5-0106
    positive_prompts: # General (Mistral finetune)
    - "chat"
    - "assistant"
    - "tell me"
    - "explain"

  - source_model: teknium/OpenHermes-2.5-Mistral-7B
    positive_prompts: # General (Mistral finetune)
    - "interact"
    - "converse"
    - "respond"
    - "express"

  - source_model: jondurbin/bagel-dpo-7b-v0.1
    positive_prompts: # Science (Mistral finetune)
    - "science"
    - "biology"
    - "chemistry"
    - "physics"
    - "Newton's laws"
    - "scientific method"
    - "periodic table"
    - "photosynthesis process"

  - source_model: meta-math/MetaMath-Mistral-7B
    positive_prompts: # Math (Mistral finetune)
    - "reason"
    - "math"
    - "mathematics"
    - "solve"
    - "count"

  - source_model: cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser
    positive_prompts: # Uncensored (Mistral finetune)
    - "dolphin"
    - "uncensored"
    - "unbiased"
    - "unfiltered"
    - "unrestricted"
    - "offensive"

  - source_model: beowolx/CodeNinja-1.0-OpenChat-7B
    positive_prompts: # Code (openchat-3.5-1210 finetune)
    - "code"
    - "script"
    - "python"
    - "javascript"
    - "programming"
    - "algorithm"

  - source_model: senseable/WestLake-7B-v2
    positive_prompts: # Roleplay (Unknown finetune)
    - "storywriting"
    - "write"
    - "scene"
    - "story"
    - "character"
    - "act as"
    - "you are"

  - source_model: snorkelai/Snorkel-Mistral-PairRM-DPO
    positive_prompts: # Question Answering (? Mistral-7B-Instruct-v0.2 finetune ?)
    - "what happens"
    - "what is"
    - "what can"
    - "why"
    - "who"
    - "can a"

🔄 Quantizationed versions

Quantizationed versions of this model is available thanks to TheBloke.

GPTQ

• TheBloke/Draco-8x7B-GPTQ

GGUF

• TheBloke/Draco-8x7B-GGUF

AWQ

• TheBloke/Draco-8x7B-AWQ

If you would like to support me:

☕ Buy Me a Coffee

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	70.89
AI2 Reasoning Challenge (25-Shot)	65.02
HellaSwag (10-Shot)	85.24
MMLU (5-Shot)	64.96
TruthfulQA (0-shot)	62.65
Winogrande (5-shot)	80.66
GSM8k (5-shot)	66.79