Why does it generate nothing but garbage?
Using llama.cpp build 2993 (902184dd) with default parameters and a simple prompt, this model generates garbage:main -m "dolphin-2.9.1-mixtral-1x22b-Q6_K.gguf" --escape --prompt "<|im_start|> system\nBe concise!<|im_end|>\n<|im_start|> user\nWho are you?<|im_end|>\n<|im_start|> assistant\n"
output:
sampling:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 512, n_batch = 2048, n_predict = -1, n_keep = 1
<s><|im_start|> system
Be concise!<|im_end|>
<|im_start|> user
Who are you?<|im_end|>
<|im_start|> assistant
, Sategorating Themarate,
104
Themarate Themarate
,347,401,14
1,7,0,3,64,3,,3,3,73,76, 37,58,
ate,
,3,71,
,5,
ar,353,12,378,
ose,3,6,30,73,140,346,17577,
2,4,73
,ose,
2.54)
... and so on.
Is there any magic trick?
It's possible that this converted successfully but not properly since it's a very odd architecture (a single expert), I'll take a look in a bit to see if I can figure anything out
@newsletter The Dolphin of Llama3 does kind of the same thing when using ChatML template, but is fixed when using the original Llama3 template. I can't test it right now myself, but you could try using the Mistral template and see if it helps. It may not though, but it is easy to test.
@newsletter The Dolphin of Llama3 does kind of the same thing when using ChatML template, but is fixed when using the original Llama3 template. I can't test it right now myself, but you could try using the Mistral template and see if it helps. It may not though, but it is easy to test.
Yes, indeed, the Mistral-22B-v0.2 prompt template really works like a charm!
main -m "dolphin-2.9.1-mixtral-1x22b-Q6_K.gguf" --escape --prompt "### System: Be concise!\n### Human: Who are you?\n### Assistant:"
output:
<s> ### System: Be concise!
### Human: Who are you?
### Assistant: I am a text-based AI model designed to provide useful information and assistance to users in a concise manner.<|im_end|> [end of text]
The Llama-3 prompt template also works fine, by the way.