Correct template for use with this model
I try to use preset with LM Studio but model generate few token and then stop generate
I also convert to ollama with following Modelfile, but got the same results
there is no problem for v2 model
FROM seallm-7b-v2.5.Q4_K_M.gguf
TEMPLATE """<|im_start|>system
{{ .System }}<eos>
<|im_start|>user
{{ .Prompt }}<eos>
<|im_start|>assistant
"""
SYSTEM """You are a helpful assistant."""
PARAMETER stop "<eos>"
PARAMETER stop "<|im_start|>"
Please suggest me
@saknarak If you're using llama.cpp, there are few BIG notes:
<bos>should always at the beginning, so in case ollama doesn't add it implicitly, you may try adding it with template
TEMPLATE """<bos><|im_start|>system
{{ .System }}<eos>
<|im_start|>user
{{ .Prompt }}<eos>
<|im_start|>assistant
"""
You must set
--repeat-penalty 1.0for llama.cpp, old versions of llama.cpp set it to 1.1, which break gemma model.What was the max new tokens used in ollama? Should set it to 1024.
If you have tried LM-studio with seallm-7b-v2.5-chatml.Q4_K_M.gguf using the ChatML format instead?
let me know if you still run into problem.
Thank you for useful information.
https://github.com/ollama/ollama/blob/main/docs/modelfile.md
But still no luck :(
This is what I have try
- with/without
- set repeat_penalty to 1
- set num_predict to 1024
- change gguf model to seallm-7b-v2.5-chatml.Q4_K_M.gguf
All of 1-4 in combinations, none of them is work.
Modelfile
FROM ./Publisher/Repository/seallm-7b-v2.5.Q4_K_M.gguf
TEMPLATE """<bos><|im_start|>system
{{ .System }}<eos>
<|im_start|>user
{{ .Prompt }}<eos>
<|im_start|>assistant
"""
SYSTEM """You are a helpful assistant."""
PARAMETER stop "<eos>"
PARAMETER stop "<|im_start|>"
PARAMETER temperature 1
PARAMETER repeat_penalty 1
PARAMETER num_predict 1024
seallm-7b-v2.5-chatml.Q4_K_M.gguf is the same result
Only SeaLLM 2.0 is working.