Poor Model Performance with Recommended Quantized Model
#21
by
nlpsingh
- opened
I am using the mistral-7b-v0.1.Q4_K_M.gguf with ctransformers and langchain and I am noticing very poor performance. I am not sure if I am doing something incorrect from my end but the model does not seem to even be able to handle the simplest of inputs. For example:
I am getting responses such as this to a basic query like "hi":
Is there anything I am missing or doing incorrectly in my usage of the model?
Could you please also include the full input in your screenshot? While the TEMPLATE in your code appears correct, the truncated portion in the screenshot seems a bit unclear to me.