Has anyone tried this gguf with agentic framework?

#6
by BigDeeper - opened

Pythogora (aka gpt-pilot) or Devika?

Cognitive Computations org

If someone were to, I recommend they use the 256k version that we recently posted. It has function calling and agentic abilities in line with: https://huggingface.co/datasets/internlm/Agent-FLAN. Should work pretty well, especially with some good prompting.

This comment has been hidden

If someone were to, I recommend they use the 256k version that we recently posted. It has function calling and agentic abilities in line with: https://huggingface.co/datasets/internlm/Agent-FLAN. Should work pretty well, especially with some good prompting.

I have just tried it, the 256k version, after converting it to Q8_0 version. Unfortunately it does not appear capable of operating stably within gpt-pilot. It gets "confused," promotes looping behavior. It seems to operate normally using ollama cli though. Llama-3-70b-Instruct.Q5_M's operation is much more stable, but it is 50GB and is very slow.

Sign up or log in to comment