djward888

/

llava-llama-3-8b-v1_1-Q8_0-GGUF

like 6

I'm sorry, I don't know what an mmproj file is. Would you mind explaining?
Yes, it ran great! I haven't had any issues.

@djward888 Thanks for uploading this! I think people keep asking for the mmproj but then not explaining because the mmproj file isn't described anywhere.

I think its a CLIP model that's needed for multimodal inference with llama.cpp

The --mmproj command line argument is needed for both llava-cli and older llama.cpp before it was removed and probably other things built on top of llama.cpp.

Here's another comment abou it.

FYI: to utilize multimodality you have to specify a compatible model (in this case llava 7b) and its belonging mmproj model. The mmproj has to be in f-16

How are you able to run this without an mmproj file?

I tried the file from here with your GGUF and am only getting non-sensical results. Repetition and hallucinations, with almost nothing about the actual image.

You're welcome!
Oh I see, that finally makes sense. Thanks so much for explaining! I haven't actually tried it with images, but I've been running it for normal coding in Jan AI. It works quite well for text generation, I just have no need for multimodality.
How did I run it? I just imported the GGUF into Jan. I'm really not an expert on why it does or doesn't work though.

Ah, because this is a llava model, I assumed you were also using it for multimodal inference.

I noticed this model is using token

128001 '<|end_of_text|>

for its EOS token instead of

128009 '<|eot_id|>'

This is a common problem for a lot of llama 3 models because it was in the initial llama 3 release. For anyone else seeing this, it can be fixed with a script included in llama.cpp by running

./gguf-py/scripts/gguf-set-metadata.py llava-llama-3-8b-v1_1.Q8_0.gguf tokenizer.ggml.eos_token_id 128009

Now the repetition problem went away and I'm getting slightly better answers (although I don't see how EOS could have affected that) but still lots of hallucination. Maybe multimodel just isn't quite there yet or I need images at a specific resolution or something.