liuhaotian

/

llava-v1.6-mistral-7b

like 225

+1

Would it be possible to add the processor configs to the other llava 1.6 models after this is confirmed to be working?

+1

Would it be possible to add the processor configs to the other llava 1.6 models after this is confirmed to be working?

I have already confirmed this to be working as I came up with this after hours of trying and merging fixes from different sources

I have some error when running it
KeyError: 'model.image_newline'

Fix: Worked with sglang==0.1.9

I got it working with sglang==0.1.11 and 0.1.12 too

Can we get it merged? I would like to use this model in Google Colab.

Since this seems stalled, I made equivalent changes in a fork of this repo: https://huggingface.co/SurfaceData/llava-v1.6-mistral-7b-sglang

@fozziethebeat I tried to convert your model but it also failed;

Traceback (most recent call last):
  File "/content/drive/MyDrive/AI/llama.cpp/convert.py", line 1486, in <module>
    main()
  File "/content/drive/MyDrive/AI/llama.cpp/convert.py", line 1472, in main
    model   = convert_model_names(model, params, args.skip_unknown)
  File "/content/drive/MyDrive/AI/llama.cpp/convert.py", line 1217, in convert_model_names
    raise Exception(f"Unexpected tensor name: {name}. Use --skip-unknown to ignore it (e.g. LLaVA)")
Exception: Unexpected tensor name: model.image_newline. Use --skip-unknown to ignore it (e.g. LLaVA)

What is this conversion script for? I only prepared the model to work for SGLang, so can't make any promises it'll work for other types of conversion.

it's the script coming with llama.cpp to convert a pytorch model to GGUF. I get exactly the same error, so I thought it's the same problem but probably it's not. https://github.com/ggerganov/llama.cpp/blob/master/convert.py

This PR is not meant for llama cpp and it is only a coincidence if it work with llama cpp. This is strictly for SGLang only.

@olafgeibig llama.cpp has a detailed explanation on how to convert llava and llava1.6 models to gguf format.
https://github.com/ggerganov/llama.cpp/tree/master/examples/llava

Also no need to convert it since there are already pre converted llava 1.6 models. Just search llava 1.6 gguf in huggingface