mistralai

/

Mixtral-8x22B-v0.1

like 191

There was an issue with the last PR - we used the V3 tokenizer, but this base model actually uses the V1 tokenizer. This should fix the issue!

@Rocketknight1 does it affect the vocab size? Model and tokenizer sizes are not matching. So model is failing to load.

@lbathen can you give me some code to reproduce that issue? From here it looks like the tokenizer and the model both have a vocab size of 32000

@Rocketknight1 I confirmed that both show same vocab of 32K now. I had pulled the wrong revision :)

Is this going to be merged soon?

@Rocketknight1 Could you merge this in? It's working on my end and I'm thankful to have this model back.

This command should get it running for anyone who needs it:

python -m vllm.entrypoints.openai.api_server --model mistralai/Mixtral-8x22B-v0.1 --revision c356b81 --served-model-name mistralai/Mixtral-8x22B-v0.1 --max-logprobs 100 --gpu-memory-utilization=0.85 --disable-log-requests --disable-log-stats --port 5001 --tensor-parallel-size 8

My apologies! Merging this PR!