tokenizer_config.json different
Hello byroneverson! 🙂
Thank you for this abliterated model, I was hoping that you would maybe abliterate it so it was a nice surprise when you did!
Question: I've noticed that the tokenizer_config.json is different than the one provided in the original repo.
The reason it caught my attention is due to a warning llama.cpp provided:
llm_load_vocab: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
So I was wondering, is the reason that it is different because you ran into this issue linked here?
Because there's a solution provided there that might help avoid the current warning in llama.cpp and allow you to use the original tokenizer_config.json.
Unless of course, the tokenizer_config.json is different due to the abliteration process. In which case feel free to ignore my ramblings here!
I also wanted thank you for all the other abliterated models, I've downloaded pretty much all of the ones that you did.
Hi thanks for checking out this abliteration (and the others).
You are correct, that error was preventing me from using my personal gguf-my-repo space to create a demo gguf. I came across the post you linked above but was not a fan of the solution due to the fact that it would require anyone that wants to quant my model (such as yourself) to hunt down the solution. That solution would also have required me to make some changes to my gguf-my-repo space which is not a problem but also not ideal (my ML time is limited).
The hope was to find a solution that would allow things to convert smoothly as is. After noticing that the stock glm4-9b had a different added_tokens_decoder and additional_special_tokens in its tokenizer config I kind of just went along with it to see if it would allow the conversion. I wanted it to be an small oversight by THUDM but that is a reach lol. My hopes were that the parameters were mostly a redundancy that wouldn't make much of a difference to actual model output. Considering the error was about the <|user|> token I probably could have maybe just added that to the config but instead decided to play it safe and add the whole lot to longwriter tokenizer config, I was in rush due to a my new dev environment and an already delayed abliterated request (always seeking timely abliterations where possible).
I am assuming the warning happens during the conversion process because I just tested the gguf and did not receive a warning when loading the model. I could do some more playing around with the tokenizer config to see if only adding <|user|> to added_tokens_decoder or additional_special_tokens would result in a conversion that does not have the original error or this eot/eos/eog warning, but I guess I am wondering if this warning makes any real world difference in the usage of the model once converted. If it turns out to be a warning of redundancy during conversion due to the eot token already being handled (tokenizer.model and tokenizer.config token ids should match for <|endoftext|> token) then maybe it doesn't matter, on the other hand if llama.cpp is no longer detecting the main eot token in the array of eog tokens then obviously that is a more serious matter. As far as I can tell, my demo gguf is stopping as expected. Please let me know if you are encountering missed stops, something that generally seems off, or know more than me about the tokenization conversion between hf and gguf and have missed something critical.
In any case, I really do appreciate you checking out my abliterations. Currently have solar-preview-pro-instruct on my workbench atm, its being a little fickle but I think I will get it without too much collateral damage. Maybe then TheDrummer/Star-Command-R-32B-v1 will seem less problematic and will give me confidence to tackle nemotron. hehe
Thanks,
Byron