is this a single expert of all the experts merged?
I planned on doing this and releasing a mistral 22b model myself, but i was going to merge all the experts with "model stock" merge method in mergekit. Did you merge the experts or is this a single lone expert?
@Vezora Welp i tried to extract the models, but the code i used to extract them seemed to break the models. They only generated empty tokens. Like invisible text basically. If you have code to extract them you are willing to share that would be nice. Id be willing to try again and try a few merge methods to see if i can get it right without needing to finetune
“I will be releasing a blog post soon on how I did this, I still will release a paper with testing and results, but I'm gonna rush out a paper before hand so I can share how I did this. “ 😁