Why merge the same model 5 times?
AlphaMonarch punches far above 7b, ime... but for stories-RP, this model is excessively purple posed, probably too much. but it's still my favorite 7B model. I would really love to see an MOE (2x7 - 4x7), I bet it would be even more of a killer with the right tunes, maybe with less purple tunes to balance it. Or this combined with Experiment26.
Update: There is a moe of exp26 and alphamonarch: https://huggingface.co/jsfs11/MixtureofMerges-MoE-2x7b-v6
This one looks interesting too: https://huggingface.co/jsfs11/MixtureofMerges-MoE-4x7b-v5
If someone bumps this posts, I'll get back to you and tell you how they went.
@UniversalLove333 This is inspired by this config: https://huggingface.co/froggeric/WestLake-10.7B-v2/discussions/1
Yes, I also hate its prose haha. These models look interesting, let me know if this addresses this issue!