Hi! Here is a little review.
This review may look harsh, but I don't mean it in any case. Like a fellow creator, I think that a direct approach is more effective. But you are also free to think that I'm just a random dork from the internet and ignore it, m'kay?
I use AI to explore ideas and stories, create character cards with scenarios, etc. I mostly use the Midnight-Miku 70b 1.5 and Koboldcpp as the backend, but I also tried around 100+ different LLMs and finetunes. Undi and Neversleep stuff too. And I must say honestly that it is a true talent to be able to constantly make even 70b feel like 7b in the cognitive dep xD
There is something with their datasets or the way they finetune the models. They all feel very crude with a focus on the negative part of the bio. And I see the same tropes here. I cannot run such models in f16 or even Q4, but at the same time, Midnight-Miku runs on iQ2_k nearly perfectly. And I also tried the iQ3_k_m version from a different quant just to be sure.
• I tested this on my cards/scenarios and it constantly ignores the style that I set from the first message and examples.
• After a short time, it may forget to act as the narrator and turn into the chat.
• I remember there was a problem with Mistral 7b and 8x7b with repetitiveness - we have one here too. And it may be wild sometimes: for example, deciding that the character must stay in shock for 6 messages, repeating the same thing over and over again. Yes, I tried to regenerate and change the settings.
• "OOC" stuff works from time to time. If to compare with Miku, there I don't need to do that at all. I just can directly 'speak' with AI and it will understand that I'm out of character right now. And DPO may fix that, but it will kill most of the remaining creativity (at least this is my experience with DPO so far).
• I noticed a similar behavior with Airoboros finetune from Jondurbin. Were by adding too many toxic instructions it turns interaction with the model into a constant 'regenerate' button masher. Because even his recent Llama3-70b Airoboros finetune feels just like old Yi 34b.
• The text quality is like in Neversleep's finetunes. If to compare, my favorite so far is Midnight-Miku 70b 1.5 and Bagel 34b 0.2. Again, maybe this is just my taste, but they like to focus on emotions and small details that keep the narrative interesting.
This is my experience so far. Again, I'm not trying to be rude, just pointing out the 'problems' that I met.
I hope this will be somehow useful and good luck in this humble journey!