Using VN style dialogue is a pretty great idea, however...
Are there any plans to make a llama 3 trained on all characters? It'd be nice to have a model trained for group chats that can be used for Silly Taverns VN mode.
Did a real quick test for something like that with this (fine tuned on all of the game's conversation parsed and cleaned, with every script label roughly as 1-2 data entries). The first results weren't promising (the characters didn't act like themselves or know their own info) but tbf the dataset could use a little more work + personal info usually added per character dataset could help. If there's interest, can try looking into this again when I have time.
Thanks! Question though, just so I know what I'm getting into. When you tested it, did you use Example dialogue and lorebooks? And for the context string, did you build it normally or did you build it with lorebooks with custom depth(example dialogue included) so that it stays close to the end of the context and not all the way back at the beginning of the context? Cus if not, then there's still a good chance I could get a good experience out of this model.
Btw, before I forget, has the model been trained with dialogues from other VNs before having DDLC dialogue trained over top of it?
Wanted to make sure, cus I'm pretty sure Llama 3 isn't trained on a lot of VN data (unless zuck is a massive weeb and we never knew) so I'm not sure it will give very great inputs due to lack of data.
It was tested with only some example dialogue. In hindsight, could be why it didn’t perform well, but usually have tested with only this like with other models: would try to "bake in" the lore or at least character info to minimize use for lorebook (started as a personal habit to "save" tokens for limited computation, limited context window- from testing at least the Monika models usually held well without lorebooks and even example dialogue).
The dataset was just the conversation text and speakers without context string (it was rushed that way since underestimated the time I had for it). And it was fine-tuned on base Llama-3. By experience, the Monika models that were fine-tuned on non-base Llama-3 seemed to be more erratic like they would lose more of their common sense/knowledge or even break more for some reason, so stuck to base model for fine-tunes. That said, could look into it again- to be fair those were the early days of Llama-3 so many things could have been fixed and there’s definitely lots of room for improvement.