@jukofyork Yes, check this model page I posted all the PPL results, if you can please provide your PPL results to compare.

@dranger003 Do you have a bash script or something you used to make these with? If so can you share it and I will just leave it to run overnight.

This model seem pretty clueless at coding I think :/ People on reddit seem to love it and say it's good at other stuff though, but not impressed so far...

Not very good at writing either:

USER

Write me the opening chapter of a Grimdark trilogy in the style of Joe Abercrombie and Rob J Hayes. Use third person personal and feature internal monologues of the characters. The POV character for chapter 1 is a cultist who has just escaped his cult. He is dressed in dirty yellow robes and his only possession is a mysterious small mirror. The story starts with him arriving at an apparently deserted ghost town in the edge of the desert.

ASSISTANT

Chapter 1: Ghosts of the Past

The sun was a blistering ball of fire in the cloudless sky, beating down on the dusty, deserted streets of the ghost town. Sand had gathered in the corners of the abandoned buildings, and the air was thick with the scent of decay and neglect.

Brother Kael, a young man with a gaunt face and wild, fearful eyes, stumbled into the town, his dirty yellow robes hanging loosely on his thin frame. He clutched a small mirror in one hand, its surface covered by a layer of grime, its purpose unknown even to him. He had fled his cult, a secret order that worshiped a long-forgotten god of chaos and destruction, and now he was alone, hungry, and terrified.

What have I done? Kael thought, his heart pounding in his chest. I've run from the only life I knew, and now I'm as good as dead. They'll find me, and when they do, the punishment will be worse than death.

Kael's eyes darted nervously around the town, half-expecting to see his former brethren lurking in the shadows, ready to drag him back to face the High Priest's wrath. But the town seemed truly abandoned, a forgotten relic of a bygone era. The buildings, once grand and imposing, were now little more than crumbling ruins, their wooden beams weathered and rotten.

Perhaps I can rest here, Kael thought, his exhaustion overcoming his fear for a moment. I'll find shelter, something to eat, and then I'll keep moving. They'll never expect me to come here, of all places.

As he made his way further into the town, Kael's footsteps echoed eerily in the empty streets. He passed a dilapidated tavern, its sign creaking in the slight breeze, and peered inside. The place was deserted, the tables and chairs overturned and covered in a thick layer of dust.

Looks like this town hasn't seen a living soul in years, Kael thought with a shudder. Might as well be a tomb.

He continued on, his eyes scanning the buildings for any signs of life, but finding none. The silence was oppressive, and Kael found himself holding his breath, as if expecting some terrible monster to leap out at him from the shadows.

Finally, Kael reached the town's center, a small square surrounded by crumbling shops and homes. In the middle stood a weathered statue, its features eroded by time and sand. Kael approached it warily, his eyes narrowing as he tried to make out the figure it depicted.

"Who are you supposed to be?" he whispered, running a hand over the statue's weathered face. "Some long-forgotten hero, I suppose. Well, you won't be saving me from my fate."

A sudden noise startled Kael, and he spun around, his heart racing. But there was nothing, only the silent, watchful buildings and the relentless sun.

I'm losing my mind, he thought, a bitter laugh escaping his lips. How fitting for a traitor and a coward.

Kael lowered himself to the ground, leaning his back against the statue's base. He closed his eyes, the mirror still clutched tightly in his hand. Despite his fear and exhaustion, sleep soon overtook him, and he slipped into a fitful slumber, haunted by dreams of his cult's dark rituals and the blood-soaked altar of their forgotten god.

As the sun began to set, painting the sky in hues of crimson and gold, Kael awoke with a start. For a moment, he forgot where he was, disoriented by the strange surroundings. Then, memory came rushing back, and he scrambled to his feet, his eyes darting around anxiously.

I must keep moving, he thought, his gaze falling on the mirror in his hand. But where? And how far will I have to run before I'm truly safe?

Little did Kael know that his journey had only just begun, and that the mirror he carried held a power that would change his life forever. In the desolate ghost town, under the watchful gaze of forgotten heroes, Kael's path would lead him down a dark and brutal road, where survival meant embracing the very darkness he had fled.

To be continued...

--

Goliath-120B writes way better than this... :/

Sadly I think this is a hard-pass for me, but still happy to run the perplexity tests before deleting it if you want?

I don't have a script, no. I still run everything manually by hand like the old days... maybe its time to use AI.

I don't have a script, no. I still run everything manually by hand like the old days... maybe its time to use AI.

No problem: deepseek-coder-33b is very good at making bash scripts!

I'll have a look later and see if I can come up with something. I'll just start with the lowest quants to see if there is a big difference first.

OK, I'm going to eat my words - this thing I really good at writing if you keep refining it:

Chapter 1: Whispers of Entropy

The sun, a merciless tyrant, hung low in the cloudless sky, casting its scorching rays upon the ghost town that lay forgotten at the edge of the desert. Sand, like the remnants of a long-lost civilization, had invaded the streets and buildings, transforming them into ruins that whispered tales of decay and despair.

Brother Kael, his yellow robes tattered and stained with the dust of his journey, staggered into this desolate place, an arrow protruding from his shoulder. His eyes, wild and feverish, darted anxiously between the abandoned structures, seeking refuge from unseen pursuers. The cult he had fled, the Order of the Shattered Veil, and their dark god, Malachi, would not let him go so easily.

Fools, Kael thought, his breath coming in ragged gasps. I should've known they'd send their hounds after me. But I, Kael, the chosen vessel of Malachi, will not be taken so easily.

In his hand, he clutched a small, oval-shaped mirror, its surface shimmering like oil, reflecting the harsh light of the desert sun. It was a gift from the high priest of the Order, who had recognized the darkness within Kael and sought to harness its power. The mirror, they said, held the key to unlocking the true nature of entropy.

As Kael ventured deeper into the ghost town, his footsteps echoed hollowly against the weathered walls. The buildings, once majestic, now stood in ruins, their wooden beams cracked and their roofs caved in. It was as if the town itself mirrored the chaos and madness that dwelled within him.

His gaze fell upon a crumbling fountain in the town square, its once-clear water now a stagnant pool, filled with the bones of small creatures that had come to drink and found only death. Kael's eyes widened as he approached, his reflection distorted in the murky water.

"The mirror," he whispered, his voice hoarse and strained. "It shows me the truth. The world in its raw, chaotic state."

The mirror had been his constant companion, a reminder of the power he wielded. It had shown him visions of destruction and chaos, and he had embraced them as a prophet of Malachi. But now, it was his only solace, a connection to the god he had once served.

A sudden movement caught Kael's attention, and he spun around, his heart hammering in his chest. But there was nothing, only the silent, watchful town, its secrets buried beneath the sand.

As he continued his journey, his mind wandered back to the night he had fled the Order. It had been a night of blood and madness, a ritual gone awry. Kael closed his eyes, the memory assaulting his senses. He saw the horrified faces of his brethren, their robes soaked in blood, their screams echoing in the sacred chamber. And in the center of it all, he stood, the mirror in his hand, a dark power coursing through his veins.

A shudder ran down Kael's spine as he pushed the memory away. He knew he could never escape his past, but he would not let it define his future. Malachi had chosen him, and he would embrace that destiny, no matter the cost.

His gaze fell upon a dilapidated tavern, its wooden sign creaking mournfully in the desert breeze. With a cautious step, he entered, his eyes adjusting to the dim interior. The room was in disarray, the tables and chairs overturned, their legs broken, as if a violent struggle had taken place long ago.

And then, he saw him. A figure, shrouded in a golden robe, his face hidden behind a mask of pure gold that revealed only a pair of piercing eyes. He sat in the shadows, watching, waiting.

Kael froze, his breath catching in his throat. This mysterious stranger seemed to exude an aura of power and knowledge.

"You seek refuge, brother," the figure said, his voice like the soft hiss of the desert wind. "But you cannot escape your past, nor the god that dwells within you."

Kael's eyes widened, his heart pounding. "Who are you?" he asked, his voice laced with suspicion. "How do you know my thoughts?"

The figure rose, his golden robe shimmering in the dim light. "I am many things, brother. A traveler, a seeker of knowledge, and perhaps, an ally. I, too, have walked the path of chaos and heard the whispers of entropy."

Kael's gaze flicked to the mirror in his hand, its surface shimmering in response to the man's words. "What do you want with me?" he asked, his voice laced with defiance.

The figure took a step forward, his eyes never leaving Kael's. "I seek understanding, brother. The mirror you hold is but a fragment of a greater power. Together, we might unlock its true potential."

Kael's grip tightened on the mirror, his fingers leaving smudges on its shimmering surface. "I will not give it up," he snarled, taking a step back. "It is my connection to Malachi, my birthright."

The figure in gold chuckled, the sound echoing eerily in the empty tavern. "Then perhaps, brother, we shall journey together. For I sense a darkness within you, a power that yearns to be unleashed. And I, for one, am intrigued."

To be continued...

Maybe it's a keeper after all! :D

I've set off the FP16 perplexity run so can double check vs your results to start with: it's around 3.5 hours, but the quants should be much quicker as should be able to fit anything < Q8 in VRAM.

I've set off the FP16 perplexity run so can double check vs your results to start with: it's around 3.5 hours, but the quants should be much quicker as should be able to fit anything < Q8 in VRAM.

Final estimate: PPL = 4.3845 +/- 0.02468

Matches exactly, so will try some more now.

4.4428 + 0.02508 = 4.46788
4.4685 - 0.02526 = 4.44324

It's just outside the 95% in-sample confidence interval, but statistically there is very significant data-leakage between wiki.train.raw and wiki.test.raw:

wiki.train.raw

 = Valkyria Chronicles III = 
 
 Senjō no Valkyria 3 : Unrecorded Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to as Valkyria Chronicles III outside Japan , is a tactical role @-@ playing video game developed by Sega and Media.Vision for the PlayStation Portable . Released in January 2011 in Japan , it is the third game in the Valkyria series . Employing the same fusion of tactical and real @-@ time gameplay as its predecessors , the story runs parallel to the first game and follows the " Nameless " , a penal military unit serving the nation of Gallia during the Second Europan War who perform secret black operations and are pitted against the Imperial unit " Calamaty Raven " . 
 The game began development in 2010 , carrying over a large portion of the work done on Valkyria Chronicles II . While it retained the standard features of the series , it also underwent multiple adjustments , such as making the game more forgiving for series newcomers . Character designer Raita Honjou and composer Hitoshi Sakimoto both returned from previous entries , along with Valkyria Chronicles II director Takeshi Ozawa . A large team of writers handled the script . The game 's opening theme was sung by May 'n . 
 It met with positive sales in Japan , and was praised by both Japanese and western critics . After release , it received downloadable content , along with an expanded edition in November of that year . It was also adapted into manga and an original video animation series . Due to low sales of Valkyria Chronicles II , Valkyria Chronicles III was not localized , but a fan translation compatible with the game 's expanded edition was released in 2014 . Media.Vision would return to the franchise with the development of Valkyria : Azure Revolution for the PlayStation 4 . 
 
 = = Gameplay = = 
 
 As with previous Valkyira Chronicles games , Valkyria Chronicles III is a tactical role @-@ playing game where players take control of a military unit and take part in missions against enemy forces . Stories are told through comic book @-@ like panels with animated character portraits , with characters speaking partially through voiced speech bubbles and partially through unvoiced text . The player progresses through a series of linear missions , gradually unlocked as maps that can be freely scanned through and replayed as they are unlocked . The route to each story location on the map varies depending on an individual player 's approach : when one option is selected , the other is sealed off to the player . Outside missions , the player characters rest in a camp , where units can be customized and character growth occurs . Alongside the main story missions are character @-@ specific sub missions relating to different squad members . After the game 's completion , additional episodes are unlocked , some of them having a higher difficulty than those found in the rest of the game . There are also love simulation elements related to the game 's two main heroines , although they take a very minor role . 
 The game 's battle system , the BliTZ system , is carried over directly from Valkyira Chronicles . During missions , players select each unit using a top @-@ down perspective of the battlefield map : once a character is selected , the player moves the character around the battlefield in third @-@ person . A character can only act once per @-@ turn , but characters can be granted multiple turns at the expense of other characters ' turns . Each character has a field and distance of movement limited by their Action Gauge . Up to nine characters can be assigned to a single mission . During gameplay , characters will call out if something happens to them , such as their health points ( HP ) getting low or being knocked out by enemy attacks . Each character has specific " Potentials " , skills unique to each character . They are divided into " Personal Potential " , which are innate skills that remain unaltered unless otherwise dictated by the story and can either help or impede a character , and " Battle Potentials " , which are grown throughout the game and always grant boons to a character . To learn Battle Potentials , each character has a unique " Masters Table " , a grid @-@ based skill table that can be used to acquire and link different skills . Characters also have Special Abilities that grant them temporary boosts on the battlefield : Kurt can activate " Direct Command " and move around the battlefield without depleting his Action Point gauge , the character Reila can shift into her " Valkyria Form " and become invincible , while Imca can target multiple enemy units with her heavy weapon . 
 Troops are divided into five classes : Scouts , Shocktroopers , Engineers , Lancers and Armored Soldier . Troopers can switch classes by changing their assigned weapon . Changing class does not greatly affect the stats gained while in a previous class . With victory in battle , experience points are awarded to the squad , which are distributed into five different attributes shared by the entire squad , a feature differing from early games ' method of distributing to different unit types . 
 
 = = Plot = = 

wiki.test.raw

 
 = Robert Boulter = 
 
 Robert Boulter is an English film , television and theatre actor . He had a guest @-@ starring role on the television series The Bill in 2000 . This was followed by a starring role in the play Herons written by Simon Stephens , which was performed in 2001 at the Royal Court Theatre . He had a guest role in the television series Judge John Deed in 2002 . In 2004 Boulter landed a role as " Craig " in the episode " Teddy 's Story " of the television series The Long Firm ; he starred alongside actors Mark Strong and Derek Jacobi . He was cast in the 2005 theatre productions of the Philip Ridley play Mercury Fur , which was performed at the Drum Theatre in Plymouth and the Menier Chocolate Factory in London . He was directed by John Tiffany and starred alongside Ben Whishaw , Shane Zaza , Harry Kent , Fraser Ayres , Sophie Stanton and Dominic Hall . 
 In 2006 , Boulter starred alongside Whishaw in the play Citizenship written by Mark Ravenhill . He appeared on a 2006 episode of the television series , Doctors , followed by a role in the 2007 theatre production of How to Curse directed by Josie Rourke . How to Curse was performed at Bush Theatre in the London Borough of Hammersmith and Fulham . Boulter starred in two films in 2008 , Daylight Robbery by filmmaker Paris Leonti , and Donkey Punch directed by Olly Blackburn . In May 2008 , Boulter made a guest appearance on a two @-@ part episode arc of the television series Waking the Dead , followed by an appearance on the television series Survivors in November 2008 . He had a recurring role in ten episodes of the television series Casualty in 2010 , as " Kieron Fletcher " . Boulter starred in the 2011 film Mercenaries directed by Paris Leonti . 
 
 = = Career = = 
 
 
 = = = 2000 – 2005 = = = 
 
 In 2000 Boulter had a guest @-@ starring role on the television series The Bill ; he portrayed " Scott Parry " in the episode , " In Safe Hands " . Boulter starred as " Scott " in the play Herons written by Simon Stephens , which was performed in 2001 at the Royal Court Theatre . A review of Boulter 's performance in The Independent on Sunday described him as " horribly menacing " in the role , and he received critical reviews in The Herald , and Evening Standard . He appeared in the television series Judge John Deed in 2002 as " Addem Armitage " in the episode " Political Expediency " , and had a role as a different character " Toby Steele " on The Bill . 
 He had a recurring role in 2003 on two episodes of The Bill , as character " Connor Price " . In 2004 Boulter landed a role as " Craig " in the episode " Teddy 's Story " of the television series The Long Firm ; he starred alongside actors Mark Strong and Derek Jacobi . Boulter starred as " Darren " , in the 2005 theatre productions of the Philip Ridley play Mercury Fur . It was performed at the Drum Theatre in Plymouth , and the Menier Chocolate Factory in London . He was directed by John Tiffany and starred alongside Ben Whishaw , Shane Zaza , Harry Kent , Fraser Ayres , Sophie Stanton and Dominic Hall . Boulter received a favorable review in The Daily Telegraph : " The acting is shatteringly intense , with wired performances from Ben Whishaw ( now unrecognisable from his performance as Trevor Nunn 's Hamlet ) , Robert Boulter , Shane Zaza and Fraser Ayres . " The Guardian noted , " Ben Whishaw and Robert Boulter offer tenderness amid the savagery . " 
 
 = = = 2006 – present = = = 

The use of the equals sign and the strange spacing between all punctuation, etc.

For a fair test it would need a third unrelated dataset, but I think it's easy to conclude: if it's already so close to the 95% in-sample confidence interval with this much data-leakage, it will be impossible to tell if either data for the imatrix calculation is better (as least for these larger quants).

I also think the strange spacing between all punctuation could be a problem in general with using wiki.train.raw to calculate the imtarix - it will be far from what the tokenizer will have seen in its training corpus and many of the tokens the tokenizer will be generating will be odd or multipart for no reason... One quick test of this would be to do a search and replace to remove these extra spaces and see if the token count for the sample goes way down.

I think this also points to data-leakage:

I know the Q6_K is just within the 95% CI, but the values looked at together tell a different story.

Are those your results or did you copy/paste mine?

Are those your results or did you copy/paste mine?

I copied your wiki.test results, but double checked our FP16 results matched: see the Perplexity (wiki.test) vs Perplexity (groups_merged.txt) headers (sorry should have labelled them better).

The latest iq4_xs quant still inserts phantom spaces during inference, despite the latest updates to llama.cpp. This is a clear indication of a fundamental issue that lies at the core of the problem.
The creator of these quants, dranger003, has nothing to do with it.
All questions should be directed to those who implemented the importance matrix in llama.cpp.

The latest iq4_xs quant still inserts phantom spaces during inference, despite the latest updates to llama.cpp. This is a clear indication of a fundamental issue that lies at the core of the problem.
The creator of these quants, dranger003, has nothing to do with it.
All questions should be directed to those who implemented the importance matrix in llama.cpp.

Oh, that sounds bad. In my situation, not only iq4_xs but also iq3 family did the same thing. It's really annoying in code generation since it breaks indents, and I gave up and am using auto-formatter.
I assumed it's because kinda tokenizer problem that can be fixed by the latest update, which finally turned out to be failed.
I wonder not many people notice this problem.

It might be worth trying this again using the new tokenizer (and recreate the imatrix because of the different tokens getting activated, etc).

I'm just redoing dbrx-instructand already noticing a vastly lower perplexity during the imatrix computation compared to before.

@jukofyork All the weights were already updated, which commit are you referring to?

@jukofyork All the weights were already updated, which commit are you referring to?

Sorry, I didn't see the text:

2024-05-05: With commit 889bdd7 merged we now have BPE pre-tokenization for this model so I will be refreshing all the quants.

Did you notice any improvements after the tokenization fix? I haven't tested it yet but did notice that dbrx-instructonly had a PPL of around 15-20% higher than wizard-lm-2 post-fix, compared to around 80-100% higher before the tokenization fix!

I haven't updated the PPL so I'm not sure, but I use this model all the time and I haven't noticed any issues.