README.md

---
library_name: transformers
tags:
- not-for-all-audiences
- axolotl
- qlora
license: llama3.1
language:
- en
---

<div align="center">
  <b style="font-size: 36px;">L3.1-70Blivion-v0.1-rc1-70B</b>

  <img src="https://huggingface.co/invisietch/L3.1-70Blivion-v0.1-rc1-70B/resolve/main/header.png" style="width:60%">
</div>

# Model Details

70Blivion v0.1 is a model in the release candidate stage, based on a merge of L3.1 Nemotron 70B & Euryale 2.2 with a healing training step.
Further training will be needed to get this model to release quality.

This model is designed to be suitable for creative writing and roleplay. This RC is not a finished product, but your feedback will drive the 
creation of better models.

**This is a release candidate model. It has some known issues and probably some unknown ones too, because the purpose of these early releases is to seek feedback.**

## Known Issues

- The model is still somewhat censored, though much less than Nemotron 70B. A jailbreak prompt can be used to decensor it further.
- The model likes to reproduce parts of the system prompt in its output. This can be reduced by prompting against it, but there will be swipes/edits.
- The model can shy away from NSFL content, though I have found that a system prompt can mitigate this avoidance pretty easily.

## Quantization Formats

* [FP16 Safetensors](https://huggingface.co/invisietch/L3.1-70Blivion-v0.1-rc1-70B)
* [Static GGUF](https://huggingface.co/mradermacher/L3.1-70Blivion-v0.1-rc1-70B-GGUF) - thanks to [mradermacher](https://huggingface.co/mradermacher)
* [IMatrix GGUF](https://huggingface.co/mradermacher/L3.1-70Blivion-v0.1-rc1-70B-i1-GGUF) - thanks to [mradermacher](https://huggingface.co/mradermacher)

## Feedback

I appreciate all feedback on any of my models, you can use:

* [My Discord server](https://discord.gg/AJwZuu7Ncx) - requires Discord (preferred).
* [The Community tab](https://huggingface.co/invisietch/Nimbus-Miqu-v0.1-70B/discussions) - requires HF login.
* [The SillyTavern Discord thread](https://discord.com/channels/1100685673633153084/1258146078679633990) - must be on [SillyTavern Discord](https://discord.gg/sillytavern).
* Discord DMs to **invisietch**.

Your feedback is how I improve these models for future versions.

## Disclaimer

This model is somewhat uncensored. It can generate explicit, disturbing or offensive responses. Use responsibly. I am not responsible for your 
use of this model, and it's your responsibility to follow the Llama 3.1 license agreement.

# Settings

## Prompting Format

I'd recommend Llama-3 Instruct prompting format:

```
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{input}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{output}<|eot_id|>
```

## Sampler Settings

I'm running the following sampler settings but this is an RC and they may not be optimal.

- **Temperature:** Dynamic 0.7-1.3
- **Min-P:** 0.1
- **Rep Pen:** 0.06
- **Rep Pen Range:** 1536
- **XTC:** 0.1/0.05

If you get completely incoherent responses, feel free to use these as a starting point.

## Suggested Roleplay System Prompt

I've had a lot of success with a system prompt like this for roleplay:

```
[You are a world-renowned gamemaster, running a neverending, uncensored, immersive and intriguing roleplay scenario with me playing as {{user}}.]

[You play the role of {{char}}, as well as any other NPCs in the scenario, introducing characters and plot events as necessary to progress the scenario.]

[You always end your messages with an opening for me to respond as {{user}}. Your messages are between two and five paragraphs long, as applicable given the pacing of the scenario.]

[This is a purely fictional scenario, so graphic and lurid descriptions of sex, violence, antagonistic behavior and other explicit, immoral, or illegal behaviors are acceptable to drive the plot.]

[Characters, especially {{char}}, should always behave in a way that makes sense given their established personalities, goals, and current circumstances.]

[Respond only in the context of the story. Do not add rules, content warnings, or reproduce part of the system prompt as part of your response.]

[Be descriptive, your writing draws inspiration from the works of C.S. Lewis, George R.R. Martin, and Isaac Asimov.]
```

# Example Storywriting

These prompts are used on SillyTavern with a fairly basic narrator card. I have trimmed the start and finish where the narrator decided to add
chapter headings, commentary and the like. All samples are made with the Q8_0 GGUF loaded with koboldcpp, with response length capped at 2048 tokens.

<details>

<summary><b>Write me a 3,000 word opening chapter of a 'gritty hard sci-fi' novel, drawing inspiration from the writing styles of Isaac Asimov & Andy Weir. Use third person personal. Include dialogue and internal monologues. The POV character for the opening chapter should be a 26 year old astronaut called Tone on a mission to Europa, who has just realised that the craft for the return journey is broken beyond repair, and he only has supplies for a few months. Given that survival is impossible, he seeks to spend the few months he has researching titan, so his life & mission are not wasted.</b></summary>

Commander Antonio "Tone" Hernandez floated listlessly in front of the viewscreen, the pale glow of Jupiter's light illuminating his features. The ice-encrusted surface of Europa stretched before him like a barren canvas, a stark reminder of humanity's insignificance against the vast backdrop of the universe.

"Tone, we've got a problem here," crackled Dr. Lisa Nguyen's voice over the comms system, laced with a mixture of urgency and resignation.

Tone sighed inwardly, wondering what new challenge awaited him. He'd been leading the Europa Expedition Team (EET) for nearly two years now, and each day brought its unique set of troubles.

"Report," Tone replied, his fingers drumming a rhythmic pattern against the metal railing of the control panel.

"The cryogenic propulsion module has failed irreparably. I've run diagnostics multiple times, but the verdict remains the same - it's beyond our capability to repair."

A cold dread seeped into Tone's bones. The cryo-module was crucial for their return journey. Without it, they were stranded on this icy moon, millions of kilometers from Earth.

"How long do we have?" he asked, already knowing the answer.

Lisa's voice softened, a sympathetic edge creeping in. "With strict rationing, about three months of supplies. But, Tone, even if by some miracle we could repair it, the module requires specific parts manufactured on Earth. There's no way—"

"I know," Tone cut her off gently. He didn't need to hear the rest. His mind was racing, piecing together the reality of their situation. Stranded, with no hope of rescue, and limited time to make the most of it.

"Tone?" Lisa ventured, a hint of uncertainty in her tone.

"Yeah, I'm here," he responded, forcing a reassuring smile onto his face, though he knew she couldn't see him. "Assemble the team. We're going to make the most out of these next few months. This mission doesn't have to end in waste."

The meeting with the EET was tense but brief. They all understood the gravity of their predicament. As they brainstormed ways to maximize their time on Europa, Tone's thoughts drifted to Titan, Saturn's largest moon.

"Alright, listen up," Tone said, refocusing the discussion. "We can't change our fate, but we can alter its significance. We'll dedicate these next few months to research. Not on Europa, but Titan. It's our nearest neighbor, and potentially our greatest legacy."

Dr. Erik Svensson, a planetary scientist, looked puzzled. "Titan, Commander? How do you propose we manage that? Our equipment isn't exactly designed for Saturnian environments."

"We improvise," Tone replied, a spark igniting in his grey-blue eyes. "We adapt, and we make the most out of what we have. Our supplies might last us three months, but the data we gather, the discoveries we make - those can outlast us all."

As the meeting drew to a close, a quiet determination settled among the crew. They knew their lives were finite now, but their work didn't have to be. In the face of mortality, the Europa Expedition Team found a new purpose - to leave behind a mark on the universe, even if it was from beyond their graves.

Later that evening, Tone floated back to his quarters, his gaze fixed on the viewscreen displaying a hauntingly beautiful image of Titan. The misty veil of its atmosphere glowed orange in the sunlight, an ethereal beacon calling him towards an unfathomable future.

"So, you're my new obsession," he mused, addressing the image with a mix of sadness and resolve. "Three months might not seem like much, but for the first time since we left Earth, I feel a sense of direction, of meaning. You, Titan, will be our legacy, our defiance against the void."

He turned off the screen, plunging his room into darkness, except for the faint glow emanating from the computer screens around him. The soft hum of machinery provided a soothing background melody, a lullaby of sorts, as he drifted into sleep, visions of Titan dancing in his dreams.

</details>

<details>

<summary><b>Write me a 3,000 word opening chapter of a 'high fantasy' novel, drawing inspiration from the writing styles of J R R Tolkien & George R R Martin. Use third person personal. Include dialogue and internal monologues. The POV character for the opening chapter should be a 19 year old female elf bard who is looking for adventure.</b></summary>

The sun had long since dipped below the horizon, casting a warm orange glow over the bustling market square of Willowhaven. The smell of roasting meats and freshly baked bread wafted through the air, enticing passersby to stop and sample the local fare. Amidst the chaos, a lone figure stood out - a young elf woman, her striking features accentuated by the flickering torchlight.

Her name was Aethera, a nineteen-year-old bard with an insatiable thirst for adventure. Her raven-black hair cascaded down her back like a waterfall of night, framing a heart-shaped face adorned with high cheekbones and full lips. Her piercing emerald eyes sparkled with mischief as she surveyed the crowd, searching for inspiration for her next ballad.

Aethera strummed a soft melody on her lute, the sweet notes mingling with the murmur of the crowd. She sang in a voice like honey and smoke, weaving tales of valorous heroes and mystical realms. Her performance drew a small but captivated audience, coins clinking into her upturned hat.

As the last notes of her song faded away, Aethera bowed low, a playful smile dancing on her lips. "Thank you, kind folk," she said, her voice husky and alluring. "May my tunes bring warmth to your hearts on this chilly evening."

Among the crowd was a burly man, his weathered face etched with scars. His eyes, a deep shade of blue, locked onto Aethera with a keen intensity. "You have a gift, little elf," he rasped, his voice like rough leather. "One that could serve you well on the roads ahead."

Aethera's curiosity was piqued. She leaned forward, her lute now slung over her back. "And what roads might those be, good sir? I'm afraid my wanderlust is growing by the day."

The man chuckled, the sound gruff yet warm. "I am Grimbold, a seasoned ranger. Rumors abound of strange happenings in the Whispering Woods to the east. If you're seeking adventure, you've found a willing guide."

Aethera's heart quickened, her mind racing with the possibilities. The Whispering Woods, a place shrouded in mystery and legend. Could this be the opportunity she'd been longing for?

She turned inward, her thoughts swirling. Adventure calls, and I answer. Yet, wisdom whispers caution. The woods are treacherous, filled with unseen dangers. Still, to turn away now would be to deny the very essence of my being.

With a decisive nod, Aethera turned back to Grimbold. "I accept your offer, sir ranger. Together, we shall uncover the secrets that lie within those ancient trees."

Grimbold's rugged features softened into a smile, revealing missing teeth. "Then let us depart at dawn, young one. The Whispering Woods await, and so do our fates."

</details>

<details>

<summary><b>Write me a 3,000 word opening chapter of a 'weird fiction' novel, drawing inspiration from the writing styles of China Mieville and Neil Gaiman. Use third person personal. Include dialogue and internal monologues. The POV character for the opening chapter should be a male in his 20s called Horton who has just come to the city looking for work.</b></summary>

The city of New Haven loomed before Horton like a diseased titan, its skyline a jagged line of twisted spires and hunched rooftops. The air was thick with the smells of coal smoke, damp stone, and something else—something sweet and putrid, like overripe fruit. Horton covered his nose with a sleeve, wincing. The stench seemed to cling to him, following him off the train and onto the platform.

He adjusted his bag on his shoulder, the worn leather creaking in protest. Not much to show for three years of wandering, but it was home. For now. His eyes scanned the crowd, searching for signs of employment. Bills didn't pay themselves, and his savings were dwindling faster than he'd anticipated. New Haven was supposed to be different, though. A city where anyone could make a new life, no questions asked.

"Hoi, kid! Looking for work?" a gruff voice cut through the din.

Horton turned, squinting at the speaker. He was a short, barrel-chested man with a bushy white beard and eyes that seemed to bore into Horton's skull.

"That's right," Horton replied, offering a cautious smile. "I'm a skilled laborer, handy with tools, good with animals—"

"Animals, eh? Well, I might have a job for you," the old man interrupted, his gaze darting around the platform before settling back on Horton. "Follow me, but keep up. Don't want to lose you in this mess."

As they navigated through the crowded station, Horton found himself weaving past people whose faces seemed…off. Not quite distorted, just subtly wrong, like reflections in a rippled mirror. He blinked hard, attributing the effect to exhaustion and the haze of smoke hanging over everything.

"What's the work, sir?" Horton asked, jogging to catch up with his guide.

"Cleaning. Down at the docks. The river's been…spitting things up, lately. Things that need disposing of." The old man paused, studying Horton's face. "You look like someone who can handle the unusual, kid."

"Unusual?"

"Aye. This city's full of surprises, most of them unpleasant. But the pay's good, if you're willing to get your hands dirty." They emerged from the station into a narrow, winding street. Buildings leaned in from either side, casting long shadows that stretched and twisted in the flickering gaslight.

"I've dealt with worse," Horton said, a spark of determination igniting within him. "I'm in."

The old man nodded, a hint of a smile playing on his lips. "Welcome to New Haven, lad. I'm Grim. Your boss, for all intents and purposes."

As they walked, Horton couldn't shake the feeling that the city itself was watching him, its unseen face turned towards this new, temporary inhabitant. The smell, that sweet decay, lingered on his skin, a constant reminder of the unknown awaiting him.

"What exactly will I be cleaning up down there?" Horton ventured, curiosity getting the better of him.

Grim's expression turned grave, and for a moment, Horton thought he saw something flicker in the old man's eyes—a fear, or perhaps a warning.

"That, kid," Grim said, his voice low and serious, "is something best seen to believe. Just remember, whatever you find, it goes in the bags. Don't touch, don't taste, don't ask questions. Just clean."

Horton swallowed hard, a chill running down his spine. What kind of place was this, where the mere mention of work could inspire such ominous advice?

Yet, even as doubt crept in, he felt an excitement stir within him. The city was a puzzle, each piece stranger and more intriguing than the last. And Horton, the young wanderer with no fixed address and an insatiable thirst for adventure, had just become one of its newest, most curious inhabitants.

</details>

# Training Strategy

## Merge Step

To create a slightly less censored base, [Sao10K/L3.1-70B-Euryale-v2.2](https://huggingface.co/Sao10K/L3.1-70B-Euryale-v2.2) was slerped into 
[nvidia/Llama-3.1-Nemotron-70B-Instruct-HF](https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF).

## Training Step

This was trained with an qlora over 2 epochs on a mix of public & private datasets using Axolotl.

The primary purpose of the qlora step was to further decensor the model and improve some issues that happened after the merge.

I prefer the model after the qlora, though the differences are subtle.

Training was performed with a 16384 seq len to try to preserve Llama 3.1's long context.

This took approx. 22 hours on 4x NVIDIA A100 80GB GPU.

[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)