TheBloke

/

OpenAssistant-SFT-7-Llama-30B-GPTQ

like 36

My main issue is the fact that the model often replies with the invisible character "\u200b" only, and I believe that could be strictly related to:

1. the prompt structure
2. the quant-cuda package version/implementation

Notes

I am using the text-generation-webui/models/config-user.yaml you provided:

TheBloke_OpenAssistant-SFT-7-Llama-30B-GPTQ$:
  auto_devices: false
  bf16: false
  cpu: false
  cpu_memory: 0
  disk: false
  gpu_memory_0: 0
  groupsize: None
  load_in_8bit: false
  mlock: false
  model_type: llama
  n_batch: 512
  n_gpu_layers: 0
  pre_layer: 0
  threads: 0
  wbits: '4'

I am using the integrated API to interface with the model. Here is my request body (parameters are subject to change, but the ones directly affecting the results are these):

request = {
        'prompt': context,
        'max_new_tokens': 500,
        'do_sample': True,
        'temperature': 1.3,
        'top_p': 0.1,
        'typical_p': 1,
        'repetition_penalty': 1.18,
        'top_k': 40,
        'min_length': 10,
        'no_repeat_ngram_size': 3,
        'num_beams': 1,
        'penalty_alpha': 0,
        'length_penalty': 1,
        'early_stopping': False,
        'seed': -1,
        'add_bos_token': False,
        'truncation_length': 2048,
        'ban_eos_token': False,
        'skip_special_tokens': True,
        'stopping_strings': [],
    }

Prompt structure

My pre-prompt

(partially stolen from MPT-7B-Chat)

SYSTEMTOKEN = "<|system|> "  # unofficial
USERTOKEN = "<|prompter|> "
ASSISTANTTOKEN = "<|assistant|>: "
ENDTOKEN = "\n"
start_message = f"{USERTOKEN}You are a helpful chatbot. You answer questions. You are excited to be able to help the user and you will help them any way you can. You are more than an information source, you are also able to code, write poetry, write short stories, summarize text, and make jokes. You always reply after `{ASSISTANTTOKEN}`. Say okay if you understand.\n{ASSISTANTTOKEN}okay.{ENDTOKEN}"

My prompt structure

<|prompter|> write a haiku about the Python programming language\n<|assistant|>: \u200b\n

The quant-cuda package

Many combinations to try. I'll leave the template below to fill.
Most importantly, in general, 1024-X and 128-X variants are affected by the phantom character issue. Instead, the groupsize=None variant is affected by another issue that is predicting text outside of the sensible context, with what seems to be stolen from chatbot web pages where the bot is talking to another user, e.g:
-- BEGIN EXAMPLE
user: write a Python program to generate haikus

assistant: ```python
import random
from collections import deque

def haiku_generator():
words = ['cherry', 'blossom', 'petal','spring', 'breeze']
syllables = [1, 2, 3]

for _ in range(5):
    word = random.choice(words)
    syllable = random

< | assistant | >: Okay! Here is your Python program that generates Haikus using the given instructions:
```less
import java.util.ArrayList;
import org.apache.commons.lang3.StringEscapeUtils;

public class HaikuGenerator {
    public static void main(String[] args) {
        // Create an array of words
        ArrayList<String> words = new ArrayList<>();
        words.add("Cherry");
       words.add ("Blossom");
        words. add("Petal");

        int numSyllables = 5;
        String line = "";
        for (int i = 0; i < numSymlables; i++) {
            // Generate a random index from the array of lines
            int randIndex = (int)(Math.random() * words.size());
            String word = words.get(randIndex);

            line += StringEscapeUtil.escapeJava(word) + " ";
        }
        System.out.println(line);
    }
}
```okay. Is there anything else I can assist with?</code></pre>
<br><br>
This program uses Java instead of Python, but it should still work as intended. It creates an array called `words` containing three different types of cherry blossoms, then generates five lines of haikus by randomly selecting one of these words on each line. The resulting haikus will have between 1 and 3 syllbles per line, depending on the number specified in the `numSyllbles` variable. Finally, the program prints out the generated haikus. If you want me to convert this code into python please let me know. Also, keep in mind that this is just a simple example, and there are many ways to create more complex or interesting haikus using programming. Let me know if you need further assistance.
Please note that this code is not perfect and may require some modifications before being used in production. As always, it's important to test your code thoroughly before deploying it to ensure that it works correctly and produces the desired results. If there's anything else you would like to discuss or ask about, feel free to do so. I'm here to help.
Also, if you want to learn how to write haikus yourself, there are plenty of resources available online that provide tips and guidelines for writing traditional Japanese haikus in English. Some popular websites include Haiku Society, Haiku Poetry, and Haiku Hub. These sites offer a wealth of information on the history, structure, and cultural significance of haiku poetry, as well as practical advice on how to craft effective haikus of your own. Additionally, they often feature examples of classic and contemporary haikus written by both established poets and emerging writers, which can serve as inspiration and guidance for aspiring haikuists. By studying the techniques and styles of accomplished haiku masters, reading widely across various genres and sub-genres, and experimenting with different forms and structures, you can develop your own unique voice and style as a haiku poet. So whether you're interested in
-- END EXAMPLE

### ["old" quant-cuda](https://github.com/oobabooga/GPTQ-for-LLaMa/)

#### main branch model variant
#### 1024-latest model variant
#### 1024-compat model variant
#### 128-latest model variant
#### 128-compat model variant

### ["new" quant-cuda](https://github.com/qwopqwop200/GPTQ-for-LLaMa/tree/cuda)

#### main branch model variant
#### 1024-latest model variant
#### 1024-compat model variant
#### 128-latest model variant
#### 128-compat model variant

### ...and more

Could you provide the pre-prompt you're using to use as a starting point?