license: apache-2.0
tags:
- freeze
- generated_from_trainer
- autoquant
- exl2
base_model: mistralai/Mistral-7B-Instruct-v0.2
model-index:
- name: rubra-11b-h
results: []
rubra-11b-h
Model description
This is a mistral based model trained on a lot of code and conversational data. The goal of this model was to retain the abilities of mistralai/Mistral-7B-Instruct-v0.2 while making it better at reasoning.
- 32k context window
- Rope-theta = 1e6
Fine tuning instruct models has proven to be a challenge when the goal is to create a model that is not meant for a particular task. Specifically, fine tuning an instruct model often leads to destructive forgetting in the resulting model. This is why most fine tunes select a base (non-instruct) language model. The rubra-v0.1 series models aim to add desired traits like multi-turn chat capabilites while retaining knowledge found in instruct models
Chat format
Your prompt should be surrounded by [INST] and [/INST] tokens. The very first instruction should begin with a begin of sentence id. The next instructions should not. The assistant generation will be ended by the end-of-sentence token id.
E.g.
text = "<s>[INST] What is your favourite condiment? [/INST]"
"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s> "
"[INST] Do you have mayonnaise recipes? [/INST]"
This format is available as a chat template via the apply_chat_template() method:
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto
model = AutoModelForCausalLM.from_pretrained("sanjay920/rubra-11b-h")
tokenizer = AutoTokenizer.from_pretrained("sanjay920/rubra-11b-h")
messages = [
{"role": "user", "content": "What is your favourite condiment?"},
{"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
{"role": "user", "content": "Do you have mayonnaise recipes?"}
]
encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")
model_inputs = encodeds.to(device)
model.to(device)
generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])
GGUF
GGUF quants are available at sanjay920/rubra-11b-h-GGUF
Create a new discussion for other quant variant requests!
LM Studio
Set "Chat Preset" to Mistral Instruct
Training procedure
- Block expansion and further training
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 12
- total_train_batch_size: 12
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 3.0
- mixed_precision_training: Native AMP
Training results
{
"epoch": 3.0,
"train_loss": 0.5844329195557438,
"train_runtime": 307371.8718,
"train_samples_per_second": 0.246,
"train_steps_per_second": 0.021
}
Framework versions
- Transformers 4.38.2
- Pytorch 2.2.1+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2
Evaluation Results
########## Average ##########
score
model
gpt-4-0613 9.020312
gpt-3.5-turbo-0125 7.970219
rubra-11b-h 7.800000
Mistral-7B-Instruct-v0.2 7.793750
Mistral-7B-Instruct-v0.1 6.721875
Llama-2-13b-chat 6.650000
Intended uses & limitations
This model is still under development. It hasn't been aligned for safety, so it may produce biased generations. It appears to produce thoughtful answers compared to other small LLMs. See below:
Examples
Needle in a haystack test:
The input text can be found in needle-in-a-haystack.txt
- 14,334 input tokens
- "Random number is 23." inserted into the middle of the long document text. Asked the LLM what the random number is.
rubra-11b-h:
The random number is 23.
- Correct
mistral-7b-instruct-v0.2:
The random number generated from this text is 23. However, it's important to note that the method used to generate this number from the text is not a secure or reliable one and should not be used for any purpose where security is a concern. This was just a fun exercise to extract a number from the text.
As for the content of the text, it discusses the idea of creating a technology hub or "silicon valley" by attracting the right people - specifically, rich people and nerds. The author argues that these two groups are the limiting reagents in the reaction that produces startups, as they are the only ones present when startups get started. The text also touches on the idea that universities can act as magnets for these people and that personality and youth are important factors in creating a successful technology hub.
- Correct
hermes-2-pro:
the<dummy00028>'1 Question1 question2<dummy00028> <dummy00015>..... in3<dummy00028>d<dummy00028>}<dummy00015>l kennis4<dummy00012>il<dummy00028>y<dummy00028>}2<dummy00028>-^{ a<dummy00028>}3<dummy00012>. a<dummy00028>}<dummy00015> toion<dummy00028>} people a a<dummy00028>}5<dummy00005>- a Question1<dummy00028>}y<dummy00028>}6<dummy00012>. a<dummy00028>}1<dummy00028>}<dummy00015>l<dummy00012>,<dummy00015>l<dummy00028>},7<dummy00012>. <dummy00015>l Fuß in the<dummy00028>
a a<dummy00028>'2 a<dummy00028>}4 ál a Question3<dummy00012>o kennis in if to more a<dummy00028> or a<dummy00028>'5<dummy00005> 1<dummy00028>}1<dummy00005>s<dummy00028>y<dummy00028>y<dummy00028>'6 Question9 Question1<dummy00028>'2 Questions<dummy00028>y biologie1<dummy00028>y<dummy00028>y<dummy00028>-3<dummy00028>'1 Question. a<dummy00028>* the<dummy00028>}<dummy00015>y Question a<dummy00028>4<dummy00012>,ll Question1 Question6<dummy00028>4<dummy00028>}2<dummy00028>}1<dummy00028>}<dummy00015>l<dummy00028>
if a Questionia kennisa<dummy00005> <dummy00015> a<dummy00028>h<dummy00028>}2<dummy00028>a égal
- Incorrect. Unusable output