|
--- |
|
license: llama3 |
|
language: |
|
- en |
|
- de |
|
- fr |
|
- it |
|
- pt |
|
- hi |
|
- es |
|
- th |
|
tags: |
|
- llama |
|
base_model: mattshumer/ref_70_e3 |
|
pipeline_tag: text-generation |
|
library_name: ggml |
|
datasets: |
|
- froggeric/imatrix |
|
metrics: |
|
- perplexity |
|
--- |
|
|
|
# Reflection-Llama-3.1-70B-GGUF |
|
|
|
 |
|
|
|
GGUF quantized models of [mattshumer/ref_70_e3](https://huggingface.co/mattshumer/ref_70_e3) |
|
|
|
> This is the new, working version of the Reflection Llama 3.1 70B model. |
|
|
|
**Reflection Llama-3.1 70B is (purportedly) the world's top open-source LLM, trained with a new technique called Reflection-Tuning that teaches a LLM to detect mistakes in its reasoning and correct course.** |
|
|
|
| Quantization | Size | Split | iMatrix | |
|
| ------------ | ------ | ----- | ------- | |
|
| FP16 | 141GB | true | false | |
|
| Q8_0_L | ??.?GB | true | false | |
|
| Q8_0 | ??.?GB | true | false | |
|
| Q6_K_L | ??.?GB | true | false | |
|
| Q6_K | 57.9GB | true | false | |
|
| Q5_K_L | 52.6GB | true | false | |
|
| Q5_K_M | ??.?GB | true | false | |
|
| Q5_K_S | 48.7GB | false | false | |
|
| Q4_K_L | 45.3GB | false | false | |
|
| Q4_K_M | ??.?GB | false | false | |
|
| Q4_K_S | 40.3GB | false | false | |
|
| IQ4_NL | 38.2GB | false | true | |
|
| IQ4_XS | ??.?GB | false | true | |
|
| Q3_K_XL | 37.2GB | false | false | |
|
| Q3_K_L | 37.1GB | false | false | |
|
| Q3_K_M | 34.3GB | false | false | |
|
| IQ3_M | ??.?GB | false | true | |
|
| Q3_K_S | ??.?GB | false | false | |
|
| IQ3_S | ??.?GB | false | true | |
|
| Q2_K_L | 29.4GB | false | false | |
|
| IQ3_XS | ??.?GB | false | true | |
|
| IQ3_XXS | ??.?GB | false | true | |
|
| Q2_K | ??.?GB | false | true | |
|
| Q2_K_S | ??.?GB | false | true | |
|
| IQ2_M | 23.0GB | false | true | |
|
| IQ2_S | 21.2GB | false | true | |
|
| IQ2_XS | 20.2GB | false | true | |
|
| IQ2_XXS | 18.2GB | false | true | |
|
| IQ1_M | 16.0GB | false | true | |
|
| IQ1_S | 14.6GB | false | true | |
|
|
|
The `_L` or `_XL` suffix means that the token embeddings and output weight are at fp16 precision. |
|
|
|
The iMatrix dataset is bartowski's, which you can find here: [calibration_datav3.txt](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8) |
|
|
|
Computation is done on static Q6_K for 125 chunks. |
|
|
|
## Model Info |
|
|
|
The model was not trained on 3 epoches, because it's identical to the 2nd epoch run [mattshumer/Reflection-Llama-3.1-70B-ep2-working](https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B-ep2-working) (it's possible this is also fake). |
|
|
|
The fine-tuning was done using LoRA with rank 256 on the Llama-3.1-70B-Instruct model. |
|
|
|
## Benchmarks |
|
<div style="position: relative;"> |
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/60518f3731c5be7f3dd5ebc3/zNs-ZFs0SbnomH7mikiOU.png" alt="Your image description"> |
|
<div style="position: absolute; top: 50%; left: -20%; width: 140%; height: 5px; background-color: red; transform: rotate(10deg);"></div> |
|
<div style="position: absolute; top: 50%; left: -20%; width: 140%; height: 5px; background-color: red; transform: rotate(-10deg);"></div> |
|
</div> |
|
|
|
**Warning: These are likely false scores and cannot be replicated with this model.** |
|
|
|
All benchmarks tested have been checked for contamination by running [LMSys's LLM Decontaminator](https://github.com/lm-sys/llm-decontaminator). When benchmarking, we isolate the `<output>` and benchmark on solely that section. |
|
|
|
Trained from Llama 3.1 70B Instruct, you can sample from Reflection Llama-3.1 70B using the same code, pipelines, etc. as any other Llama model. It even uses the stock Llama 3.1 chat template format (though, we've trained in a few new special tokens to aid in reasoning and reflection). |
|
|
|
During sampling, the model will start by generating reasoning inside `<thinking>` and `</thinking>` tags, and then once it is satisfied with its reasoning, it will output the final answer inside `<output>` and `</output>` tags. Each of these tags are special tokens, trained into the model. |
|
|
|
This enables the model to separate its internal thoughts and reasoning from its final answer, improving the experience for the user. |
|
|
|
Inside the `<thinking>` section, the model may output one or more `<reflection>` tags, which signals the model has caught an error in its reasoning and will attempt to correct it before providing a final answer. |
|
|
|
## System Prompt |
|
|
|
The system prompt used for training this model is: |
|
|
|
``` |
|
You are a world-class AI system, capable of complex reasoning and reflection. Reason through the query inside <thinking> tags, and then provide your final response inside <output> tags. If you detect that you made a mistake in your reasoning at any point, correct yourself inside <reflection> tags. |
|
``` |
|
|
|
We recommend using this exact system prompt to get the best results from Reflection Llama-3.1 70B. You may also want to experiment combining this system prompt with your own custom instructions to customize the behavior of the model. |
|
|
|
## Chat Format |
|
|
|
The model uses the standard Llama 3.1 chat format. Here’s an example: |
|
|
|
``` |
|
<|begin_of_text|><|start_header_id|>system<|end_header_id|> |
|
|
|
You are a world-class AI system, capable of complex reasoning and reflection. Reason through the query inside <thinking> tags, and then provide your final response inside <output> tags. If you detect that you made a mistake in your reasoning at any point, correct yourself inside <reflection> tags.<|eot_id|><|start_header_id|>user<|end_header_id|> |
|
|
|
What is 2+2?<|eot_id|><|start_header_id|>assistant<|end_header_id|> |
|
``` |
|
|
|
## Tips for Performance |
|
|
|
- A temperature of `.7` and a Top P of `.95` is recommended. |
|
- For increased accuracy, append `Think carefully.` at the end of your prompt. |
|
|
|
## Dataset / Report |
|
|
|
Both the dataset and a brief report detailing how we trained this model will be released next week, alongside our Reflection 405B model that we expect will be the top-performing LLM in the world, including closed-source models. |
|
|
|
Thanks to Jason Kuperberg and Josh Bickett from the [HyperWrite](https://hyperwriteai.com) team for reviewing drafts of the report we'll be releasing next week. |
