File size: 6,997 Bytes
84a78b4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 |
---
license: llama3.1
language:
- en
tags:
- llama3.1
- rag
- brag
- chatrag
- chatqa
---
# QuantFactory/BRAG-Llama-3.1-8b-v0.1-GGUF
This is quantized version of [maximalists/BRAG-Llama-3.1-8b-v0.1](https://huggingface.co/maximalists/BRAG-Llama-3.1-8b-v0.1) created using llama.cpp
# Original Model Card
# BRAG-Llama-3.1-8b-v0.1 Model Card
## Model Description
BRAG-Llama-3.1-8b-v0.1 is part of the [BRAG series of SLMs (Small Language Models)](https://huggingface.co/collections/maximalists/brag-v01-66aefc10e3a6c29c496c7476), specifically trained for RAG (Retrieval-Augmented Generation) tasks, including:
1. RAG with tables and text.
2. RAG with conversational chat.
**Authors**: [Pratik Bhavasar](https://www.linkedin.com/in/bhavsarpratik/), [Ravi Theja](https://www.linkedin.com/in/ravidesetty/)
## Key Features
- **Capabilities**: RAG tasks with both tables and text, as well as conversational chat.
- **Model Size**: 8 billion parameters
- **Context Length**: Supports up to 128k tokens
- **Language**: Trained and evaluated for English, but the base model has multi-lingual capabilities.
## Other Resources
[BRAG-v0.1 Model Collection](https://huggingface.co/collections/maximalists/brag-v01-66aefc10e3a6c29c496c7476) | [Blog](https://themaximalists.substack.com/p/brag)
## Performance
| Model Type | Model Name | Model Size | Context Length | ChatRAG-Bench (all) |
|------------|---------------------------|------------|----------------|---------------------|
| LLM | Command-R-Plus | -- | 128k | 50.93 |
| LLM | GPT-4-Turbo-2024-04-09 | -- | 128k | 54.03 |
| SLM | ChatQA-1.5-8B | 8b | 8k | 55.17 |
| BRAG SLM | BRAG-Qwen2-7b-v0.1 | 7b | 128k | 53.23 |
| BRAG SLM | BRAG-Llama-3.1-8b-v0.1 | 8b | 128k | 52.29 |
| BRAG SLM | BRAG-Llama-3-8b-v0.1 | 8b | 8k | 51.70 |
| BRAG Ultra SLM | BRAG-Qwen2-1.5b-v0.1 | 1.5b | 32k | 46.43 |
## Usage
#### Prompt Format
Below is the message prompt format required for using the model.
```
messages = [
{"role": "system", "content": "You are an assistant who gives helpful, detailed, and polite answers to the user's questions based on the context with appropriate reasoning as required. Indicate when the answer cannot be found in the context."},
{"role": "user", "content": """Context: <CONTEXT INFORMATION> \n\n <USER QUERY>"""},
]
```
#### Running with the `pipeline` API
```python
import transformers
import torch
model_id = "maximalists/BRAG-Llama-3.1-8b-v0.1"
pipeline = transformers.pipeline(
"text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.bfloat16},
device_map="auto",
)
messages = [
{"role": "system", "content": "You are an assistant who gives helpful, detailed, and polite answers to the user's questions based on the context with appropriate reasoning as required. Indicate when the answer cannot be found in the context."},
{"role": "user", "content": """Context:\nArchitecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n\nTo whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?"""},
]
outputs = pipeline(
messages,
max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])
```
#### Running the model on a single / multi GPU
```python
# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_id = "maximalists/BRAG-Llama-3.1-8b-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
)
messages = [
{"role": "system", "content": "You are an assistant who gives helpful, detailed, and polite answers to the user's questions based on the context with appropriate reasoning as required. Indicate when the answer cannot be found in the context."},
{"role": "user", "content": """Context:\nArchitecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n\nTo whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?"""},
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to("cuda")
generated_ids = model.generate(
model_inputs.input_ids,
max_new_tokens=256
)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
```
## Limitations
The model is specifically trained for short contexts and may not perform well with longer ones. It has been fine-tuned on an English dataset. To avoid underperformance and the potential for hallucinations, please use the system prompt mentioned above.
## Citation
To cite this model, please use the following:
```bibtex
@misc{BRAG-Llama-3.1-8b-v0.1,
title = {BRAG-Llama-3.1-8b-v0.1},
year = {2024},
publisher = {HuggingFace},
url = {https://huggingface.co/maximalists/BRAG-Llama-3.1-8b-v0.1},
author = {Pratik Bhavsar and Ravi Theja}
}
```
## Additional Information
For more details on the BRAG series and updates, please refer to the official [blog](https://themaximalists.substack.com/p/brag).
|