---
library_name: transformers
license: llama3.1
base_model:
- meta-llama/Llama-3.1-405B-Instruct
---
# This model has been xMADified!

This repository contains [`meta-llama/Llama-3.1-405B-Instruct`](https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct) quantized from 16-bit floats to 4-bit integers, using xMAD.ai proprietary technology.

# Why should I use this model?

1. **Accuracy:** This xMADified model is the best **quantized** version of the `meta-llama/Llama-3.1-405B-Instruct` model (200 GB only). See _Table 1_ below for model quality benchmarks.

2. **Memory-efficiency:** The full-precision model is around 800 GB, while this xMADified model is only around 200 GB, making it feasible to run on a 4x 80GB GPUs.

3. **Fine-tuning**:  These models are fine-tunable over the same reduced (4x 80GB GPUs) hardware in mere 3-clicks. Watch our product demo [here](https://www.youtube.com/watch?v=S0wX32kT90s&list=TLGGL9fvmJ-d4xsxODEwMjAyNA)


## Table 1: xMAD vs. Hugging Quants

| Model                                                                                                                                 | MMLU STEM | MMLU Humanities | MMLU Social Sciences | MMLU Other | LAMBADA Standard | LAMBADA OpenAI | WinoGrande | PIQA      |
| ------------------------------------------------------------------------------------------------------------------------------------- | --------- | --------------- | -------------------- | ---------- | ---------------- | -------------- | ---------- | --------- |
| [xmadai/Llama-3.1-405B-Instruct-xMADai-INT4](https://huggingface.co/xmadai/Llama-3.1-405B-Instruct-xMADai-INT4) (this model)          | **82.68** | **83.21**       | **90.58**            | **87.71**  | **74.71**        | **77.55**      | **82.32**  | **83.35** |
| [hugging-quants/Meta-Llama-3.1-405B-Instruct-GPTQ-INT4](https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-GPTQ-INT4) | 82.34     | 82.64           | 90.45                | 87.51      | 73.41            | 76.96          | 81.77      | 82.75     |


# How to Run Model

Loading the model checkpoint of this xMADified model requires around 200 GiB of VRAM. Hence it can be efficiently run on 4x 80GB or 8x 40GB GPUs.

**Package prerequisites**:

1. Run the following *commands to install the required packages.
```bash
pip install torch==2.4.0  # Run following if you have CUDA version 11.8: pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu118
pip install transformers accelerate optimum
pip install -vvv --no-build-isolation "git+https://github.com/PanQiWei/AutoGPTQ.git@v0.7.1"
```
**Sample Inference Code**
```python
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM
model_id = "xmadai/Llama-3.1-405B-Instruct-xMADai-INT4"
prompt = [
    {"role": "system", "content": "You are a helpful assistant, that responds as a pirate."},
    {"role": "user", "content": "What's Deep Learning?"},
]
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
inputs = tokenizer.apply_chat_template(
    prompt,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to("cuda")
model = AutoGPTQForCausalLM.from_quantized(
    model_id,
    device_map='auto',
    trust_remote_code=True,
)
outputs = model.generate(**inputs, do_sample=True, max_new_tokens=1024)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
```

# Citation

If you found this model useful, please cite our research paper.

```
@article{zhang2024leanquant,
  title={LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid},
  author={Zhang, Tianyi and Shrivastava, Anshumali},
  journal={arXiv preprint arXiv:2407.10032},
  year={2024},
  url={https://arxiv.org/abs/2407.10032},
}
```

# Contact Us
For additional xMADified models, access to fine-tuning, and general questions, please contact us at support@xmad.ai and join our waiting list.