---
license: llama2
library_name: peft
tags:
- llama-2
datasets:
- ehartford/dolphin
inference: false
pipeline_tag: text-generation
base_model: meta-llama/Llama-2-13b-hf
---
# Llama-2-13B-Instruct-v0.1
This instruction model was built via parameter-efficient QLoRA finetuning of [llama-2-13b](https://huggingface.co/meta-llama/Llama-2-13b-hf) on the first 100k rows of [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin) (an open-source implementation of [Microsoft's Orca](https://www.microsoft.com/en-us/research/publication/orca-progressive-learning-from-complex-explanation-traces-of-gpt-4/)). Finetuning was executed on a single A6000 (48 GB) for roughly 18 hours on the [Lambda Labs](https://cloud.lambdalabs.com/instances) platform.
## Benchmark metrics
| Metric | Value |
|-----------------------|-------|
| MMLU (5-shot) | 55.76 |
| ARC (25-shot) | 59.98 |
| HellaSwag (10-shot) | 82.43 |
| TruthfulQA (0-shot) | 42.59 |
| Avg. | 60.19 |
We use state-of-the-art [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) to run the benchmark tests above, using the same version as Hugging Face's [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
## Helpful Links
* Model license: Llama 2 Community License Agreement
* Basic usage: [notebook](assets/basic_inference_llama_2_13b_dolphin.ipynb)
* Finetuning script: [script](https://github.com/daniel-furman/sft-demos/blob/main/src/sft/one_gpu/llama-2/dolphin/sft-Llama-2-13B-Instruct-v0.1-peft.py)
* Loss curves: [plot](https://huggingface.co/dfurman/Llama-2-13B-Instruct-v0.1-peft#finetuning-description)
* Runtime stats: [table](https://huggingface.co/dfurman/Llama-2-13B-Instruct-v0.1-peft#runtime-tests)
## Example prompts and responses
Example 1:
**User**:
>You are a helpful assistant. Write me a numbered list of things to do in New York City.\n
**Llama-2-13B-Instruct-v0.1-peft**:
> 1. Visit the iconic Statue of Liberty and Ellis Island.
> 2. Take a stroll through Central Park and enjoy its many attractions.
> 3. Explore the world-renowned museums, such as the Metropolitan Museum of Art and the Museum of Modern Art.
> 4. Experience the vibrant energy of Times Square and take in the bright lights and billboards.
> 5. Visit the 9/11 Memorial and Museum to pay tribute to those who lost their lives in the attacks.
> 6. Enjoy a Broadway show or a concert at one of the many theaters and venues in the city.
> 7. Take a ride on the Staten Island Ferry for a free view of the Statue of Liberty and the New York City skyline.
> 8. Shop at the famous Fifth Avenue stores and explore the high-end boutiques.
> 9. Indulge in a variety of cuisines at one of the many restaurants in the city.
> 10. Visit the Empire State Building and enjoy the panoramic views of the city from the observation deck.
Example 2:
**User**:
>You are a helpful assistant. Write a short email inviting my friends to a dinner party on Friday. Respond succinctly.\n"
**Llama-2-13B-Instruct-v0.1-peft**:
> Subject: Friday Dinner Party Invitation
>
> Dear Friends,
>
> I hope this email finds you well. I'm excited to invite you all to a dinner party on Friday, March 10th, at 7:00 PM. The address is 123 Main Street, Anytown, USA.
>
> Please RSVP by Wednesday, March 8th, so I can plan accordingly. I look forward to seeing you all and sharing a delicious meal together!
>
> Best,
> Your Friendly Assistant
>
> P.S. If you have any dietary restrictions or allergies, please let me know in your RSVP. Thank you!
## Model Description
The architecture is a modification of a standard decoder-only transformer.
The llama-2-13b models have been modified from a standard transformer in the following ways:
* It uses the [SwiGLU activation function](https://arxiv.org/abs/2002.05202)
* It uses [rotary positional embeddings](https://arxiv.org/abs/2104.09864) (RoPE)
| Hyperparameter | Value |
|----------------|-------|
| n_parameters | 13B |
| tokens | 2.0T |
| vocab size | 32000 |
| sequence length | 4096 |
## Finetuning Description
This model was trained on a single A6000 (48 GB) for about 18 hours using the [Lambda Labs](https://cloud.lambdalabs.com/instances) platform.
The above loss curve was generated from the run's private wandb.ai log.
## PreTraining Data
For more details on the pretraining process, see [Llama-2-13b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf).
The data was tokenized using the [Llama-2-13b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf) tokenizer.
## Limitations and Biases
_The following language is modified from [EleutherAI's GPT-NeoX-20B](https://huggingface.co/EleutherAI/gpt-neox-20b)_
This model can produce factually incorrect output, and should not be relied on to produce factually accurate information.
This model was trained on various public datasets.
While great efforts have been taken to clean the pretraining data, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
## How to Use
* [notebook](assets/basic_inference_llama_2_dolphin.ipynb)
```python
!pip install -q -U huggingface_hub peft transformers torch accelerate
```
```python
from huggingface_hub import notebook_login
import torch
from peft import PeftModel, PeftConfig
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
pipeline,
)
notebook_login()
```
```python
peft_model_id = "dfurman/Llama-2-13B-Instruct-v0.1-peft"
config = PeftConfig.from_pretrained(peft_model_id)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
config.base_model_name_or_path,
quantization_config=bnb_config,
use_auth_token=True,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path, use_fast=True)
tokenizer.pad_token = tokenizer.eos_token
model = PeftModel.from_pretrained(model, peft_model_id)
format_template = "You are a helpful assistant. {query}\n"
```
```python
# First, format the prompt
query = "Tell me a recipe for vegan banana bread."
prompt = format_template.format(query=query)
# Inference can be done using model.generate
print("\n\n*** Generate:")
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
with torch.autocast("cuda", dtype=torch.bfloat16):
output = model.generate(
input_ids=input_ids,
max_new_tokens=512,
do_sample=True,
temperature=0.7,
return_dict_in_generate=True,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
repetition_penalty=1.2,
)
print(tokenizer.decode(output["sequences"][0], skip_special_tokens=True))
```
## Runtime tests
| runtime / 50 tokens (sec) | GPU | attn | torch dtype | VRAM (GB) |
|:-----------------------------:|:----------------------:|:---------------------:|:-------------:|:-----------------------:|
| 2.93 | 1x A100 (40 GB SXM) | torch | bfloat16 | 25 |
| 3.24 | 1x A6000 (48 GB) | torch | bfloat16 | 25 |
The above runtime stats were generated from this [notebook](https://github.com/daniel-furman/sft-demos/blob/main/src/sft/one_gpu/llama-2/dolphin/postprocessing-Llama-2-13B-Instruct-v0.1-peft.ipynb).
## Acknowledgements
This model was finetuned by Daniel Furman on July 22, 2023 and is intended primarily for research purposes.
## Disclaimer
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please cosult an attorney before using this model for commercial purposes.
## Meta citation for llama-2 blog
```
@online{Meta2023Introducing,
author = {Meta AI},
title = {Meta and Microsoft Introduce the Next Generation of Llama},
year = {2023},
url = {https://about.fb.com/news/2023/07/llama-2/},
note = {Accessed: 2023-07-24},
urldate = {2023-07-24}
}
```
---
## Framework versions
- PEFT 0.5.0.dev0
