Update README.md
Browse files
README.md
CHANGED
@@ -10,7 +10,14 @@ pipeline_tag: text-generation
|
|
10 |
base_model: meta-llama/Llama-2-13b-hf
|
11 |
---
|
12 |
|
13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
14 |
|
15 |
This instruction model was built via parameter-efficient QLoRA finetuning of [llama-2-13b](https://huggingface.co/meta-llama/Llama-2-13b-hf) on the first 100k rows of [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin) (an open-source implementation of [Microsoft's Orca](https://www.microsoft.com/en-us/research/publication/orca-progressive-learning-from-complex-explanation-traces-of-gpt-4/)). Finetuning was executed on a single A6000 (48 GB) for roughly 18 hours on the [Lambda Labs](https://cloud.lambdalabs.com/instances) platform.
|
16 |
|
@@ -30,9 +37,9 @@ We use state-of-the-art [Language Model Evaluation Harness](https://github.com/E
|
|
30 |
|
31 |
* Model license: Llama 2 Community License Agreement
|
32 |
* Basic usage: [notebook](assets/basic_inference_llama_2_13b_dolphin.ipynb)
|
33 |
-
* Finetuning script: [script](https://github.com/daniel-furman/sft-demos/blob/main/src/sft/one_gpu/llama-2/dolphin/sft-
|
34 |
-
* Loss curves: [plot](https://huggingface.co/dfurman/
|
35 |
-
* Runtime stats: [table](https://huggingface.co/dfurman/
|
36 |
|
37 |
## Example prompts and responses
|
38 |
|
@@ -41,7 +48,7 @@ Example 1:
|
|
41 |
**User**:
|
42 |
>You are a helpful assistant. Write me a numbered list of things to do in New York City.\n
|
43 |
|
44 |
-
**
|
45 |
|
46 |
> 1. Visit the iconic Statue of Liberty and Ellis Island.
|
47 |
> 2. Take a stroll through Central Park and enjoy its many attractions.
|
@@ -61,7 +68,7 @@ Example 2:
|
|
61 |
**User**:
|
62 |
>You are a helpful assistant. Write a short email inviting my friends to a dinner party on Friday. Respond succinctly.\n"
|
63 |
|
64 |
-
**
|
65 |
|
66 |
> Subject: Friday Dinner Party Invitation
|
67 |
>
|
@@ -97,7 +104,7 @@ The llama-2-13b models have been modified from a standard transformer in the fol
|
|
97 |
|
98 |
This model was trained on a single A6000 (48 GB) for about 18 hours using the [Lambda Labs](https://cloud.lambdalabs.com/instances) platform.
|
99 |
|
100 |
-
![loss curves](https://raw.githubusercontent.com/daniel-furman/sft-demos/main/assets/
|
101 |
|
102 |
The above loss curve was generated from the run's private wandb.ai log.
|
103 |
|
@@ -138,7 +145,7 @@ notebook_login()
|
|
138 |
```
|
139 |
|
140 |
```python
|
141 |
-
peft_model_id = "dfurman/
|
142 |
config = PeftConfig.from_pretrained(peft_model_id)
|
143 |
|
144 |
bnb_config = BitsAndBytesConfig(
|
@@ -194,7 +201,7 @@ print(tokenizer.decode(output["sequences"][0], skip_special_tokens=True))
|
|
194 |
| 2.93 | 1x A100 (40 GB SXM) | torch | bfloat16 | 25 |
|
195 |
| 3.24 | 1x A6000 (48 GB) | torch | bfloat16 | 25 |
|
196 |
|
197 |
-
The above runtime stats were generated from this [notebook](https://github.com/daniel-furman/sft-demos/blob/main/src/sft/one_gpu/llama-2/dolphin/postprocessing-
|
198 |
|
199 |
## Acknowledgements
|
200 |
|
|
|
10 |
base_model: meta-llama/Llama-2-13b-hf
|
11 |
---
|
12 |
|
13 |
+
<div align="center">
|
14 |
+
|
15 |
+
<img src="./assets/llama.png" width="150px">
|
16 |
+
|
17 |
+
</div>
|
18 |
+
|
19 |
+
|
20 |
+
# Llama-2-13B-Instruct-v0.1 🦙🐬
|
21 |
|
22 |
This instruction model was built via parameter-efficient QLoRA finetuning of [llama-2-13b](https://huggingface.co/meta-llama/Llama-2-13b-hf) on the first 100k rows of [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin) (an open-source implementation of [Microsoft's Orca](https://www.microsoft.com/en-us/research/publication/orca-progressive-learning-from-complex-explanation-traces-of-gpt-4/)). Finetuning was executed on a single A6000 (48 GB) for roughly 18 hours on the [Lambda Labs](https://cloud.lambdalabs.com/instances) platform.
|
23 |
|
|
|
37 |
|
38 |
* Model license: Llama 2 Community License Agreement
|
39 |
* Basic usage: [notebook](assets/basic_inference_llama_2_13b_dolphin.ipynb)
|
40 |
+
* Finetuning script: [script](https://github.com/daniel-furman/sft-demos/blob/main/src/sft/one_gpu/llama-2/dolphin/sft-Llama-2-13B-Instruct-v0.1-peft.py)
|
41 |
+
* Loss curves: [plot](https://huggingface.co/dfurman/Llama-2-13B-Instruct-v0.1-peft#finetuning-description)
|
42 |
+
* Runtime stats: [table](https://huggingface.co/dfurman/Llama-2-13B-Instruct-v0.1-peft#runtime-tests)
|
43 |
|
44 |
## Example prompts and responses
|
45 |
|
|
|
48 |
**User**:
|
49 |
>You are a helpful assistant. Write me a numbered list of things to do in New York City.\n
|
50 |
|
51 |
+
**Llama-2-13B-Instruct-v0.1-peft**:
|
52 |
|
53 |
> 1. Visit the iconic Statue of Liberty and Ellis Island.
|
54 |
> 2. Take a stroll through Central Park and enjoy its many attractions.
|
|
|
68 |
**User**:
|
69 |
>You are a helpful assistant. Write a short email inviting my friends to a dinner party on Friday. Respond succinctly.\n"
|
70 |
|
71 |
+
**Llama-2-13B-Instruct-v0.1-peft**:
|
72 |
|
73 |
> Subject: Friday Dinner Party Invitation
|
74 |
>
|
|
|
104 |
|
105 |
This model was trained on a single A6000 (48 GB) for about 18 hours using the [Lambda Labs](https://cloud.lambdalabs.com/instances) platform.
|
106 |
|
107 |
+
![loss curves](https://raw.githubusercontent.com/daniel-furman/sft-demos/main/assets/jul_24_23_1_13_00_log_loss_curves_Llama-2-13B-Instruct-v0.1.png)
|
108 |
|
109 |
The above loss curve was generated from the run's private wandb.ai log.
|
110 |
|
|
|
145 |
```
|
146 |
|
147 |
```python
|
148 |
+
peft_model_id = "dfurman/Llama-2-13B-Instruct-v0.1-peft"
|
149 |
config = PeftConfig.from_pretrained(peft_model_id)
|
150 |
|
151 |
bnb_config = BitsAndBytesConfig(
|
|
|
201 |
| 2.93 | 1x A100 (40 GB SXM) | torch | bfloat16 | 25 |
|
202 |
| 3.24 | 1x A6000 (48 GB) | torch | bfloat16 | 25 |
|
203 |
|
204 |
+
The above runtime stats were generated from this [notebook](https://github.com/daniel-furman/sft-demos/blob/main/src/sft/one_gpu/llama-2/dolphin/postprocessing-Llama-2-13B-Instruct-v0.1-peft.ipynb).
|
205 |
|
206 |
## Acknowledgements
|
207 |
|