File size: 1,026 Bytes
fc83c7a 27585b5 fc83c7a 27585b5 10ac2c8 27585b5 10ac2c8 27585b5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
---
license: apache-2.0
datasets:
- databricks/databricks-dolly-15k
---
# Instruct_Phi2_Dolly15K
Fine-tuned from phi2，used Dolly15k for the dataset. 90% for training, 10% validation. Trained for 2.0 epochs using QLora. Trained with 1024 context window.
# Model Details
* **Trained by**: trained by HenryJJ.
* **Model type:** **Instruct_Phi2_Dolly15K** is an auto-regressive language model based on the phi 2 transformer architecture.
* **Language(s)**: English
* **License for HenryJJ/Instruct_Phi2_Dolly15K**: apache-2.0 license
# Prompting
## Prompt Template With Context
chatml format
```
<|im_start|>system
{instruction}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```
## Prompt Template Without Context
```
<|im_start|>system
{instruction}<|im_end|>
<|im_start|>assistant
```
# Training script:
Fully opensourced at: https://github.com/hengjiUSTC/learn-llm/blob/main/trl_finetune.py. Run on 1 A10G instance for 4 hours.
```
python3 trl_finetune.py --config configs/phi2-dolly.yml
``` |