File size: 1,817 Bytes
a25977f 5294cb7 c96faea 1732c47 a25977f 5294cb7 0f1544f 5294cb7 0f1544f 5294cb7 0f1544f 5294cb7 0f1544f c96faea 0f1544f c96faea 0f1544f c96faea 0f1544f c96faea 0f1544f c96faea 0f1544f c96faea 0f1544f c96faea 0f1544f 2be038b 1732c47 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
---
library_name: peft
tags:
- tiiuae
- code
- instruct
- databricks-dolly-15k
- falcon-40b
datasets:
- databricks/databricks-dolly-15k
base_model: tiiuae/falcon-40b
license: apache-2.0
---
### Finetuning Overview:
**Model Used:** tiiuae/falcon-40b
**Dataset:** Databricks-dolly-15k
#### Dataset Insights:
The Databricks-dolly-15k dataset, comprising over 15,000 records, stands as a testament to the dedication of numerous Databricks professionals. Aimed at refining the interactive capabilities of systems like ChatGPT, the dataset offers:
- Prompt/response pairs across eight distinct instruction categories.
- A blend of the seven categories from the InstructGPT paper and an open-ended category.
- Original content, devoid of generative AI influence and primarily offline-sourced, with exceptions for Wikipedia references.
- Interactive sessions where contributors could address and rephrase peer questions.
Note: Some data categories incorporate Wikipedia references, evident from bracketed citation numbers, e.g., [42]. Exclusion is recommended for downstream applications.
#### Finetuning Details:
Leveraging [MonsterAPI](https://monsterapi.ai)'s no-code [LLM finetuner](https://docs.monsterapi.ai/fine-tune-a-large-language-model-llm), our finetuning emphasized:
- **Cost-Effectiveness:** A complete run at just `$11.8`.
- **Efficiency:** Using an A6000 48GB GPU, the session concluded in 5 hours and 40 minutes.
#### Hyperparameters & Additional Details:
- **Epochs:** 1
- **Learning Rate:** 0.0002
- **Data Split:** Training 90% / Validation 10%
- **Gradient Accumulation Steps:** 4
---
### Prompt Structure:
```
### INSTRUCTION:
[instruction]
[context]
### RESPONSE:
[response]
```
Loss metrics
Training loss:
---
license: apache-2.0 |