Update README.md (#1)
Browse files- Update README.md (704093fb3d36ad12d970386311c3473ba65d0ca5)
Co-authored-by: Anton Shapkin <[email protected]>
README.md
CHANGED
|
@@ -10,6 +10,14 @@ This is CodeLlama model fine-tuned on Kotlin Exercices dataset.
|
|
| 10 |
|
| 11 |
The model was trained on one A100 GPU with following hyperparameters:
|
| 12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
# Fine-tuning data
|
| 14 |
|
| 15 |
For this model we used 15K exmaples of Kotlin Exercices dataset. For more information about the dataset follow th link.
|
|
|
|
| 10 |
|
| 11 |
The model was trained on one A100 GPU with following hyperparameters:
|
| 12 |
|
| 13 |
+
| **Hyperparameter** | **Value** |
|
| 14 |
+
|:---------------------------:|:----------------------------------------:|
|
| 15 |
+
| `warmup` | 10% |
|
| 16 |
+
| `max_lr` | 1e-4 |
|
| 17 |
+
| `scheduler` | linear |
|
| 18 |
+
| `total_batch_size` | 256 (~130K tokens per step) |
|
| 19 |
+
|
| 20 |
+
|
| 21 |
# Fine-tuning data
|
| 22 |
|
| 23 |
For this model we used 15K exmaples of Kotlin Exercices dataset. For more information about the dataset follow th link.
|