Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
datasets:
|
| 3 |
+
- HuggingFaceFW/fineweb
|
| 4 |
+
base_model:
|
| 5 |
+
- openai-community/gpt2
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
# NanoGPT Speedrun
|
| 9 |
+
|
| 10 |
+
Following https://github.com/KellerJordan/modded-nanogpt for fun (learning).
|
| 11 |
+
|
| 12 |
+
## Run Info
|
| 13 |
+
|
| 14 |
+
**baseline/**
|
| 15 |
+
|
| 16 |
+
- Run on lightning cloud, using one L40S
|
| 17 |
+
- Batch size set to 32
|
| 18 |
+
- VRAM usage: 26.95GB (25698MB reported in `nvidia-smi`)
|
| 19 |
+
- 4 seconds per step, total 3200 steps
|
| 20 |
+
- Checkpoint saved every 320 steps
|