pretrain eval

README.md

@@ -44,15 +44,17 @@ This model **isn't** designed for immediate use but rather for Continued Pretrai
 
 The objective is to streamline the cognitive or reasoning core, eliminating any redundant knowledge from the model.
 
-[loss, val_loss]()
+[loss, val_loss](https://api.wandb.ai/links/mtasic85/strnx9rl)
 
-[val_ppl]()
+[val_ppl](https://api.wandb.ai/links/mtasic85/ljwxf4am)
 
-[epoch]()
+[epoch](https://api.wandb.ai/links/mtasic85/edyph869)
 
-[learning_rate]()
+[learning_rate](https://api.wandb.ai/links/mtasic85/eswxyger)
 
-## lm-evaluation-harness
+## Pretrain Evaluation
+
+### lm-evaluation-harness
 
 ```bash
 litgpt evaluate --tasks 'hellaswag,gsm8k,truthfulqa_mc2,mmlu,winogrande,arc_challenge' --out_dir 'evaluate-quick/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
@@ -256,10 +258,6 @@ litgpt evaluate --tasks 'arc_challenge,boolq,gpqa,hellaswag,openbookqa,piqa,trut
 |truthfulqa_mc2                 |      2|none            |     0|acc        |↑  |0.5061|±  |0.0167|
 |winogrande                     |      1|none            |     0|acc        |↑  |0.4933|±  |0.0141|
 
-```bash
-litgpt evaluate --tasks 'mmlu_multilingual,mgsm' --out_dir 'evaluate-multilinguals/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
-```
-
 ```bash
 litgpt evaluate --tasks 'wikitext,qasper' --out_dir 'evaluate-long/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
 ```