File size: 585 Bytes
533c7c6 de9e48f ad6f442 533c7c6 de9e48f b2361ea de9e48f ad6f442 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
---
license: mit
language:
- en
tags:
- babylm
---
# Lil-Bevo-X
Lil-Bevo-X is UT Austin's submission to the BabyLM challenge, specifically the *strict* track.
[Link to GitHub Repo](https://github.com/venkatasg/Lil-Bevo)
## Model training regime:
1. 5 epochs on MAESTRO dataset (85M non-language music tokens) combined with strict small dataset.
2. 50 epochs of pretraining with sequence length of 128 on strict dataset.
3. 150 epochs of pretraining with sequence length of 512 on strict dataset.
4. 10 epochs of targeted MLM.
This README will be updated with more details soon. |