|
--- |
|
license: mit |
|
language: |
|
- en |
|
tags: |
|
- babylm |
|
--- |
|
|
|
# Lil-Bevo-X |
|
|
|
Lil-Bevo-X is UT Austin's submission to the BabyLM challenge, specifically the *strict* track. |
|
|
|
[Link to GitHub Repo](https://github.com/venkatasg/Lil-Bevo) |
|
|
|
## Model training regime: |
|
1. 5 epochs on MAESTRO dataset (85M non-language music tokens) combined with strict small dataset. |
|
2. 50 epochs of pretraining with sequence length of 128 on strict dataset. |
|
3. 150 epochs of pretraining with sequence length of 512 on strict dataset. |
|
4. 10 epochs of targeted MLM. |
|
|
|
|
|
This README will be updated with more details soon. |
