Update README.md
Browse files
README.md
CHANGED
|
@@ -5,6 +5,7 @@ datasets:
|
|
| 5 |
---
|
| 6 |
|
| 7 |
This is a one-layer base model with the LlaMA 2 architecture trained on 6B tokens of the algebraic-stack part of the Proof-pile 2 dataset. \
|
|
|
|
| 8 |
The tokenizer is the LlaMA 2 one. I used the following hyper parameters:\
|
| 9 |
d<sub>model</sub> = 512 \
|
| 10 |
d<sub>ff</sub> = 2048 \
|
|
|
|
| 5 |
---
|
| 6 |
|
| 7 |
This is a one-layer base model with the LlaMA 2 architecture trained on 6B tokens of the algebraic-stack part of the Proof-pile 2 dataset. \
|
| 8 |
+
It's output distribution is thus mostly concerned with code.
|
| 9 |
The tokenizer is the LlaMA 2 one. I used the following hyper parameters:\
|
| 10 |
d<sub>model</sub> = 512 \
|
| 11 |
d<sub>ff</sub> = 2048 \
|