OpenNLPLab
commited on
Commit
•
590b4b9
1
Parent(s):
9e5c19e
Update README.md
Browse files
README.md
CHANGED
@@ -25,6 +25,9 @@ This official repository unveils the TransNormerLLM3 model along with its open-s
|
|
25 |
- **TransNormerLLM3-15B** features **14.83 billion** parameters. It is structured with **42 layers**, includes **40 attention heads**, and has a total **embedding size of 5120**.
|
26 |
- **TransNormerLLM3-15B** is purely intergrated with **[Lightning Attention-2](http://arxiv.org/abs/2401.04658)**, which can maintain a **stable TGS** during training of **unlimited sequence lengths**, up until encountering firm limitations like GPU memory constraints.
|
27 |
- **Titoken** tokenizer is used with a total **vocabulary size** of about **100,000**.
|
|
|
|
|
|
|
28 |
|
29 |
### Pre-training Logbook
|
30 |
* Realtime Track: https://api.wandb.ai/links/opennlplab/kip314lq
|
@@ -93,5 +96,6 @@ If you wish to cite our work, please use the following reference:
|
|
93 |
```
|
94 |
|
95 |
<p align="center">
|
|
|
96 |
- OpenNLPLab @2024 -
|
97 |
</p>
|
|
|
25 |
- **TransNormerLLM3-15B** features **14.83 billion** parameters. It is structured with **42 layers**, includes **40 attention heads**, and has a total **embedding size of 5120**.
|
26 |
- **TransNormerLLM3-15B** is purely intergrated with **[Lightning Attention-2](http://arxiv.org/abs/2401.04658)**, which can maintain a **stable TGS** during training of **unlimited sequence lengths**, up until encountering firm limitations like GPU memory constraints.
|
27 |
- **Titoken** tokenizer is used with a total **vocabulary size** of about **100,000**.
|
28 |
+
<p align="center">
|
29 |
+
<img src="./images/TransNormer3.jpg" width="75%" />
|
30 |
+
</p>
|
31 |
|
32 |
### Pre-training Logbook
|
33 |
* Realtime Track: https://api.wandb.ai/links/opennlplab/kip314lq
|
|
|
96 |
```
|
97 |
|
98 |
<p align="center">
|
99 |
+
<img src="./images/lightning3-leopard.jpg" width="67%" />
|
100 |
- OpenNLPLab @2024 -
|
101 |
</p>
|