Update README.md
Browse files
README.md
CHANGED
|
@@ -3,5 +3,66 @@ language:
|
|
| 3 |
- fi
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
---
|
|
|
|
| 6 |
|
| 7 |
-
GPT-3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
- fi
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
---
|
| 6 |
+
Generative Pretrained Transformer with 1.5B parameteres for Finnish.
|
| 7 |
|
| 8 |
+
TurkuNLP Finnish GPT-3-models are a model family of pretrained monolingual GPT-style language models that are based on BLOOM-architecture.
|
| 9 |
+
Note that the models are pure language models, meaning that they are not [instruction finetuned](https://arxiv.org/abs/2203.02155) for dialogue
|
| 10 |
+
or answering questions.
|
| 11 |
+
|
| 12 |
+
These models are intended to be used as foundational models that can be e.g. instruction finetuned to serve as modern chat-models.
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
**Parameters**
|
| 17 |
+
| Model | Layers | Dim | Heads | Params |
|
| 18 |
+
|--------|--------|------|-------|--------|
|
| 19 |
+
| Small | 12 | 768 | 12 | 186M |
|
| 20 |
+
| Medium | 24 | 1024 | 16 | 437M |
|
| 21 |
+
| Large | 24 | 1536 | 16 | 881M |
|
| 22 |
+
| XL | 24 | 2064 | 24 | 1.5B |
|
| 23 |
+
| ”3B” | 32 | 2560 | 32 | 2.8B |
|
| 24 |
+
| ”8B” | 32 | 4096 | 32 | 7.5B |
|
| 25 |
+
| "13B" | 40 | 5120 | 40 | 13.3B |
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
**Datasets**
|
| 29 |
+
|
| 30 |
+
We used a combination of multiple Finnish resources.
|
| 31 |
+
|
| 32 |
+
* Finnish Internet Parsebank https://turkunlp.org/finnish_nlp.html
|
| 33 |
+
mC4 multilingual colossal, cleaned Common Crawl https://huggingface.co/datasets/mc4
|
| 34 |
+
* Common Crawl Finnish https://TODO
|
| 35 |
+
* Finnish Wikipedia https://fi.wikipedia.org/wiki
|
| 36 |
+
* Lönnrot Projekti Lönnrot http://www.lonnrot.net/
|
| 37 |
+
* ePub National library ”epub” collection
|
| 38 |
+
* National library ”lehdet” collection
|
| 39 |
+
* Suomi24 The Suomi 24 Corpus 2001-2020 http://urn.fi/urn:nbn:fi:lb-2021101527
|
| 40 |
+
* Reddit r/Suomi submissions and comments https://www.reddit.com/r/Suomi
|
| 41 |
+
* STT Finnish News Agency Archive 1992-2018 http://urn.fi/urn:nbn:fi:lb-2019041501
|
| 42 |
+
* Yle Finnish News Archive 2011-2018 http://urn.fi/urn:nbn:fi:lb-2017070501
|
| 43 |
+
* Yle Finnish News Archive 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050401
|
| 44 |
+
* Yle News Archive Easy-to-read Finnish 2011-2018 http://urn.fi/urn:nbn:fi:lb-2019050901
|
| 45 |
+
* Yle News Archive Easy-to-read Finnish 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050701
|
| 46 |
+
* ROOTS TODO
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
**Sampling ratios**
|
| 50 |
+
|
| 51 |
+
|Dataset | Chars | Ratio | Weight | W.Ratio |
|
| 52 |
+
|----------|--------|---------|--------|---------|
|
| 53 |
+
|Parsebank | 35.0B | 16.9\% | 1.5 | 22.7\%|
|
| 54 |
+
|mC4-Fi | 46.3B | 22.4\% | 1.0 | 20.0\%|
|
| 55 |
+
|CC-Fi | 79.6B | 38.5\% | 1.0 | 34.4\%|
|
| 56 |
+
|Fiwiki | 0.8B | 0.4\% | 3.0 | 1.0\%|
|
| 57 |
+
|Lönnrot | 0.8B | 0.4\% | 3.0 | 1.0\%|
|
| 58 |
+
|Yle | 1.6B | 0.8\% | 2.0 | 1.4\%|
|
| 59 |
+
|STT | 2.2B | 1.1\% | 2.0 | 1.9\%|
|
| 60 |
+
|ePub | 13.5B | 6.5\% | 1.0 | 5.8\%|
|
| 61 |
+
|Lehdet | 5.8B | 2.8\% | 1.0 | 2.5\%|
|
| 62 |
+
|Suomi24 | 20.6B | 9.9\% | 1.0 | 8.9\%|
|
| 63 |
+
|Reddit-Fi | 0.7B | 0.4\% | 1.0 | 0.3\%|
|
| 64 |
+
|**TOTAL** | **207.0B** | **100.0\%** | **N/A** | **100.0\%** |
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
More documentation and a paper coming soon.
|