Update README.md
Browse files
README.md
CHANGED
|
@@ -17,42 +17,5 @@ inference: false
|
|
| 17 |
* Many thanks to the [Google TPU Research Cloud](https://sites.research.google/trc/about/) for providing access to a TPU cluster!
|
| 18 |
* Continuation of work started during the [Hugging Face community week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organized by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google, for the project [Pre-train T5 from scratch in Dutch](https://discuss.huggingface.co/t/pretrain-t5-from-scratch-in-dutch/8109).
|
| 19 |
* Using improved training script - no more exceptions during training, so no restarting required.
|
| 20 |
-
* All models trained with
|
| 21 |
-
* Thanks to @gsarti for creating the [t5-flax-gcp repository](https://github.com/gsarti/t5-flax-gcp)!
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
| |`t5-base-dutch` |`t5-v1.1-base-dutch` |`t5-v1.1-large-dutch-cased`| `t5-v1.1-base-dutch-uncased`|
|
| 25 |
-
|-----------------------|-------------------------|-------------------------|---------------------------|-----------------------------|
|
| 26 |
-
|`tokenizer` |`cased` |`uncased` |`cased` |`uncased` |
|
| 27 |
-
|`source model config` |`google/t5-base` |`google/t5-v1_1-base` |`google/t5-v1_1-large` |`google/t5-v1_1_base` |
|
| 28 |
-
|`dataset` |`yhavinga/mc4_nl_cleaned`|`yhavinga/mc4_nl_cleaned`|`yhavinga/mc4_nl_cleaned` |`yhavinga/mc4_nl_cleaned` |
|
| 29 |
-
|`tpu vm` | two | one | three | one |
|
| 30 |
-
|`finished` | | YES | | |
|
| 31 |
-
|*Hyperparameters* | | | | |
|
| 32 |
-
|`epochs` | 1 | 1 | 4 | 2 |
|
| 33 |
-
|`per-device batch size`| 16 | 16 | 2 | 8 |
|
| 34 |
-
|`tot. batch size` | 128 | 128 | 16 | ? |
|
| 35 |
-
|`steps` | 508 976 | 508 976 | 8 428 012 | ? |
|
| 36 |
-
|`max seq. length` | 512 | 512 | 1024 | 1024 |
|
| 37 |
-
|`tot. tok. trained on` | 33B | 33B | 138B | ? |
|
| 38 |
-
|`optimizer` | adafactor | adafactor | adafactor | adafactor |
|
| 39 |
-
|`warmup steps` | 10000 | 10000 | 10000 | 10000 |
|
| 40 |
-
|`learning rate` | 0.005 | 0.005 | 0.005 | 0.005 |
|
| 41 |
-
|`weigth decay` | 0.01 | 0.01 | 0.01 | 0.001 |
|
| 42 |
-
|`tie embeds` |`false` |`false` |`false` |`false` |
|
| 43 |
-
|`validation split size`| 15K examples | 15K examples | 15K examples | 15K examples |
|
| 44 |
-
|*Model config* | | | | |
|
| 45 |
-
|`d_ff` | 3072 | 2048 | 2816 | 2048 |
|
| 46 |
-
|`d_kv` | 64 | 64 | 64 | 64 |
|
| 47 |
-
|`d_model` | 768 | 768 | 1024 | 768 |
|
| 48 |
-
|`dropout rate` | 0.1 | 0.1 | 0.1 (0.0 wh. pre-train.) | 0.1 (0.0 wh. pre-train.) |
|
| 49 |
-
|`ff projection` |`relu` |`gated-gelu` |`gated-gelu` |`gated-relu` |
|
| 50 |
-
|`num decoder layers` | 12 | 12 | 24 | 12 |
|
| 51 |
-
|`num heads` | 12 | 12 | 16 | 12 |
|
| 52 |
-
|`num layers` | 12 | 12 | 24 | 12 |
|
| 53 |
-
|`rel. attn. buckets` | 32 | 32 | 32 | 32 |
|
| 54 |
-
|`vocab size` | 32103 | 32103 | 32103 | 32103 |
|
| 55 |
-
|*Training time* | ~ 100 hours | 101 hours | ~ 370 hours | ? |
|
| 56 |
-
|*Evaluation* | | | | |
|
| 57 |
-
|`accuracy` | | 0.6976 | | |
|
| 58 |
-
|`loss` | | 1.379 | | |
|
|
|
|
| 17 |
* Many thanks to the [Google TPU Research Cloud](https://sites.research.google/trc/about/) for providing access to a TPU cluster!
|
| 18 |
* Continuation of work started during the [Hugging Face community week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organized by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google, for the project [Pre-train T5 from scratch in Dutch](https://discuss.huggingface.co/t/pretrain-t5-from-scratch-in-dutch/8109).
|
| 19 |
* Using improved training script - no more exceptions during training, so no restarting required.
|
| 20 |
+
* All models trained with tensorboard metrics.
|
| 21 |
+
* Thanks to @gsarti for creating the [t5-flax-gcp repository](https://github.com/gsarti/t5-flax-gcp)!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|