File size: 1,547 Bytes
5a05946 d87f480 73146d5 5a05946 d87f480 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
---
language:
- "en"
thumbnail: "https://styles.redditmedia.com/t5_2to41/styles/communityIcon_qedoavxzocr61.png?width=256&s=9c7c19b81474c3788279b8d6d6823e791d0524fc"
datasets:
- "reddit_tifu (subset: short)"
widget:
- text: "I told my friend"
license: mit
---
# mgfrantz/distilgpt2-finetuned-reddit-tifu
This model was trained to as practice for fine-tuning a causal language model.
There was no intended use case for this model besides having some fun seeing how different things might be screwed up.
## Data
This model was trained on "short" subset of [`reddit_tifu`](https://huggingface.co/datasets/reddit_tifu) dataset.
The data was split into 90% train and 10% validation using `dataset.train_test_split`, with a seed of 0.
To prepare the data for training, the `"tldr"` and `"documents"` fields were joined by `"\n\n"`.
When multiple items were in the `"tldr"` or `"documents"` fields, only the first item was selected for joining.
These joined documents were tokenized using the `"distilgpt2"` tokenizer.
Finally, tokenized texts were concatenated end-to-end and split into blocks of 128 tokens.
**TODO:** Add a different separation token between documents that can be used to stop generation.
## Training
This model was trained in Colab by fine-tuning [`distilgpt2`](https://huggingface.co/distilgpt2) for 174390 steps (3 epochs).
Default training arguments were used, except for `learning_rate=2e-5` and `weight_decay=0.01`.
At the conclusion of training, a training loss of 3.52 and a validation loss of 3.44 were observed. |