|
--- |
|
language: |
|
- "en" |
|
thumbnail: "https://styles.redditmedia.com/t5_2to41/styles/communityIcon_qedoavxzocr61.png?width=256&s=9c7c19b81474c3788279b8d6d6823e791d0524fc" |
|
datasets: |
|
- "reddit_tifu (subset: short)" |
|
widget: |
|
- text: "I told my friend" |
|
license: mit |
|
--- |
|
|
|
# mgfrantz/distilgpt2-finetuned-reddit-tifu |
|
|
|
This model was trained to as practice for fine-tuning a causal language model. |
|
There was no intended use case for this model besides having some fun seeing how different things might be screwed up. |
|
|
|
## Data |
|
|
|
This model was trained on "short" subset of [`reddit_tifu`](https://huggingface.co/datasets/reddit_tifu) dataset. |
|
The data was split into 90% train and 10% validation using `dataset.train_test_split`, with a seed of 0. |
|
|
|
To prepare the data for training, the `"tldr"` and `"documents"` fields were joined by `"\n\n"`. |
|
When multiple items were in the `"tldr"` or `"documents"` fields, only the first item was selected for joining. |
|
These joined documents were tokenized using the `"distilgpt2"` tokenizer. |
|
|
|
Finally, tokenized texts were concatenated end-to-end and split into blocks of 128 tokens. |
|
|
|
**TODO:** Add a different separation token between documents that can be used to stop generation. |
|
|
|
## Training |
|
|
|
This model was trained in Colab by fine-tuning [`distilgpt2`](https://huggingface.co/distilgpt2) for 174390 steps (3 epochs). |
|
Default training arguments were used, except for `learning_rate=2e-5` and `weight_decay=0.01`. |
|
At the conclusion of training, a training loss of 3.52 and a validation loss of 3.44 were observed. |
