Edit model card

flan-t5-large-spelling-peft

This model is an experimental peft adapter for google/flan-t5-large trained on the wiki.en dataset from oliverguhr/spelling.

It achieves the following results on the evaluation set:

  • Loss: 0.2537
  • Rouge1: 95.8905
  • Rouge2: 91.9178
  • Rougel: 95.8459
  • Rougelsum: 95.8393
  • Gen Len: 33.61

Model description

This an experimental model that should be capable of fixing typos and punctuation.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, pipeline

model_id = "google/flan-t5-large"
peft_model_id = "jbochi/flan-t5-large-spelling-peft"

model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
model.load_adapter(peft_model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)

pipe = pipeline("text2text-generation", model=model, tokenizer=tokenizer)
pipe("Fix spelling: This restuarant is awesome")
# [{'generated_text': 'This restaurant is awesome'}]

Intended uses & limitations

Intented for research purposes.

  • It may produce artifacts.
  • Doesn't seen capable of fixing multiple errors in a single sentence.
  • It doesn't support languages other than English.
  • It was fine-tuned with a max_length of 100 tokens.

Training and evaluation data

Data from oliverguhr/spelling, with a "Fix spelling: " prefix added to every example.

The model was only evaluated on the first 100 test examples only during training.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.001
  • train_batch_size: 64
  • eval_batch_size: 64
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Rouge1 Rouge2 Rougel Rougelsum Gen Len
0.3359 0.05 500 0.2738 95.8385 91.6723 95.7821 95.766 33.5
0.2853 0.11 1000 0.2702 95.7124 91.5043 95.656 95.651 33.53
0.2691 0.16 1500 0.2691 95.735 91.7108 95.7039 95.7067 33.41
0.2596 0.21 2000 0.2663 95.9819 92.0897 95.9519 95.9488 33.51
0.2536 0.27 2500 0.2621 95.7519 91.5445 95.6614 95.6622 33.49
0.2472 0.32 3000 0.2626 95.7052 91.7321 95.6476 95.6512 33.58
0.2448 0.37 3500 0.2669 95.8003 91.7949 95.7536 95.7576 33.57
0.2345 0.43 4000 0.2582 95.8784 92.008 95.8284 95.8343 33.65
0.2345 0.48 4500 0.2629 95.8131 91.9088 95.7624 95.766 33.63
0.2284 0.53 5000 0.2585 95.8552 91.9833 95.8105 95.8135 33.62
0.2266 0.59 5500 0.2591 95.9205 92.0577 95.8689 95.8718 33.61
0.2281 0.64 6000 0.2605 95.9172 91.9782 95.874 95.8638 33.59
0.2228 0.69 6500 0.2566 95.7612 91.7858 95.7129 95.7058 33.63
0.2202 0.75 7000 0.2561 95.9468 92.0914 95.9018 95.8941 33.64
0.218 0.8 7500 0.2579 95.9468 92.0914 95.9018 95.8941 33.64
0.2162 0.85 8000 0.2523 95.8231 91.9464 95.7727 95.7758 33.66
0.2135 0.91 8500 0.2549 95.8388 91.9804 95.7914 95.7917 33.63
0.2124 0.96 9000 0.2537 95.8905 91.9178 95.8459 95.8393 33.61

Framework versions

  • Transformers 4.35.2
  • Pytorch 2.1.0+cu121
  • Datasets 2.16.0
  • Tokenizers 0.15.0
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference API
Unable to determine this model's library. Check the docs .

Model tree for jbochi/flan-t5-large-spelling-peft

Finetuned
(105)
this model