README.md

metadata

tags:
  - generated_from_trainer
  - distilbart
model-index:
  - name: distilbart-finetuned-summarization
    results: []
license: apache-2.0
datasets:
  - cnn_dailymail
  - xsum
  - samsum
  - ccdv/pubmed-summarization
language:
  - en
metrics:
  - rouge

distilgpt2-finetuned-finance

This model is a further fine-tuned version of distilbart-cnn-12-6 on the the combination of 4 different summarisation datasets:

• cnn_dailymail
• samsum
• xsum
• ccdv/pubmed-summarization

Please check out the offical model page and paper:

• sshleifer/distilbart-cnn-12-6
• Pre-trained Summarization Distillation

Training and evaluation data

One can reproduce the dataset using the following code:

from datasets import DatasetDict, load_dataset
from datasets import concatenate_datasets

xsum_dataset = load_dataset("xsum")
pubmed_dataset = load_dataset("ccdv/pubmed-summarization").rename_column("article", "document").rename_column("abstract", "summary")
cnn_dataset = load_dataset("cnn_dailymail", '3.0.0').rename_column("article", "document").rename_column("highlights", "summary")
samsum_dataset = load_dataset("samsum").rename_column("dialogue", "document")

summary_train = concatenate_datasets([xsum_dataset["train"], pubmed_dataset["train"], cnn_dataset["train"], samsum_dataset["train"]])
summary_validation = concatenate_datasets([xsum_dataset["validation"], pubmed_dataset["validation"], cnn_dataset["validation"], samsum_dataset["validation"]])
summary_test = concatenate_datasets([xsum_dataset["test"], pubmed_dataset["test"], cnn_dataset["test"], samsum_dataset["test"]])

raw_datasets = DatasetDict()
raw_datasets["train"] = summary_train
raw_datasets["validation"] = summary_validation
raw_datasets["test"] = summary_test

Inference example

from transformers import pipeline

pipe = pipeline("text2text-generation", model="lxyuan/distilbart-finetuned-summarization")

text = """The tower is 324 metres (1,063 ft) tall, about the same height as
an 81-storey building, and the tallest structure in Paris. Its base is square,
measuring 125 metres (410 ft) on each side. During its construction, the
Eiffel Tower surpassed the Washington Monument to become the tallest man-made
structure in the world, a title it held for 41 years until the Chrysler Building
in New York City was finished in 1930. It was the first structure to reach a
height of 300 metres. Due to the addition of a broadcasting aerial at the top
of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres
(17 ft). Excluding transmitters, the Eiffel Tower is the second tallest
free-standing structure in France after the Millau Viaduct.
"""

pipe(text)

>>>"""The Eiffel Tower is the tallest man-made structure in the world .
The tower is 324 metres tall, about the same height as an 81-storey building .
Due to the addition of a broadcasting aerial in 1957, it is now taller than
the Chrysler Building by 5.2 metres .
"""

Training procedure

Notebook link: here

Training hyperparameters

The following hyperparameters were used during training:

• evaluation_strategy="epoch",
• save_strategy="epoch",
• logging_strategy="epoch",
• learning_rate=2e-5,
• per_device_train_batch_size=2,
• per_device_eval_batch_size=2,
• gradient_accumulation_steps=64,
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• weight_decay=0.01,
• save_total_limit=2,
• num_train_epochs=10,
• predict_with_generate=True,
• fp16=True,
• push_to_hub=True

Training results

Training is still in progress

Epoch	Training Loss	Validation Loss	Rouge1	Rouge2	RougeL	RougeLsum	Gen Len
0	1.779700	1.719054	40.0039	17.9071	27.8825	34.8886	88.8936

Framework versions

• Transformers 4.30.2
• Pytorch 2.0.1+cu117
• Datasets 2.13.1
• Tokenizers 0.13.3