FLAN-T5-NLP-Paper-to-Question-Generation

This model is a fine-tuned version of google/flan-t5-large on an allenai/QASPER: a dataset for question answering on scientific research papers -based NLP-Paper-to-QA-Generation dataset.

Target Task

• NLP Paper's Abstract + Introduction --> {Question} [SEP] {Answer}
• Question-based Summarization
• Long Document Summarization
• Scientific Paper Summarization

(1) How to use: Inference on CPU ( Code Snippets )

• Inference can be slow on CPU

Load model directly

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("UNIST-Eunchan/FLAN-T5-NLP-Paper-to-Question-Generation")
model = AutoModelForSeq2SeqLM.from_pretrained("UNIST-Eunchan/FLAN-T5-NLP-Paper-to-Question-Generation")

Prompting Input

txt =  r""" 
Generate Question, Answer pair correspond to the following research paper. 
[Abstract] + {text['abstract']} + [Introduction] + {text['introduction']}
Question, Answer:
""".replace("\n", "")

inputs = tokenizer(txt, max_length = 1024, truncation=True, padding="max_length", return_tensors="pt")

For Multiple Question Generation (👍)

num_generate_sequence = 4 #8, 16, 2, 1
summaries = model.generate(input_ids =inputs["input_ids"], max_new_tokens=100, do_sample = True, top_p = 0.95, num_return_sequences = num_generate_sequence)

For Single Question Generation

summaries = model.generate(input_ids =inputs["input_ids"], max_new_tokens=100, do_sample = True, top_p = 0.95)

decoded_summaries = [tokenizer.decode(s, skip_special_tokens=False, clean_up_tokenization_spaces=True) for s in summaries]
decoded_summaries = [d.replace("<n>", " ").replace(tokenizer.pad_token, "").replace(tokenizer.eos_token, "") for d in decoded_summaries]

(2) Faster Inference on GPU

• about 60x faster than (1) [CPU --> COLAB T4 GPU]

Additional Installation

!pip install accelerate -q
!pip install bitsandbytes -q
!pip install optimum -q

Load model directly

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM,BitsAndBytesConfig
from optimum.bettertransformer import BetterTransformer

# load model in 4-bit
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

tokenizer = AutoTokenizer.from_pretrained("UNIST-Eunchan/FLAN-T5-NLP-Paper-to-Question-Generation")
model = AutoModelForSeq2SeqLM.from_pretrained("UNIST-Eunchan/FLAN-T5-NLP-Paper-to-Question-Generation", quantization_config=quantization_config)
model = BetterTransformer.transform(model)

For Multiple Question Generation (👍)

# use to(device)

num_generate_sequence = 16 # (about 20 sec with Colab T4 GPU)
summaries = model.generate(input_ids =inputs["input_ids"].to(device), max_new_tokens=100, do_sample = True, top_p = 0.95, num_return_sequences = num_generate_sequence)

Training results

It achieves the following results on the evaluation set:

• Loss: 0.4504

Training Loss	Epoch	Step	Validation Loss
No log	0.99	46	34.6109
29.7732	1.99	92	16.5236
29.7732	2.98	138	4.6887
7.9911	3.97	184	0.5679
7.9911	4.97	230	0.4795
0.6152	5.96	276	0.4577
0.6152	6.95	322	0.4523
0.4811	7.95	368	0.4509
0.4811	8.94	414	0.4505
0.4721	9.93	460	0.4504

Model description

• FLAN-T5-Large (783M)

Generated Output Example

• Our model generate 16 different Q-A Pair with top-p sampling.

input: r""" 
Generate Question, Answer pair correspond to the following research paper. 
[Abstract] In this work, we explore prompt tuning, a simple yet effective mechanism for learning soft prompts to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signal from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3's few-shot learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method closes the gap and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant in that large models are costly to share and serve, and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed prefix tuning of Li and Liang (2021), and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning. [Introduction] With the wide success of pre-trained large language models, a range of techniques has arisen to adapt these general-purpose models to downstream tasks. ELMo (Peters et al., 2018) proposed freezing the pre-trained model and learning a task-specific weighting of its per-layer representations. However, since GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), the dominant adaptation technique has been model tuning (or fine-tuning), where all model parameters are tuned during adaptation, as proposed by Howard and Ruder (2018).More recently, Brown et al. (2020) showed that prompt design (or priming) is surprisingly effective at modulating a frozen GPT-3 model’s behavior through text prompts. Prompts are typically composed of a task description and/or several canonical examples. This return to freezing pre-trained models is appealing, especially as model size continues to increase. Rather than requiring a separate copy of the model for each downstream task, a single generalist model can simultaneously serve many different tasks. Unfortunately, prompt-based adaptation has several key drawbacks. Task description is error-prone and requires human involvement, and the effectiveness of a prompt is limited by how much conditioning text can fit into the model’s input. As a result, downstream task quality still lags far behind that of tuned models. For instance, GPT-3 175B fewshot performance on SuperGLUE is 17.5 points below fine-tuned T5-XXL (Raffel et al., 2020) (71.8 vs. 89.3) despite using 16 times more parameters. Several efforts to automate prompt design have been recently proposed. Shin et al. (2020) propose a search algorithm over the discrete space of words, guided by the downstream application training data. While this technique outperforms manual prompt design, there is still a gap relative to model tuning. Li and Liang (2021) propose prefix tuning and show strong results on generative tasks. This method freezes the model parameters and backpropagates the error during tuning to prefix activations prepended to each layer in the encoder stack, including the input layer. Hambardzumyan et al. (2021) simplify this recipe by restricting the trainable parameters to the input and output subnetworks of a masked language model, and show reasonable results on classifications tasks. In this paper, we propose prompt tuning as a further simplification for adapting language models. We freeze the entire pre-trained model and only allow an additional k tunable tokens per downstream task to be prepended to the input text. This soft prompt is trained end-to-end and can condense the signal from a full labeled dataset, allowing our method to outperform few-shot prompts and close the quality gap with model tuning (Figure 1). At the same time, since a single pre-trained model is recycled for all downstream tasks, we retain the efficient serving benefits of frozen models (Figure 2). While we developed our method concurrently with Li and Liang (2021) and Hambardzumyan et al. (2021), we are the first to show that prompt tuning alone (with no intermediate-layer prefixes or task-specific output layers) is sufficient to be competitive with model tuning. Through detailed experiments in sections 2–3, we demonstrate that language model capacity is a key ingredient for these approaches to succeed. As Figure 1 shows, prompt tuning becomes more competitive with scale. We compare with similar approaches in Section 4. Explicitly separating task-specific parameters from the generalist parameters needed for general language-understanding has a range of additional benefits. We show in Section 5 that by capturing the task definition in the prompt while keeping the generalist parameters fixed, we are able to achieve better resilience to domain shifts. In Section 6, we show that prompt ensembling, learning multiple prompts for the same task, can boost quality and is more efficient than classic model ensembling. Finally, in Section 7, we investigate the interpretability of our learned soft prompts. In sum, our key contributions are: 1. Proposing prompt tuning and showing its competitiveness with model tuning in the regime of large language models. 2. Ablating many design choices, and showing quality and robustness improve with scale. 3. Showing prompt tuning outperforms model tuning on domain shift problems. 4. Proposing prompt ensembling and showing its effectiveness. 
Question, Answer:
""".replace("\n", "")

output= [' What was the size of each untrained model?[SEP] The size of the model can be a combination of the size of all the parameters in a model',
 ' What are the benefits of using soft prompts?[SEP] They reduce the need to use manual prompt design and conserve machine training data',
 ' What is the sample size of dataset?[SEP] 22840',
 ' How does the method outperform some of the pre-trained models?[SEP] They successfully tune their model for two tasks, one for a few shot and the other for several downstream tasks.',
 ' What is the sample size of the experiments?[SEP]135 for a simple task?[SEP]32 for a more complicated task',
 ' What is the baseline model they tested? [SEP] GPT-3 model, with four state-of-the-art examples in a masked language model',
 ' What task accuracy is given by prompts?[SEP]Mixed task efficiency was 93% and accuracy 85% compared to normal noise level',
 ' What metrics do they use?[SEP] EMO score, VSD, and SVM scores',
 ' What metrics are used to assess the performance of the soft prompt training?[SEP] quality of translation, accuracy of text-to-text, robustness of domain transfer, error rate.',
 ' How much do they experiment with the T5 baseline?[SEP] The baseline is used for simulated benchmarks.',
 ' Which task are they applying their method to?[SEP]They test their approach on classifications tasks',
 " Why do they show that their approach outperforms GPT-3's few-shot? [SEP] This is a large project that uses a multi-task approach to train GPT-3 models. In this paper, they demonstrate that the current method outperforms both the GPT-3 few-shot and the Li and Liang prefix tuning. They also show that the prefix tuning performed much better than the model tuning. What is the difference between their experiments",
 ' How do they compare with other techniques? [SEP] They provide a comparison for each approach.',
 ' Which task is the GPT-3 model most applicable to?[SEP]Classification tasks. For which tasks does the model need a subnetwork?[SEP]Classification tasks for GPT-3',
 ' What is the baseline test case used for this experiment?[SEP]Pompets for a variety of tasks are trained using the same method. This is the baseline, and the baseline is used for all applications.',
 ' What was the size of their model?[SEP] They experimented with 0.5 m.m and 0.5 m.m respectively.']

Inference Examples

If Inference API generate bad, you can use model.generate() in your code for better output!

• (1) Attention is All You Need
  • (https://arxiv.org/abs/1706.03762)
• (2) The Power of Scale for Parameter-Efficient Prompt Tuning
  • (https://arxiv.org/abs/2104.08691)
• (3)(LK-99 Paper/ Not an NLP paper) The First Room-Temperature Ambient-Pressure Superconductor
  • (https://arxiv.org/abs/2307.12008)
• (4) Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
  • (https://arxiv.org/abs/2202.06935)
• (5) Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
  • (https://arxiv.org/abs/2311.03099)

Training and evaluation data

• Used Dataset: UNIST-Eunchan/NLP-Paper-to-QA-Generation dataset.
• Train: dataset['train'] + dataset['test']
• Evaluation: dataset['validation']

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 184
• num_epochs: 10