System2_FigLang2022 / README.md
librarian-bot's picture
Librarian Bot: Update dataset YAML metadata for model
858d266
|
raw
history blame
4.71 kB
metadata
language: en
license: cc-by-4.0
library_name: transformers
datasets: ColumbiaNLP/FLUTE

Model description

This is the T5-3B model for System 2 as described in our paper Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE, FigLang workshop @ EMNLP 2022 (Arxiv link: https://arxiv.org/abs/2210.16407)

System 2: Jointly predicting the type of figurative language

Using type of figurative language provided as part of the training set (Chakrabarty et al., 2022), one of our models jointly predicts the type of figurative language, together with the target label and explanation:

Input <Premise> <Hypothesis> 
Output <Figurative-Language-Type> <Label> <Explanation>

How to use this model?

We provide a quick example of how you can try out System 2 in our paper with just a few lines of code:

>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> model = AutoModelForSeq2SeqLM.from_pretrained("allenai/System2_FigLang2022")

>>> tokenizer = AutoTokenizer.from_pretrained("t5-3b")
>>> input_string = "Premise: Yesterday two gangs were fighting just in front of my home. Hypothesis: Yesterday I saw two gangs fighting right in front of my house and it totally didn't make me scared at all. What is the type of figurative language involved? Is there a contradiction or entailment between the premise and hypothesis?"
>>> input_ids = tokenizer.encode(input_string, return_tensors="pt")
>>> output = model.generate(input_ids, max_length=200)
>>> tokenizer.batch_decode(output, skip_special_tokens=True)
['Answer : [Type] Sarcasm [Label] Contradiction. Explanation : Seeing two gangs of people fighting in public can be really dangerous and scary, so someone who claims that they were not scared at all is being sarcastic.']

More details about DREAM-FLUTE ...

For more details about DREAM-FLUTE, please refer to our:

This model is part of our DREAM-series of works. This is a line of research where we make use of scene elaboration for building a "mental model" of situation given in text. Check out our GitHub Repo for more!

More details about this model ...

Training and evaluation data

We use the FLUTE dataset for the FigLang2022SharedTask (https://huggingface.co/datasets/ColumbiaNLP/FLUTE) for training this model. ∼7500 samples are provided as the training set. We used a 80-20 split to create our own training (6027 samples) and validation (1507 samples) partitions on which we build our models. For details on how we make use of the training data provided in the FigLang2022 shared task, please refer to https://github.com/allenai/dream/blob/main/FigLang2022SharedTask/Process_Data_Train_Dev_split.ipynb.

Model details

This model is a fine-tuned version of t5-3b.

It achieves the following results on the evaluation set:

  • Loss: 0.6078
  • Rouge1: 62.8674
  • Rouge2: 45.0585
  • Rougel: 57.5618
  • Rougelsum: 57.5172
  • Gen Len: 50.7558

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 2
  • total_train_batch_size: 2
  • total_eval_batch_size: 2
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 3.0

Training results

Training Loss Epoch Step Validation Loss Rouge1 Rouge2 Rougel Rougelsum Gen Len
0.8068 0.33 1000 0.7251 30.6353 25.0792 30.619 30.6274 19.0
0.7276 0.66 2000 0.6715 30.8651 26.1492 30.8543 30.8519 19.0
0.7063 1.0 3000 0.6338 31.0263 26.6749 31.0094 31.0098 19.0
0.4516 1.33 4000 0.6447 30.9942 26.5984 30.9834 30.9778 19.0
0.4538 1.66 5000 0.6183 31.0179 26.7012 31.005 31.0018 19.0
0.4373 1.99 6000 0.6078 31.0085 26.7116 30.9952 30.9894 19.0
0.2743 2.32 7000 0.6910 31.0051 26.7349 30.9975 30.9851 19.0
0.2819 2.65 8000 0.6831 31.0876 26.848 31.0766 31.0753 19.0
0.2849 2.99 9000 0.6673 30.9223 26.5899 30.9165 30.9073 19.0

Framework versions

  • Transformers 4.22.0.dev0
  • Pytorch 1.12.1+cu113
  • Datasets 2.4.0
  • Tokenizers 0.12.1