task_categories:image-to-text task_categories:question-answering task_categories:zero-shot-classification task_ids:text-scoring annotations_creators:machine-generated multilinguality:monolingual size_categories:10K<n<100K language:en license:apache-2.0 arxiv:1405.0312 arxiv:2302.12189 region:us
High-Level Narratives dataset.
@inproceedings{Cafagna2023HLDG,
  title={HL Dataset: Grounding High-Level Linguistic Concepts in Vision},
  author={Michele Cafagna and Kees van Deemter and Albert Gatt},
  year={2023}
}
---
license: apache-2.0
task_categories:
- image-to-text
- question-answering
- zero-shot-classification
language:
- en
multilinguality:
  - monolingual
task_ids:
  - text-scoring
pretty_name: HL-Nattatives (High-Level Narratives Dataset)
size_categories:
- 10K<n<100K
annotations_creators:
- machine-generated
dataset_info:
  splits:
    - name: train
      num_examples: 13498
    - name: test
      num_examples: 1499
---
# Dataset Card for the High-Level Narratives Dataset

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Supported Tasks](#supported-tasks)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

The High-Level Narratives (HL-Narratives) dataset aligns **object-centric descriptions** from [COCO](https://arxiv.org/pdf/1405.0312.pdf) 
with synthetic **high-level narratives captions** automatically generated by merging **_scene_, _action_, _rationale_** captions from the [HL Dataset](https://huggingface.co/datasets/michelecafagna26/hl) using [T5](https://huggingface.co/Vamsi/T5_Paraphrase_Paws)

The HL-Naratives dataset contains 14997 images from COCO and a total of 134973 synthetic captions (3 captions per image) aligned with ~749984 object-centric captions from COCO.

**The high-level descriptions capture the human interpretations of the images**. These interpretations contain abstract concepts not directly linked to physical objects.
Each high-level description is provided with a _confidence score_, crowdsourced by an independent worker measuring the extent to which
the high-level description is likely given the corresponding image, question, and caption. The higher the score, the more the high-level caption is close to the commonsense (in a Likert scale from 1-5).

- **🗃️ Repository:** [github.com/michelecafagna26/HL-dataset](https://github.com/michelecafagna26/HL-dataset)
- **📜 Paper:** [HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales](https://arxiv.org/abs/2302.12189?context=cs.CL)

[//]: # (- **🧭 Spaces:** [Dataset explorer]&#40;https://huggingface.co/spaces/michelecafagna26/High-Level-Dataset-explorer&#41;)
- **🖊️ Contact:** michele.cafagna@um.edu.mt

### Supported Tasks

- image captioning
- multimodal text-scoring
- zero-shot evaluation

### Languages

English

## Dataset Structure

The dataset is provided with images from COCO and two metadata jsonl files containing the annotations

### Data Instances

An instance looks like this:
```json
{
  "file_name": "COCO_train2014_000000000036.jpg",
  "captions": ["In a beach, holding an umbrella means they won't get a sunburn.", 
    "The lady is posing with the sun umbrella, which was taken on the beach and is enjoying and getting pictures of her vacation.", 
    "She is holding a parasol that is taken by a lake she is vacationing and is sunny."] 
}
```

### Data Fields

- ```file_name```: original COCO filename
- ```captions```: List[str] containing 3 narrative captions for the image.

### Data Splits

There are 14997 images and 134973 high-level captions split into:
- Train-val: 13498 images and 121482 high-level captions
- Test: 1499 images and 13491 high-level captions

## Dataset Creation

The dataset has been automatically generated using T5 to merge the HL captions axis-wise.
From the paper:

> We frame the synthesis of narrative captions as a paraphrasing task. We follow a human-in-the-loop approach consisting of three stages: 
> (i) we manually annotate a small sample of gold data; 
> (ii) we fine-tune a large pre-trained language model (LPLM); 
> (iii) we use the fine-tuned model to generate a sample of data, which is manually corrected and then 
> (iv) added to the gold annotations before fine-tuning again.

### Curation Rationale

From the paper:

>We now describe how we extend the dataset to combine the three axes to compose a short `narrative', which describes the scene, action and rationale in tandem. 
> To do this, we leverage the individual axes and synthesise this part of the data using a pre-trained language model.
> Since scenes, actions, and rationales were elicited individually in a visually grounded and controlled setting, 
>a synthesised version of the three individual captions should also be true of the image to the same extent (modulo the variations in confidence that we observe).

### Source Data

- Images:  COCO
- captions annotations: automatically generated

#### Annotation process

From the paper:

> We use a version of T5 already fine-tuned on paraphrase generation as LPLM data generator. 
> We initialise the process with manually paraphrased annotations for 50 images ($3 \times 50 = 150$), fine-tune the model for 2 epochs, 
> and generate 150 captions for another 50 images, which are manually corrected and added to the original 150. 
> The model is then fine-tuned for a further two epochs. In each iteration, we reserve $10\%$ as validation data. 
> After two epochs, we observe that the validation loss does not improve further.
> Finally, in the last iteration, we use all gold data to fine-tune the model and generate synthetic high-level captions for the whole HL dataset, 
> obtaining 14,997 synthetic captions for training and 1499 for testing. In addition to the T5 paraphrase model, 
> we also experimented with LLaMA  in a few-shot setting; however, we find that T5 outperforms LLAMA in this task.

### Personal and Sensitive Information

There is no personal or sensitive information

## Considerations for Using the Data

[More Information Needed]

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

### Dataset Curators

Michele Cafagna

### Licensing Information

The Images follow the [COCO terms of Use](https://cocodataset.org/#termsofuse)
The remaining annotations are licensed under Apache-2.0 license.

### Citation Information

```BibTeX
@inproceedings{cafagna2023hl,
  title={{HL} {D}ataset: {V}isually-grounded {D}escription of {S}cenes, {A}ctions and
{R}ationales},
  author={Cafagna, Michele and van Deemter, Kees and Gatt, Albert},
  booktitle={Proceedings of the 16th International Natural Language Generation Conference (INLG'23)},
address = {Prague, Czech Republic},
  year={2023}
}
```
