license:cc-by-4.0 region:us
---
license: cc-by-4.0
---

# Dataset Card for VITATECS

## Dataset Description

### Dataset Summary

VITATECS is a diagnostic VIdeo-Text dAtaset for the evaluation of TEmporal Concept underStanding. 

### Languages

English. 

## Dataset Structure

### Data Instances

This repo contains 6 jsonl files, each of which corresponds to an aspect of temporal concepts (Direction, Intensity, Sequence, Localization, Compositionality, Type). 
Example (indented for better presentation):
```
{
    "src_dataset": "VATEX", 
    "video_name": "i0ccSYMl0vo_000027_000037.mp4", 
    "caption": "A woman is placing a waxing strip on a man's leg.", 
    "counterfactual": "A woman is removing a waxing strip from a man's leg.",
    "aspect": "Direction"
}
```

### Data Fields

- src_dataset: the name of the source dataset (VATEX or MSRVTT)
- video_name: the name of the video in the source dataset
- caption: the original caption of the video
- counterfactual: the generated counterfactual description of the video

### Dataset Statistics

|                           | Direction | Intensity | Sequence | Localization | Compositionality | Type  |
| ------------------------- | --------- | --------- | -------- | ------------ | ---------------- | ----- |
| # samples                 | 2,709     | 745       | 380      | 1,788        | 2,393            | 8,109 |
| # videos                  | 2,016     | 650       | 348      | 1,453        | 1,739            | 4,856 |
| Avg. len (caption)        | 13.02     | 13.04     | 15.58    | 14.37        | 13.29            | 11.34 |
| Avg. len (counterfactual) | 13.12     | 13.05     | 15.74    | 14.43        | 13.53            | 11.35 | 

## Dataset Creation

### Source Data

VITATECS is based on video-text pairs from [MSR-VTT](https://www.microsoft.com/en-us/research/publication/msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/)

### Annotations

#### Annotation process

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

Part of this dataset is generated by large language models and may contain toxic or biased texts. 
We mitigate this issue by leveraging [Perspective API](https://developers.perspectiveapi.com/) to filter out highly toxic generations. 

## Additional Information

### Dataset Curators

VITATECS is curated by Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun (Peking University) and Lu Hou (Huawei Noah's Ark Lab).

### Licensing Information

This dataset is under [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) license.

### Citation Information

[More Information Needed]

### Contributions

[More Information Needed]