task_categories:zero-shot-classification size_categories:1M<n<10M license:cc-by-4.0 video audio text image tetramodal multimodal youtube wikipedia region:us
---
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: val
    path: data/val-*
  - split: test
    path: data/test-*
dataset_info:
  features:
  - name: image
    dtype: image
  - name: image_url
    dtype: string
  - name: item_idx
    dtype: int64
  - name: wit_features
    struct:
    - name: attribution_passes_lang_id
      sequence: bool
    - name: caption_alt_text_description
      sequence: string
    - name: caption_reference_description
      sequence: string
    - name: caption_title_and_reference_description
      sequence: string
    - name: context_page_description
      sequence: string
    - name: context_section_description
      sequence: string
    - name: hierarchical_section_title
      sequence: string
    - name: is_main_image
      sequence: bool
    - name: language
      sequence: string
    - name: page_changed_recently
      sequence: bool
    - name: page_title
      sequence: string
    - name: page_url
      sequence: string
    - name: section_title
      sequence: string
  - name: wit_idx
    dtype: int64
  - name: youtube_title_text
    dtype: string
  - name: youtube_description_text
    dtype: string
  - name: youtube_video_content
    dtype: binary
  - name: youtube_video_starting_time
    dtype: string
  - name: youtube_subtitle_text
    dtype: string
  - name: youtube_video_size
    dtype: int64
  - name: youtube_video_file_path
    dtype: string
  splits:
  - name: train
    num_bytes: 1902638101655.625
    num_examples: 1052915
  - name: val
    num_bytes: 104485442867.25
    num_examples: 57958
  - name: test
    num_bytes: 111107332347.375
    num_examples: 61389
  download_size: 2058391040534
  dataset_size: 2118230876870.25
license: cc-by-4.0
task_categories:
- zero-shot-classification
tags:
- video
- audio
- text
- image
- tetramodal
- multimodal
- youtube
- wikipedia
pretty_name: TALI
size_categories:
- 1M<n<10M
---
# Dataset Card for "TALI"

## Table of Contents
1. Dataset Description
  1. Abstract
  2. Brief Description
2. Dataset Information
  1. Modalities
  2. Dataset Variants
  3. Dataset Statistics
  4. Data Fields
  5. Data Splits
3. Dataset Creation
4. Dataset Use
5. Additional Information

## Dataset Description

### Abstract
TALI is a large-scale, tetramodal dataset designed to facilitate a shift from unimodal and duomodal to tetramodal research in deep learning. It aligns text, video, images, and audio, providing a rich resource for innovative self-supervised learning tasks and multimodal research. TALI enables exploration of how different modalities and data/model scaling affect downstream performance, with the aim of inspiring diverse research ideas and enhancing understanding of model capabilities and robustness in deep learning.

### Brief Description
TALI (Temporally and semantically Aligned Audio, Language and Images) is a dataset that uses the Wikipedia Image Text (WIT) captions and article titles to search Youtube for videos that match the captions. It then downloads the video, audio, and subtitles from these videos. The result is a rich multimodal dataset that has multiple caption types related to both the WiT Images, and the Youtube videos. This enables learning to take place between either temporally or semantically aligned text, images, audio and video.

## Dataset Information
### Modalities
The TALI dataset consists of the following modalities:

1. Image:
  1. Wikipedia caption image
  2. Randomly sampled image from youtube video
2. Text
  1. Wikipedia Caption Text
  2. Wikipedia Title Text
  3. Wikipedia Main Body Text
  4. YouTube Subtitle Text
  5. YouTube Description Text
  6. YouTube Title Text
3. Audio
  1. YouTube Content Audio
4. Video
  1. YouTube Content Video
     
### Dataset Variants
The TALI dataset comes in three variants that differ in the training set size:

- TALI-small: Contains about 1.3 million 30-second video clips, aligned with 120K WiT entries.
- TALI-base: Contains about 6.5 million 30-second video clips, aligned with 120K WiT entries.
- TALI-big: Contains about 13 million 30-second video clips, aligned with 120K WiT entries.

The validation and test sets remain consistent across all three variants at about 80K Videos aligned to 8K wikipedia entries (10 subclips for each Wikipedia entry) each.


### Dataset Statistics
TBA


## Dataset Creation
The TALI dataset was created by starting from the WiT dataset and using either the context_page_description or page_title as a source-query to search YouTube for video that were creative commons opted-in, and, not age restricted. The top 100 result titles were returned and compared with the source-query using the CLIP text embeddings of the largest CLIP model available. The top-1 title’s video based on the CLIP ranking was chosen and downloaded. The video was broken into 30-second segments and the top-10 segments for eachvideo were chosen based on the distance between the CLIP image embedding of the first image of each segment and the video’s title text. The image, audio, and subtitle frames were extracted from these segments. At sampling time, one of these 10 segments is randomly selected, and a 10-second segment is chosen out of the 30-second clip. The result is 200 video frames (spread throughout the 10-second segment), and 160000 audio frames (10 seconds).

## Dataset Use
TALI is designed for use in a wide range of multimodal research tasks, including but not limited to:

- Multimodal understanding and reasoning
- Self-supervised learning
- Multimodal alignment and translation
- Multimodal summarization
- Multimodal question answering

## Dataset Curators: Antreas Antoniou
Citation Information: TBA
Contributions: Thanks to all contributors including data curators, annotators, and software developers.

[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)