task_categories:text-to-image size_categories:1K<n<10K language:en license:cc-by-4.0 space region:us
---
dataset_info:
  features:
  - name: image
    dtype: image
  - name: text
    dtype: string
  - name: id
    dtype: string
  - name: title
    dtype: string
  - name: description
    dtype: string
  - name: credits
    dtype: string
  - name: url
    dtype: string
  - name: Id
    dtype: string
  - name: Type
    dtype: string
  - name: Release date
    dtype: string
  - name: Related releases
    dtype: string
  - name: Size
    dtype: string
  - name: Name
    dtype: string
  - name: Distance
    dtype: string
  - name: Constellation
    dtype: string
  - name: Category
    dtype: string
  - name: Position (RA)
    dtype: string
  - name: Position (Dec)
    dtype: string
  - name: Field of view
    dtype: string
  - name: Orientation
    dtype: string
  - name: width
    dtype: int64
  - name: height
    dtype: int64
  - name: file_size
    dtype: int64
  - name: crop_w
    dtype: int64
  - name: crop_h
    dtype: int64
  - name: cropped
    dtype: bool
  - name: Related science announcements
    dtype: string
  - name: Related announcements
    dtype: string
  splits:
  - name: train
    num_bytes: 94474695584.124
    num_examples: 2706
  download_size: 61236366105
  dataset_size: 94474695584.124
license: cc-by-4.0
task_categories:
- text-to-image
language:
- en
tags:
- space
pretty_name: ESA Hubble Deep Space Images & Captions
size_categories:
- 1K<n<10K
---
# Dataset Card for ESA Hubble Deep Space Images & Captions

## Table of Contents

- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Examples](#examples)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [ESA Hubble](https://esahubble.org/)
- **Repository:** [Hubble Diffusion repository](https://github.com/Supermaxman/hubble-diffusion)
- **Point of Contact:** [Maxwell Weinzierl](mailto:maxwell.weinzierl@utdallas.edu)

### Dataset Summary

The ESA Hubble Deep Space Images & Captions dataset is composed primarily of Hubble deep space scans as high-resolution images,
along with textual descriptions written by ESA/Hubble. Metadata is also included, which enables more detailed filtering and understanding of massive space scans.
The purpose of this dataset is to enable text-to-image generation methods for generating high-quality deep space scans from prompts.
Check out [Hubble Diffusion v2](https://huggingface.co/Supermaxman/hubble-diffusion-2) for an example of a model trained on this dataset!

### Examples

#### A grazing encounter between two spiral galaxies

> In the direction of the constellation Canis Major, two spiral galaxies pass by each other like majestic ships in the night. The near-collision has been caught in images taken by the NASA/ESA Hubble Space Telescope and its Wide Field Planetary Camera 2.
>![opo9941a](https://cdn.spacetelescope.org/archives/images/thumb700x/opo9941a.jpg)
>
> Credit: NASA/ESA and The Hubble Heritage Team (STScI)

#### The magnificent starburst galaxy Messier 82

> This mosaic image of the magnificent starburst galaxy, Messier 82 (M82) is the sharpest wide-angle view ever obtained of M82. It is a galaxy remarkable for its webs of shredded clouds and flame-like plumes of glowing hydrogen blasting out from its central regions where young stars are being born 10 times faster than they are inside in our Milky Way Galaxy.
>![heic0604a](https://cdn.spacetelescope.org/archives/images/screen/heic0604a.jpg)
>
> Credit: NASA, ESA and the Hubble Heritage Team (STScI/AURA). Acknowledgment: J. Gallagher (University of Wisconsin), M. Mountain (STScI) and P. Puxley (NSF).

#### Extreme star cluster bursts into life in new Hubble image

> The star-forming region NGC 3603 - seen here in the latest Hubble Space Telescope image - contains one of the most impressive massive young star clusters in the Milky Way. Bathed in gas and dust the cluster formed in a huge rush of star formation thought to have occurred around a million years ago. The hot blue stars at the core are responsible for carving out a huge cavity in the gas seen to the right of the star cluster in NGC 3603's centre.
>![heic0715a](https://cdn.spacetelescope.org/archives/images/screen/heic0715a.jpg)
>
> Credit: NASA, ESA and the Hubble Heritage (STScI/AURA)-ESA/Hubble Collaboration

#### Statistics

- There are a total of 2,706 deep space images
- The complete uncompressed size of the dataset is 120 GB, so definitely make use of [Streaming](https://huggingface.co/docs/datasets/stream)
- The average image is 44 MB, while the max image size is 432 MB
- The average image has a height of 2,881 pixels, and an average width of 3,267 pixels

### Supported Tasks and Leaderboards

- `text-to-image`: The dataset can be used to train a model for conditional image generation from text. A conditional text-to-image generation model is presented with a text prompt, and is asked to generate an image which aligns with that text prompt. Model performance is typically measured by human judgement, as it is difficult to automatically measure the quality of generated images and how closely they match the text prompt. An example of a text-to-image model is [Stable Diffusion v2-1](https://huggingface.co/stabilityai/stable-diffusion-2-1). An example of a text-to-image model trained on this dataset is [Hubble Diffusion v2](https://huggingface.co/Supermaxman/hubble-diffusion-2).

### Languages

The text describing the images in the dataset is in English, as written by the writers from ESA/Hubble at [https://esahubble.org/](https://esahubble.org/). The associated BCP-47 code is `en`.

## Dataset Structure

### Data Instances

A typical data point comprises a high-quality deep space scan as an image, along with a textual description of that image produced by ESA/Hubble.
The textual description was derived by combining the `title` and the `description` of the image from the ESA/Hubble website.
Additionally, each data point also contains significant metadata about the image, such as the type of image, credits, the URL, the release date, and more.

An example looks as follows:

```json
{
   "image": "<encoded image>",
   "text":"A grazing encounter between two spiral galaxies: In the direction of the constellation Canis Major, two spiral galaxies pass by each other like majestic ships in the night. The near-collision has been caught in images taken by the NASA/ESA Hubble Space Telescope and its Wide Field Planetary Camera 2.",
   "id":"opo9941a",
   "title":"A grazing encounter between two spiral galaxies",
   "description":"In the direction of the constellation Canis Major, two spiral galaxies pass by each other like majestic ships in the night. The near-collision has been caught in images taken by the NASA/ESA Hubble Space Telescope and its Wide Field Planetary Camera 2.",
   "credits":"NASA/ESA and The Hubble Heritage Team (STScI)",
   "url":"https://esahubble.org/images/opo9941a/",
   "Id":"opo9941a",
   "Type":"Local Universe : Galaxy : Type : Interacting",
   "Release date":"4 November 1999, 07:00",
   "Size":"2907 x 1486 px",
   "Name":"IC 2163, NGC 2207",
   "Distance":"110 million light years",
   "Constellation":"Canis Major",
   "Category":"Galaxies",
   "Position (RA)":"6 16 25.10",
   "Position (Dec)":"-21&deg 22' 34.62\"",
   "Field of view":"4.82 x 2.47 arcminutes",
   "Orientation":"North is 191.2\u00b0 right of vertical",
   "width":2907,
   "height":1486,
   "file_size":12959406,
   "crop_w":0,
   "crop_h":0,
   "cropped":false
}
```

### Data Fields

- `image`: encoded RGB `.png` image of the deep space scan
- `text`: text description of image, a combination of `title` + ': ' + `description`
- `id`: id of the image from ESA/Hubble
- `title`: textual title of image from ESA/Hubble URL
- `description`: textual description of image from ESA/Hubble URL
- `credits`: required credits for each image from ESA/Hubble URL
- `url`: ESA/Hubble URL
- `Id`: id of the image from ESA/Hubble (from website metadata)
- `Type`: type of deep space scan
- `Release date`: release date of deep space scan
- `Size`: size of original image
- `Name`: name of celestial entities present in image
- `Distance`: distance from celestial entities present in image
- `Constellation`: constellation of celestial entities present in image
- `Category`: category of celestial entities present in image
- `Position (RA)`: coordinates for deep space scan used by Hubble telescope
- `Position (Dec)`: coordinates for deep space scan used by Hubble telescope
- `Field of view`: coordinates for deep space scan used by Hubble telescope
- `Orientation`: coordinates for deep space scan used by Hubble telescope
- `width`: width of image, same if the image did not need to be cropped, but otherwise could differ from `Size`
- `height`: height of image, same if the image did not need to be cropped, but otherwise could differ from `Size`
- `file_size`: `width` x `height` x 3 bytes, used to estimate size of raw images
- `crop_w`: width starting point of image if cropped, otherwise 0
- `crop_h`: height starting point of image if cropped, otherwise 0
- `cropped`: whether this image needed to be cropped or not

### Data Splits

The data is only provided in a single training split, as the purpose of the dataset is additional fine-tuning for the task of `text-to-image` generation.

## Dataset Creation

### Curation Rationale

The ESA Hubble Deep Space Images & Captions dataset was built to provide ease of access to extremely high-quality Hubble deep space scans.
Images from the Hubble telescope have already inspired millions, and the hope is that this dataset can be used to create inspiring models and approaches to further push interest in space & cosmology.

### Source Data

#### Initial Data Collection

All images were collected from [https://esahubble.org/](https://esahubble.org/).
Fullsize Original images & metadata were crawled from the ESA Hubble website using [Scrapy](https://scrapy.org/).
Images were downloaded as `.tiff` files, while
additional metadata was later collected for each image using the following [code](https://github.com/Supermaxman/hubble-diffusion).

As the ESA Hubble website collects images from a wide variety of sources, images were filtered to try to avoid any non-space scan images as follows:
The ESA Hubble [Advanced Image Search](http://esahubble.org/images/archive/search) enables the following filtering parameters:

- images with Minimum size greater than or equal to 400x300
- Ranking greater than or equal to Fair or better
- Type containing 'Observation'

This reduced significantly the number of images which had nothing to do with Hubble deep space scans.
A total of around 3,000 space images were collected with this method.

#### Filtering

Further automatic and manual filtering was performed to remove the following:

- improperly classified images
- space renders
- diagrams with text
- images of celestial bodies within our solar system
- images with too low a resolution

This brought the total number of deep space images down to 2,593.
This process was not perfect, and there likely remain some images in the dataset that should be removed in the future.

#### Preprocessing

Some of the deep space scans were as large as 34,372x19,345, with a bit depth of 24 (nearly 2 GB).
Unfortunately, these images were too large to upload easily
Therefore, images were automatically subdivided in half if they were above 12,000 pixels in either height or width.
Subdivided images were also tagged with additional metadata, such that users can reconstruct the original images if they would prefer.
Otherwise, metadata was copied across subdivided images.
Additionally, images were converted from RGB/RGBX `.tiff` to RGB `.png` files to avoid encoding issues.
This process resulted in 2,706 final deep space images.

### Annotations

The dataset does not contain any additional annotations.

#### Annotation process

[N/A]

#### Who are the annotators?

[N/A]

### Personal and Sensitive Information

[N/A]

## Considerations for Using the Data

### Social Impact of Dataset

The purpose of this dataset is to help inspire people to be interested in astronomy.

A system that succeeds at text-to-image generation would be able to generate inspiring deep space scans, providing interesting and inspiring art for those interested in space. This dataset provides a starting-point for building such a system by providing text and image pairs for Hubble deep space scans.

### Discussion of Biases

It is unfortunate that we currently only have English captions for these deep space scans.
In the future, expanding these captions to more languages could help spread interest in astronomy far and wide.

Additionally, these captions may be too technical for the average person to effectively utilize for a text-to-image model.

### Other Known Limitations

[N/A]

## Additional Information

### Dataset Curators

The dataset was initially created by all the wonderful researchers, engineers, scientists, and more behind the Hubble Telescope, NASA, and the ESA.
Maxwell Weinzierl collected, filtered, and preprocessed this data for ease of use.

### Licensing Information

ESA/Hubble images, videos and web texts are released under the [Creative Commons Attribution 4.0 International license](https://creativecommons.org/licenses/by/4.0/)
and may on a non-exclusive basis be reproduced without fee provided they are clearly and visibly credited.
See [https://esahubble.org/copyright/](https://esahubble.org/copyright/) for additional conditions for reproduction and copyright.

### Citation Information

If you use this dataset, please cite it as:

```bibtex
@misc{weinzierl2023hubble,
      author = {Weinzierl, Maxwell A.},
      title = {ESA Hubble Deep Space Images & Captions},
      year={2023},
      howpublished= {\url{https://huggingface.co/datasets/Supermaxman/esa-hubble}}
} 
```

### Contributions

Thanks to [@supermaxman](https://github.com/supermaxman) for adding this dataset.
