task_categories:other annotations_creators:expert-generated size_categories:100K<n<1M language:en license:cc-by-4.0 novel view synthesis inverse rendering material decomposition doi:10.57967/hf/1102 region:us
---
language:
- en
license: cc-by-4.0
tags:
- novel view synthesis
- inverse rendering
- material decomposition
annotations_creators:
- expert-generated
pretty_name: OpenIllumination
size_categories:
- 100K<n<1M
task_categories:
- other
download_size: 900G
---

# Dataset Card for OpenIllumination

## Dataset Description

- **Homepage:** https://oppo-us-research.github.io/OpenIllumination
- **Repository:** https://github.com/oppo-us-research/OpenIlluminationCapture.
- **Paper:** https://oppo-us-research.github.io/OpenIllumination/files/openillumination.pdf.
- **Leaderboard:** N/A for now.
- **Point of Contact:** lic032@ucsd.edu

### Dataset Summary

Our dataset comprises 64 objects, each captured from 70 views, under 13 lighting patterns and 142 One-Light-At-Time (OLAT) illumination, respectively. 
The 70 views are captured by 48 DSLR cameras and 22 high-speed cameras. The dataset viewer is at the [project page](https://oppo-us-research.github.io/OpenIllumination).

### Supported Tasks

* Novel view synthesis: The dataset can be used to evaluate NVS methods, such as NeRF, TensoRF, and NeuS.
* Inverse rendering: The dataset can be used to evaluate inverse rendering algorithms, which are to decompose illumination, object geometry, and object materials.

### Dataset Download

​	Since the whole dataset is very large, we provide a script [here](https://huggingface.co/datasets/OpenIllumination/OpenIllumination/blob/main/open_illumination.py]) to download according to the illumination type (lighting pattern or OLAT) and the object ID. You can also modify the code to customize it according to your requirements.

### Languages

English

## Dataset Structure

### Data Fields

For each image, the following fields are provided:

* file_path: str, the file path to an image.
* light_idx: int, the index of illuminations, from 1 to 13 for lighting patterns, or from 0 to 141 for OLAT.
* transform_matrix: list, a 4x4 matrix, representing the camera pose for this image (in OpenCV convention).
* camera_angle_x: float, can be used to compute the corresponding camera intrinsics.
* obj_mask: the object mask, can be read by ```imageio.imread(OBJ_MASK_PATH)>0```, used for PSNR evaluation. **Note that we store 0 and 1 for object masks, so you may not be able to correctly visualize it as png in this repo.**
* com_mask (optional): the union of the object mask and the support mask, can be read by ```imageio.imread(COM_MASK_PATH)>0```, used for training. We store 0 and 255 for combined masks, so you are able to view them as png in this repo correctly.

### Data Splits

The data is split into training and testing views. For each object captured under 13 lighting patterns, the training set and the testing set contain 38 and 10 views, respectively. For each object captured under OLAT, the training set and the testing set contain 17 and 5 views, respectively.

## Dataset Creation

### Curation Rationale

From the paper:

> Recent efforts have introduced some datasets that incorporate multiple illuminations in real-world settings. However, most of them are limited either in the number of views or the number of illuminations; few of them provide object-level data as well. Consequently, these existing datasets prove unsuitable for evaluating inverse rendering methods on real-world objects.
>
>
> To address this, we present a new dataset containing objects with a variety of materials, captured under multiple views and illuminations, allowing for reliable evaluation of various inverse rendering tasks with real data.

### Source Data

#### Initial Data Collection and Normalization

From the paper:

> Our dataset was acquired using a setup similar to a traditional light stage, where densely distributed cameras and controllable lights are attached to a static frame around a central platform.

### Annotations

#### Annotation process

From the paper:

> To obtain high-quality segmentation masks, we propose to use Segment-Anything (SAM) to perform instance segmentation. However, 
> we find that the performance is not satisfactory. One reason is that the object categories are highly undefined. In this case, even combining the bounding box and point prompts cannot produce satisfactory results. To address this problem, we propose to use multiple bounding-box prompts to perform segmentation for each possible part and then calculate a union of the masks as the final object mask.
>
> For objects with very detailed and thin structures, e.g. hair, we use an off-the-shelf background matting method to perform object segmentation.

#### Who are the annotators?

Linghao Chen, Isabella Liu, and Ziyang Fu.

## Additional Information

### Dataset Curators

Isabella Liu, Linghao Chen, Ziyang Fu, Liwen Wu, Haian Jin, Zhong Li, Chin Ming Ryan Wong, Yi Xu, Ravi Ramamoorthi, Zexiang Xu and Hao Su

### Licensing Information

CC BY 4.0 

### Citation Information

```bash
@article{liu2023openillumination,
	title={OpenIllumination: A Multi-Illumination Dateset for Inverse Rendering Evaluation on Real Objects},
	author={Liu, Isabella and Chen, Linghao and Fu, Ziyang and Wu, Liwen and Jin, Haian and Li, Zhong and Chin Ming Ryan Wong3 and Xu, Yi and Ravi Ramamoorthi1 and Xu, Zexiang and Su, Hao},
	year={2023}
}
```