size_categories:n<1K license:cc-by-4.0 arxiv:2309.08596 region:us
---
pretty_name: Robust e-NeRF
paperswithcode_id: robust-e-nerf-synthetic-event-dataset
license: cc-by-4.0
viewer: false
size_categories:
- n<1K
---

# Robust *e*-NeRF Synthetic Event Dataset

<table style="display: block">
  <tr>
    <td><a href="https://wengflow.github.io/robust-e-nerf"><img src="https://img.shields.io/badge/Project_Page-black?style=for-the-badge" alt="Project Page"></a></td>
    <td><a href="https://arxiv.org/abs/2309.08596"><img src="https://img.shields.io/badge/arXiv-black?style=for-the-badge" alt="arXiv"></a></td>
    <td><a href="https://github.com/wengflow/robust-e-nerf"><img src="https://img.shields.io/badge/Code-black?style=for-the-badge" alt="Code"></a></td>
    <td><a href="https://github.com/wengflow/rpg_esim"><img src="https://img.shields.io/badge/Simulator-black?style=for-the-badge" alt="Simulator"></a></td>
  </tr>
</table>

<p align="center">
  <img src="assets/dataset_easy.gif" alt="Easy" width=60%/>
  <img src="assets/dataset_medium.gif" alt="Medium" width=60%/>
  <img src="assets/dataset_hard.gif" alt="Hard" width=60%/>
</p>

This repository contains the synthetic event dataset used in [**Robust *e*-NeRF**](https://wengflow.github.io/robust-e-nerf) to study the collective effect of camera speed profile, contrast threshold variation and refractory period on the quality of NeRF reconstruction from a moving event camera. The dataset is simulated using an [improved version of ESIM](https://github.com/wengflow/rpg_esim) with three different camera configurations of increasing difficulty levels (*i.e.* *easy*, *medium* and *hard*) on seven Realistic Synthetic 360 scenes (adopted in the synthetic experiments of NeRF), resulting in a total of 21 sequence recordings. Please refer to the [Robust *e*-NeRF paper](https://arxiv.org/abs/2309.08596) for more details.

This synthetic event dataset allows for a retrospective comparison between event-based and image-based NeRF reconstruction methods, as the event sequences were simulated under highly similar conditions as the NeRF synthetic dataset. In particular, we adopt the same camera intrinsics and camera distance to the object at the origin. Furthermore, the event camera travels in a hemi-/spherical spiral motion about the object, thereby having a similar camera pose distribution for training. Apart from that, we also use the same test camera poses/views. Nonetheless, this new synthetic event dataset is not only specific to NeRF reconstruction, but also suitable for novel view synthesis, 3D reconstruction, localization and SLAM in general. 

If you use this synthetic event dataset for your work, please cite:

```bibtex
@inproceedings{low2023_robust-e-nerf,
  title = {Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion},
  author = {Low, Weng Fei and Lee, Gim Hee},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year = {2023}
}
```

## Dataset Structure and Contents

This synthetic event dataset is organized first by scene, then by level of difficulty. Each sequence recording is given in the form of a [ROS bag](http://wiki.ros.org/rosbag) named `esim.bag`, with the following data streams:

| ROS Topic | Data | Publishing Rate (Hz) |
| :--- | :--- | :--- |
| `/cam0/events` | Events | - |
| `/cam0/pose` | Camera Pose | 1000 |
| `/imu` | IMU measurements with simulated noise | 1000 |
| `/cam0/image_raw` | RGB image | 250 |
| `/cam0/depthmap` | Depth map | 10 |
| `/cam0/optic_flow` | Optical flow map | 10 |
| `/cam0/camera_info` | Camera intrinsic and lens distortion parameters | 10

It is obtained by running the improved ESIM with the associated `esim.conf` configuration file, which references camera intrinsics configuration files `pinhole_mono_nodistort_f={1111, 1250}.yaml` and camera trajectory CSV files `{hemisphere, sphere}_spiral-rev=4[...].csv`. 

The validation and test views of each scene are given in the `views/` folder, which is structured according to the NeRF synthetic dataset (except for the depth and normal maps). These views are rendered from the scene Blend-files, given in the `scenes/` folder. Specifically, we create a [Conda](https://docs.conda.io/en/latest/) environment with [Blender as a Python module](https://docs.blender.org/api/current/info_advanced_blender_as_bpy.html) installed, according to [these instructions](https://github.com/wengflow/rpg_esim#blender), to run the `bpy_render_views.py` Python script for rendering the evaluation views.

## Setup

1. Install [Git LFS](https://git-lfs.com/) according to the [official instructions](https://github.com/git-lfs/git-lfs?utm_source=gitlfs_site&utm_medium=installation_link&utm_campaign=gitlfs#installing).
2. Setup Git LFS for your user account with:
    ```bash
    git lfs install
    ```
3. Clone this dataset repository into the desired destination directory with:
    ```bash
    git lfs clone https://huggingface.co/datasets/wengflow/robust-e-nerf
    ```
4. To minimize disk usage, remove the `.git/` folder. However, this would complicate the pulling of changes in this upstream dataset repository.
