task_categories:object-detection task_categories:image-segmentation task_categories:robotics task_ids:instance-segmentation task_ids:semantic-segmentation annotations_creators:machine-generated size_categories:1K<n<10K language:en license:gpl-3.0 vision image segmentation instance segmentation object detection synthetic sim-to-real region:us
An instance segmentation dataset for robotic manipulation in a tabletop environment.
The dataset incorporates real and synthetic images for testing sim-to-real model transfer after fine-tuning.
@ARTICLE{10145828,
  author={Károly, Artúr István and Tirczka, Sebestyén and Gao, Huijun and Rudas, Imre J. and Galambos, Péter},
  journal={IEEE Transactions on Cybernetics}, 
  title={Increasing the Robustness of Deep Learning Models for Object Segmentation: A Framework for Blending Automatically Annotated Real and Synthetic Data}, 
  year={2023},
  volume={},
  number={},
  pages={1-14},
  doi={10.1109/TCYB.2023.3276485}}
---
language:
- en
license: gpl-3.0
tags:
- vision
- image segmentation
- instance segmentation
- object detection
- synthetic
- sim-to-real
annotations_creators:
- machine-generated
pretty_name: OE Dataset
size_categories:
- 1K<n<10K
task_categories:
- object-detection
- image-segmentation
- robotics
task_ids:
- instance-segmentation
- semantic-segmentation
---

# The OE Dataset!

![OE demo](https://huggingface.co/datasets/ABC-iRobotics/oe_dataset/resolve/main/OE_demo.gif "OE demo")

A dataset consisting of synthetic and real images annotated with instance segmentation masks for testing sim-to-real model performance for robotic manipulation.

### Dataset Summary

The OE Dataset is a collection of synthetic and real images of 3D-printed OE logos. Each image is annotated with instance segmentation masks. The dataset explicitly marks synthetic samples based on their creation method (either photorealistic synthetic samples or domain randomized samples) to facilitate sim-to-real performance tests on different synthetic datasets.

### Supported Tasks and Leaderboards

The dataset supports tasks such as semantic segmentation, instance segmentation, object detection, image classification, and testing sim-to-real transfer.

## Dataset Structure

### Data Instances

The instances of the dataset are 1920x1080x3 images in PNG format. The annotations are 1920x1080x4 PNG images representing the instance segmentation masks, where each instance is associated with a unique color.

### Data Fields

The data fields are:

1) 'image': 1920x1080x3 PNG image
2) 'mask': 1920x1080x4 PNG image

### Data Splits

The dataset contains training and validation splits for all image collections (real images, photorealistic synthetic images, domain randomized synthetic images) to facilitate cross-domain testing.

## Dataset Creation

### Curation Rationale

The dataset was created to provide a testbed for examining the effects of fine-tuning instance segmentation models on synthetic data (using various sim-to-real approaches).

### Source Data

The data is generated using two methods:

- Real images are recorded using a robotic setup and automatically annotated using the method proposed in [[1]](https://ieeexplore.ieee.org/abstract/document/9922852)
- Synthetic samples are generated using Blender and annotated using the [Blender Annotation Tool (BAT)](https://github.com/ABC-iRobotics/blender_annotation_tool)

### Citation Information

OE Dataset:
```bibtex
@ARTICLE{10145828,
  author={Károly, Artúr István and Tirczka, Sebestyén and Gao, Huijun and Rudas, Imre J. and Galambos, Péter},
  journal={IEEE Transactions on Cybernetics}, 
  title={Increasing the Robustness of Deep Learning Models for Object Segmentation: A Framework for Blending Automatically Annotated Real and Synthetic Data}, 
  year={2023},
  volume={},
  number={},
  pages={1-14},
  doi={10.1109/TCYB.2023.3276485}}
```

Automatically annotating real images with instance segmentation masks using a robotic arm:
```bibtex
@INPROCEEDINGS{9922852,
  author={Károly, Artúr I. and Károly, Ármin and Galambos, Péter},
  booktitle={2022 IEEE 10th Jubilee International Conference on Computational Cybernetics and Cyber-Medical Systems (ICCC)}, 
  title={Automatic Generation and Annotation of Object Segmentation Datasets Using Robotic Arm}, 
  year={2022},
  volume={},
  number={},
  pages={000063-000068},
  doi={10.1109/ICCC202255925.2022.9922852}}
```

Synthetic dataset generation and annotation method:
```bibtex
@INPROCEEDINGS{9780790,
  author={Károly, Artúr I. and Galambos, Péter},
  booktitle={2022 IEEE 20th Jubilee World Symposium on Applied Machine Intelligence and Informatics (SAMI)}, 
  title={Automated Dataset Generation with Blender for Deep Learning-based Object Segmentation}, 
  year={2022},
  volume={},
  number={},
  pages={000329-000334},
  doi={10.1109/SAMI54271.2022.9780790}}
```

Other related publications:
```bibtex
@INPROCEEDINGS{10029564,
  author={Károly, Artúr I. and Tirczka, Sebestyén and Piricz, Tamás and Galambos, Péter},
  booktitle={2022 IEEE 22nd International Symposium on Computational Intelligence and Informatics and 8th IEEE International Conference on Recent Achievements in Mechatronics, Automation, Computer Science and Robotics (CINTI-MACRo)}, 
  title={Robotic Manipulation of Pathological Slides Powered by Deep Learning and Classical Image Processing}, 
  year={2022},
  volume={},
  number={},
  pages={000387-000392},
  doi={10.1109/CINTI-MACRo57952.2022.10029564}}
```

```bibtex
@Article{app13010525,
AUTHOR = {Károly, Artúr István and Galambos, Péter},
TITLE = {Task-Specific Grasp Planning for Robotic Assembly by Fine-Tuning GQCNNs on Automatically Generated Synthetic Data},
JOURNAL = {Applied Sciences},
VOLUME = {13},
YEAR = {2023},
NUMBER = {1},
ARTICLE-NUMBER = {525},
URL = {https://www.mdpi.com/2076-3417/13/1/525},
ISSN = {2076-3417},
ABSTRACT = {In modern robot applications, there is often a need to manipulate previously unknown objects in an unstructured environment. The field of grasp-planning deals with the task of finding grasps for a given object that can be successfully executed with a robot. The predicted grasps can be evaluated according to certain criteria, such as analytical metrics, similarity to human-provided grasps, or the success rate of physical trials. The quality of a grasp also depends on the task which will be carried out after the grasping is completed. Current task-specific grasp planning approaches mostly use probabilistic methods, which utilize categorical task encoding. We argue that categorical task encoding may not be suitable for complex assembly tasks. This paper proposes a transfer-learning-based approach for task-specific grasp planning for robotic assembly. The proposed method is based on an automated pipeline that quickly and automatically generates a small-scale task-specific synthetic grasp dataset using Graspit! and Blender. This dataset is utilized to fine-tune pre-trained grasp quality convolutional neural networks (GQCNNs). The aim is to train GQCNNs that can predict grasps which do not result in a collision when placing the objects. Consequently, this paper focuses on the geometric feasibility of the predicted grasps and does not consider the dynamic effects. The fine-tuned GQCNNs are evaluated using the Moveit! Task Constructor motion planning framework, which enables the automated inspection of whether the motion planning for a task is feasible given a predicted grasp and, if not, which part of the task is responsible for the failure. Our results suggest that fine-tuning GQCNN models can result in superior grasp-planning performance (0.9 success rate compared to 0.65) in the context of an assembly task. Our method can be used to rapidly attain new task-specific grasp policies for flexible robotic assembly applications.},
DOI = {10.3390/app13010525}
}
```