task_categories:object-detection size_categories:1B<n<10B license:mit object detection doi:10.57967/hf/0986 region:us
---
license: mit
task_categories:
- object-detection
tags:
- object detection
pretty_name: M2SODAI
size_categories:
- 1B<n<10B
---
# Dataset Card for Dataset Name

## Dataset Description

- **Homepage: m2sodai.jonggyu.me** 
- **Repository: temporarily private** 
- **Paper: under review** 
- **Point of Contact: jgjang0123 [at] gmail [dot] com** 

### Dataset Summary

The M<sup>2</sup>SODAI dataset is the first multi-modal, bounding-box-labeled, and synchronized aerial dataset. 

Used Sensor:
- Hyperspectral image
- RGB image

## Dataset Structure

```md
data
├── label.txt
├── train
│   ├── 1.jpg
│   ├── 1.mat
│   ├── 1.json
│   └── ...
├── val
│   ├── 0.jpg
│   ├── 0.mat
│   └── 0.json
└── test
    ├── 17.jpg
    ├── 17.mat
    └── 17.json
```

### Data Instances

For object detection, we annotated the bounding boxes on the floating matters and ships in the RGB and HSI data.
We note that the floating matter contains buoys, rescue tubes, lifeboats, etc.
Since small objects are hard to be recognized, we refer to the infrared visualization map of the HSI data for bounding box annotation.

### Data Splits

After the data processing, we obtained 1,257 pairs of synchronized RGB and HSI data, where the total number of instances in the dataset is 11,892. 
For experiments, we randomly divided the dataset into 1,007 training data, 125 validation data, and 125 test data. 

## Dataset Creation

### Source Data

Our focus is to create a public dataset consisting of synchronized maritime aerial RGB and HSI data. 
To this end, we built a data collection system by leveraging a single-engine utility aircraft (Cessna Grand Caravan 208B).
An HSI sensor (AsiaFENIX, Specim, Oulu, Finland) and an RGB sensor (DMC, Z/I Imaging, Aalen, Germany) are equipped on the bottom of the aircraft, the direction of which is downward.
The raw data was acquired through 59 flight strips in 12 flight measurement campaigns, which cover a total area of 299.7 km<sup>2</sup>. 
During the flight strips, the aircraft maintains its speed of 260 km/h and altitude of 1 km. 

The below table shows the detailed specifications of the sensors used in the data collection. 
The HSI sensor (AsiaFENIX) scans the wavelength range from 400 nm to 1000 nm in steps of 4.5 nm, a total of 127 spectrum bands. 
The wavelength range includes visible spectrum (VIS) and near-infrared (NIR) spectrum, generally used for remote sensing and machine vision tasks.
The RGB sensor (DMC) captures high-resolution RGB data in three channels: Red (590-675 nm), Green (500-650 nm), and Blue (400-580 nm). 
We note that RGB and HSI data are collected simultaneously, in which the spatial resolutions of RGB and HSI sensors are approximately 0.1 m and 0.7 m, respectively.

|               | HSI sensor                                     | RGB sensor                                         |
|---------------|------------------------------------------------|----------------------------------------------------|
| Name          | AsiaFENIX (@Specim)                            | DMC (@Z/I Imaging)                                 |
| Spectrum      | 400-1000 nm, 127 channels (in steps of 4.5 nm) | Blue: 400-580 nm Green: 500-650 nm Red: 590-675 nm |
| Altitude      | 1km                                            | 1km                                                |
| Field of View | 40 degree                                      | 74 degree                                          |
| Resolution    | 0.7 m                                          | 0.1 m                                              |

<img src="https://s3.amazonaws.com/moonup/production/uploads/6487107c86b4bc5a09f9d62e/ihWLuijyA1j38RbSoEC7_.png"  width="50%">
Illustration of the collected raw data. We collected the data on twelve spots. The first row shows the collected raw RGB data. The second and third rows show the overall HSI data and collected raw HSI data in each flight strip.  In this figure, since the sensors have different specifications on the field of view (FoV), the raw RGB data and HSI data have different coordinates.

### Data Processing and Annotations

Since the size of the raw data is too large for object detection (HSI: 3,220<sup>2</sup> pixels, and RGB: 22,520<sup>2</sup> pixels on average), we cropped the raw data into a fixed size. We note that RGB and HSI data are cropped in size of 1600 X 1600 X 3 and 224 X 224 X 127, respectively. 
However, the problem is that the coordinates of the collected RGB and HSI pairs are not matched. Hence, we employ an image registration method to correct pixel offsets between RGB and HSI pairs. In the below figure, our data processing procedure is depicted. 

<img src="https://s3.amazonaws.com/moonup/production/uploads/6487107c86b4bc5a09f9d62e/f4XBxM5Q1Qij7T4UkCf34.png" width="50%">

1. We transform the raw RGB and HSI data into grayscale images. 
2. We apply contrast-limited adaptive histogram equalization (CLAHE)-based contrast enhancer to the grayscale RGB data and grayscale HSI data.
3. To estimate the homography matrix between the enhanced RGB data and enhanced HSI data, we carry out the oriented FAST and rotated BRIEF (ORB) feature descriptor to both data, thereby extracting features of the data.
4. We use a Brute-force matcher to find the matched feature among the ORB features; then, the homography matrix is computed from least square optimization for synchronizing the matched features.
5. We crop the registered data in the same size and generate corresponding bounding box annotation data. 

### Personal and Sensitive Information

There is no personal/sensitive information in our dataset


### Citation Information

[N/A]
