region:us
---
dataset_info:
  features:
  - name: conversations
    list:
    - name: from
      dtype: string
    - name: value
      dtype: string
  - name: image
    dtype: string
  - name: id
    dtype: string
  splits:
  - name: train
    num_bytes: 18403899
    num_examples: 65422
  - name: validation
    num_bytes: 2289458
    num_examples: 8174
  - name: test
    num_bytes: 2313629
    num_examples: 8176
  download_size: 8200395
  dataset_size: 23006986
---
# Dataset Card for "roco-instruct-65k"

## Dataset Description

- **Repository:** [ROCO GitHub Repository](https://github.com/razorx89/roco-dataset)
- **Paper:** [Radiology Objects in COntext (ROCO) dataset](https://labels.tue-image.nl/wp-content/uploads/2018/09/AM-04.pdf)
- **Point of Contact:** ROCO's original authors

### Dataset Summary

The "roco-instruct-65k" dataset is derived from the Radiology Objects in COntext (ROCO) dataset, a large-scale medical and multimodal imaging collection. The images are taken from publications available on the PubMed Central Open Access FTP mirror. The dataset was reformatted for the [LLaVA model](https://llava-vl.github.io/) in the [BabyDoctor project](https://github.com/photomz/BabyDoctor), focusing on deep analysis and diagnosis of radiology images. It includes captions, keywords, UMLS Semantic Types (SemTypes), and UMLS Concept Unique Identifiers (CUIs), and supports the creation of generative models for image captioning, classification models for image categorization, and tagging or content-based image retrieval systems. The language used is primarily English, and it covers the domain of medical imaging, specifically radiology.

### Supported Tasks and Leaderboards

- `image-classification`: The dataset can be used to train models for image classification, which involves categorizing images as either radiology or non-radiology. Success on this task is typically measured by achieving a high accuracy. This task has an active leaderboard which can be found at [ImageCLEFmed Caption 2019 and CrowdAI](https://www.imageclef.org/2019/medical/caption).

### Languages

The dataset consists entirely of medical texts in English.

## Dataset Structure

### Data Instances

The dataset is structured in a conversation format where a human provides an image with instructions for analysis, and a model responds with a diagnosis. A typical instance in the dataset looks like:

```json
{
  'conversations': [
    { "from": "human", "value": "The following image is a radiology scan. Deeply analyze and diagnose this image.\n<image>" },
    { "from": "gpt", "value": "Computed tomography scan in axial view showing obliteration of the left maxillary sinus" }
  ],
  'image': "ROCO_00002.jpg",
  'id': "00002"
}
```

### Data Fields

- `conversations`: A list containing the interaction between a human and a model regarding the image.
- `image`: A string containing the name of the image file.
- `id`: A string representing the unique identifier for the interaction.

### Data Splits

The dataset is divided into training, validation, and test sets. The exact split sizes are:

|                 | train  | validation | test  |
|-----------------|-------:|-----------:|------:|
| Data Instances  |   65000|       8200 |  8200 |

## Dataset Creation

### Curation Rationale

The "roco-instruct-65k" dataset was created to foster the development of AI models capable of performing deep analysis and diagnosis on radiology images, an essential step in automating medical imaging interpretation.

### Citation Information

[@photomz](https://github.com/photomz) uploaded this dataset to HuggingFace. Please cite the original ROCO paper when using this dataset.

```
O. Pelka, S. Koitka, J. Rückert, F. Nensa, C.M. Friedrich,
"Radiology Objects in COntext (ROCO): A Multimodal Image Dataset".
MICCAI Workshop on Large-scale Annotation of Biomedical Data and Expert Label Synthesis (LABELS) 2018, September 16, 2018, Granada, Spain. Lecture Notes on Computer Science (LNCS), vol. 11043, pp. 180-189, Springer Cham, 2018.
doi: 10.1007/978-3-030-01364-6_20
```