task_categories:question-answering annotations_creators:expert-generated annotations_creators:auto-generated language_creators:machine-generated multilinguality:monolingual size_categories:1K<n<10K source_datasets:original language:en license:cc-by-4.0 knowledge-base-qa region:us
    SciQA contains 2,565 SPARQL query - question pairs along with answers fetched from the open research knowledge graph (ORKG)     via a Virtuoso SPARQL endpoint, it is a collection of both handcrafted and autogenerated questions and queries.     The dataset is split into 70% training, 10% validation and 20% test examples. The dataset is available as JSON files.
@Article{SciQA2023,
        author={Auer, S{\"o}ren
        and Barone, Dante A. C.
        and Bartz, Cassiano
        and Cortes, Eduardo G.
        and Jaradeh, Mohamad Yaser
        and Karras, Oliver
        and Koubarakis, Manolis
        and Mouromtsev, Dmitry
        and Pliukhin, Dmitrii
        and Radyush, Daniil
        and Shilin, Ivan
        and Stocker, Markus
        and Tsalapati, Eleni},
        title={The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge},
        journal={Scientific Reports},
        year={2023},
        month={May},
        day={04},
        volume={13},
        number={1},
        pages={7240},
        abstract={Knowledge graphs have gained increasing popularity in the last decade in science and technology. However, knowledge graphs are currently relatively simple to moderate semantic structures that are mainly a collection of factual statements. Question answering (QA) benchmarks and systems were so far mainly geared towards encyclopedic knowledge graphs such as DBpedia and Wikidata. We present SciQA a scientific QA benchmark for scholarly knowledge. The benchmark leverages the Open Research Knowledge Graph (ORKG) which includes almost 170,000 resources describing research contributions of almost 15,000 scholarly articles from 709 research fields. Following a bottom-up methodology, we first manually developed a set of 100 complex questions that can be answered using this knowledge graph. Furthermore, we devised eight question templates with which we automatically generated further 2465 questions, that can also be answered with the ORKG. The questions cover a range of research fields and question types and are translated into corresponding SPARQL queries over the ORKG. Based on two preliminary evaluations, we show that the resulting SciQA benchmark represents a challenging task for next-generation QA systems. This task is part of the open competitions at the 22nd International Semantic Web Conference 2023 as the Scholarly Question Answering over Linked Data (QALD) Challenge.},
        issn={2045-2322},
        doi={10.1038/s41598-023-33607-z},
        url={https://doi.org/10.1038/s41598-023-33607-z}
    }
---
annotations_creators:
- expert-generated
- auto-generated
language:
- en
language_creators:
- machine-generated
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: 'The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge'
size_categories:
- 1K<n<10K
source_datasets:
- original
tags:
- knowledge-base-qa
task_categories:
- question-answering
task_ids: []
---

# Dataset Card for SciQA

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [SciQA Homepage]()
- **Repository:** [SciQA Repository](https://zenodo.org/record/7744048)
- **Paper:** The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge
- **Point of Contact:** [Yaser Jaradeh](mailto:Yaser.Jaradeh@tib.eu)

### Dataset Summary

SciQA contains 2,565 SPARQL query - question pairs along with answers fetched from the open research knowledge graph (ORKG) via a Virtuoso SPARQL endpoint, it is a collection of both handcrafted and autogenerated questions and queries. The dataset is split into 70% training, 10% validation and 20% test examples. 

## Dataset Structure

### Data Instances

An example of a question is given below:

```json
{
    "id": "AQ2251",
    "query_type": "Factoid",
    "question": {
        "string": "Provide a list of papers that have utilized the Depth DDPPO model and include the links to their code?"
    },
    "paraphrased_question": [],
    "query": {
        "sparql": "SELECT DISTINCT ?code\nWHERE {\n  ?model    a  orkgc:Model;\n            rdfs:label    ?model_lbl.\n  FILTER (str(?model_lbl) = \"Depth DDPPO\")\n  ?benchmark      orkgp:HAS_DATASET        ?dataset.\n  ?cont           orkgp:HAS_BENCHMARK      ?benchmark.\n  ?cont           orkgp:HAS_MODEL          ?model;\n                  orkgp:HAS_SOURCE_CODE    ?code.\n}"
    },
    "template_id": "T07",
    "auto_generated": true,
    "query_shape": "Tree",
    "query_class": "WHICH-WHAT",
    "number_of_patterns": 4,
}

```
### Data Fields

- `id`: the id of the question
- `question`: a string containing the question
- `paraphrased_question`: a set of paraphrased versions of the question
- `query`: a SPARQL query that answers the question
- `query_type`: the type of the query
- `query_template`: an optional template of the query
- `query_shape`: a string indicating the shape of the query
- `query_class`: a string indicating the class of the query
- `auto_generated`: a boolean indicating whether the question is auto-generated or not
- `number_of_patterns`: an integer number indicating the number of gtaph patterns in the query

### Data Splits

The dataset is split into 70% training, 10% validation and 20% test questions.

## Additional Information

### Licensing Information

SciQA is licensed under the [Creative Commons Attribution 4.0 International License (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/).

### Citation Information

```bibtex
﻿@Article{SciQA2023,
  author={Auer, S{\"o}ren
  and Barone, Dante A. C.
  and Bartz, Cassiano
  and Cortes, Eduardo G.
  and Jaradeh, Mohamad Yaser
  and Karras, Oliver
  and Koubarakis, Manolis
  and Mouromtsev, Dmitry
  and Pliukhin, Dmitrii
  and Radyush, Daniil
  and Shilin, Ivan
  and Stocker, Markus
  and Tsalapati, Eleni},
  title={The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge},
  journal={Scientific Reports},
  year={2023},
  month={May},
  day={04},
  volume={13},
  number={1},
  pages={7240},
  abstract={Knowledge graphs have gained increasing popularity in the last decade in science and technology. However, knowledge graphs are currently relatively simple to moderate semantic structures that are mainly a collection of factual statements. Question answering (QA) benchmarks and systems were so far mainly geared towards encyclopedic knowledge graphs such as DBpedia and Wikidata. We present SciQA a scientific QA benchmark for scholarly knowledge. The benchmark leverages the Open Research Knowledge Graph (ORKG) which includes almost 170,000 resources describing research contributions of almost 15,000 scholarly articles from 709 research fields. Following a bottom-up methodology, we first manually developed a set of 100 complex questions that can be answered using this knowledge graph. Furthermore, we devised eight question templates with which we automatically generated further 2465 questions, that can also be answered with the ORKG. The questions cover a range of research fields and question types and are translated into corresponding SPARQL queries over the ORKG. Based on two preliminary evaluations, we show that the resulting SciQA benchmark represents a challenging task for next-generation QA systems. This task is part of the open competitions at the 22nd International Semantic Web Conference 2023 as the Scholarly Question Answering over Linked Data (QALD) Challenge.},
  issn={2045-2322},
  doi={10.1038/s41598-023-33607-z},
  url={https://doi.org/10.1038/s41598-023-33607-z}
}


```

### Contributions

Thanks to [@YaserJaradeh](https://github.com/YaserJaradeh) for adding this dataset.