task_categories:question-answering size_categories:10K<n<100K language:en license:apache-2.0 arxiv:1705.04146 arxiv:2305.15017 region:us
---
dataset_info:
  features:
  - name: question
    dtype: string
  - name: options
    sequence: string
  - name: rationale
    dtype: string
  - name: correct
    dtype: string
  - name: chain
    dtype: string
  splits:
  - name: train
    num_bytes: 71619470
    num_examples: 97467
  - name: test
    num_bytes: 191142
    num_examples: 254
  - name: validation
    num_bytes: 191976
    num_examples: 254
  download_size: 40727556
  dataset_size: 72002588
license: apache-2.0
task_categories:
- question-answering
language:
- en
pretty_name: AQUA-RAT with Calculator
size_categories:
- 10K<n<100K
---
# Dataset Card for "Calc-aqua_rat"

### Summary

This dataset is an instance of [aqua_rat](https://huggingface.co/datasets/aqua_rat) dataset extended for the in-context calls of calculator,
represented by the `exec` calls to a `sympy` library.

### Supported Tasks
The dataset is intended for training Chain-of-Thought reasoning models able to use external tools to enhance the factuality of their responses.
This dataset presents in-context scenarios where models can out-source the computations in the reasoning chain to a calculator.

### Construction Process

The dataset was constructed automatically by evaluating all candidate calls to a `sympy` library that were extracted from the originally-annotated 
*rationale*s. The selection of candidates is pivoted by the matching of equals ('=') symbols in the chain, where the left-hand side of the equation is evaluated,
and accepted as a correct gadget call, if the result occurs closely on the right-hand side. 
Therefore, the extraction of calculator calls may inhibit false negatives (where the calculator could have been used but was not), but not any known
false positives.

A full description of the extraction process can be found in the [corresponding parse script](https://github.com/markcheeky/gadgets/blob/7799a7841940b15593d4667219424ee71c74327e/gadgets/aqua.py#L19),

**If you find an issue in the dataset or in the fresh version of the parsing script, we'd be happy if you report it, or create a PR.**


## Dataset Structure
The dataset can be loaded by simply choosing a split (`train`, `validation` or `test`) and calling:
```python
import datasets
dataset_val = datasets.load_dataset("MU-NLPC/Calc-aqua_rat", split="validation")
print(dataset_val[0])  # see the output below
```
### Data Instances
The samples of Calc-aqua_rat have this format (newline-reformated for better readability):

```python
{'question': 'Three birds are flying at a fast rate of 900 kilometers per hour. What is their speed in miles per minute? [1km = 0.6 miles]',
 'options': ['A)32400', 'B)6000', 'C)600', 'D)60000', 'E)10'],
 'correct': 'A',
 'rationale': 'To calculate the equivalent of miles in a kilometer\n
               0.6 kilometers = 1 mile\n
               900 kilometers = (0.6)*900 = 540 miles\n
               In 1 hour there are 60 minutes\n
               Speed in miles/minutes = 60 * 540 = 32400\n
               Correct answer - A',
 'chain':     'To calculate the equivalent of miles in a kilometer\n
               0.6 kilometers \n= 1 mile\n
               900 kilometers \n= (0.6)*900\n= \n<gadget id="calculator">(0.6)*900</gadget>\n<output>540</output>\n540 miles\n
               In 1 hour there are 60 minutes\n
               Speed in miles/minutes\n= 60 * 540\n= \n<gadget id="calculator">60 * 540</gadget>\n<output>32_400</output>\n32400\n
               Correct answer - 32400\n.
               Final result is <result>32400</result>'
}
```

The enclosing HTML tags (e.g. **`<gadget id="calculator">(0.6)*900</gadget>\n<output>540</output>`**) represent the inputs and outputs
to the `sympy.parse_expr().evalf()` method (in our code [here](https://github.com/markcheeky/gadgets/blob/7799a7841940b15593d4667219424ee71c74327e/gadgets/gadget.py#L28)).

Note that the format of the dataset is consistent with [MU-NLPC/Calc-gsm8k](https://huggingface.co/datasets/MU-NLPC/Calc-gsm8k).

### Data Fields

* **question**: A natural language definition of the problem to solve.
* **options**: 5 possible options (A, B, C, D and E), among which one is correct
* **correct**: The correct option
* **rationale**: A natural language sequence of steps leading to a solution of the given problem.
* **chain**: A natural language sequence of steps with inserted calculator calls and outputs of the sympy calculator.

### Data Splits
The samples in data splits are consistent with the original [aqua_rat](https://huggingface.co/datasets/aqua_rat) dataset, containing:

* **train** split of 97467 samples,
* **validation** split of 254 samples,
* **test** split of 254.
* 
## Licensing
Apache-2.0, consistently with the original aqua-rat dataset.

## Cite
If you use this dataset in research, please cite the original [aqua-rat paper](https://arxiv.org/pdf/1705.04146.pdf) and our report as follows:

```bibtex
@article{kadlcik2023calcx,
         title={Calc-X: Enriching Arithmetical Chain-of-Thoughts Datasets by Interaction with Symbolic Systems}, 
         author={Marek Kadlčík and Michal Štefánik},
         year={2023},
         eprint={2305.15017},
         archivePrefix={arXiv},
         primaryClass={cs.LG}
}
```