task_categories:text-generation task_categories:text2text-generation task_categories:translation size_categories:1K<n<10K source_datasets:original source_datasets:extended|p3 language:en license:apache-2.0 code arxiv:2302.01973 arxiv:2106.05784 region:us
Translating Python Programming Puzzles (TP3) is a code translation benchmark created from the verification functions from the questions in the original Python Programming Puzzles dataset (Schuster et al., 2021) to create this dataset. These functions are hand-crafted by the authors and are used to check if an answer satisfies the constraints of the puzzle. These puzzles range in difficulty from basic character checking to competitive programming problems. Thus, each verification function is written by an expert python programmer and requires a significant understanding of programming to translate. In total, there are 370 python functions to translate.
@article{orlanski2023measuring,
  title={Measuring The Impact Of Programming Language Distribution},
  author={Orlanski, Gabriel and Xiao, Kefan and Garcia, Xavier and Hui, Jeffrey and Howland, Joshua and Malmaud, Jonathan and Austin, Jacob and Singh, Rishah and Catasta, Michele},
  journal={arXiv preprint arXiv:2302.01973},
  year={2023}
}
@inproceedings{
    schuster2021programming,
    title={Programming Puzzles},
    author={Tal Schuster and Ashwin Kalyan and Alex Polozov and Adam Tauman Kalai},
    booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
    year={2021},
    url={https://arxiv.org/abs/2106.05784}
}
---
license: apache-2.0
task_categories:
- text-generation
- text2text-generation
- translation
language:
- en
tags:
- code
pretty_name: BabelCode TP3
size_categories:
- 1K<n<10K
source_datasets:
  - original
  - extended|p3
---
# Dataset Card for Translating Python Programming Puzzles (TP3)

## Dataset Description

- **Repository:** [GitHub Repository](https://github.com/google-research/babelcode)
- **Paper:** [Measuring The Impact Of Programming Language Distribution](https://arxiv.org/abs/2302.01973)

### How To Use This Dataset

To use this dataset, you can either use the original [BabelCode Repo](https://github.com/google-research/babelcode), or you can use the [`bc_eval` Metric](https://huggingface.co/spaces/gabeorlanski/bc_eval).

### Dataset Summary

The Translating Python Programming Puzzles (TP3) dataset is created from the verification functions in the [Python Programming Puzzles dataset (Schuster et al., 2021)](https://github.com/microsoft/PythonProgrammingPuzzles) to create this dataset. These functions are hand-crafted by the
authors and are used to check if an answer satisfies the constraints of the puzzle. These puzzles range in difficulty from basic character checking to competitive programming problems.

### Supported Tasks and Leaderboards

### Languages
BC-TP3 supports:
* C++
* C#
* Dart
* Go
* Haskell
* Java
* Javascript
* Julia
* Kotlin
* Lua
* PHP
* R
* Rust
* Scala
* TypeScript

## Dataset Structure

```python
>>> from datasets import load_dataset
>>> load_dataset("gabeorlanski/tp3")
DatasetDict({
    test: Dataset({
        features: ['qid', 'title', 'language', 'text', 'signature_with_docstring', 'signature', 'arguments', 'source', 'question_info'],
        num_rows: 5920
    })
})
```

### Data Fields

- `qid`: The question ID used for running tests.
- `title`: The title of the question.
- `language`: The programming language of the example.
- `text`: The description of the problem.
- `signature`: The signature for the problem.
- `signature_with_docstring`: The signature with the adequately formatted docstring for the given problem.
- `arguments`: The arguments of the problem.
- `source`: The source solution in Python.
- `question_info`: The dict of information used for executing predictions. It has the keys:
  - `test_code`: The raw testing script used in the language. If you want to use this, replace `PLACEHOLDER_FN_NAME` (and `PLACEHOLDER_CLS_NAME` if needed) with the corresponding entry points. Next, replace `PLACEHOLDER_CODE_BODY` with the postprocessed prediction.  
  - `test_list`: The raw json line of the list of tests for the problem. To load them, use `json.loads`
  - `test_case_ids`: The list of test case ids for the problem. These are used to determine if a prediction passes or not.
  - `entry_fn_name`: The function's name to use an entry point.
  - `entry_cls_name`: The class name to use an entry point. 
  - `commands`: The commands used to execute the prediction. Includes a `__FILENAME__` hole that is replaced with the filename.
  - `timeouts`: The default timeouts for each command.
  - `extension`: The extension for the prediction file.

**NOTE:** If you want to use a different function name (or class name for languages that require class names) for the prediction, you must update the `entry_fn_name` and `entry_cls_name` accordingly. For example, if you have the original question with `entry_fn_name` of `add`, but want to change it to `f`, you must update `ds["question_info"]["entry_fn_name"]` to `f`:
```python
>>> from datasets import load_dataset
>>> ds = load_dataset("gabeorlanski/bc-mbpp")['test']
>>> # The original entry_fn_name
>>> ds[0]['question_info']['entry_fn_name']
removeOcc
>>> # You MUST update the corresponding entry_fn_name
>>> ds[0]['question_info']['entry_fn_name'] = 'f'
>>> ds[0]['question_info']['entry_fn_name']
f
```
## Dataset Creation
See section 2 and section 4.4 of the [BabelCode Paper](https://arxiv.org/abs/2302.01973) to learn more about how the datasets are translated. 

For information on how the original P3 dataset was collected, please see [Programming Puzzles paper](https://arxiv.org/abs/2106.05784). 

### Dataset Curators
Google Research

### Licensing Information
CC-BY-4.0

### Citation Information
```
@article{orlanski2023measuring,
  title={Measuring The Impact Of Programming Language Distribution},
  author={Orlanski, Gabriel and Xiao, Kefan and Garcia, Xavier and Hui, Jeffrey and Howland, Joshua and Malmaud, Jonathan and Austin, Jacob and Singh, Rishah and Catasta, Michele},
  journal={arXiv preprint arXiv:2302.01973},
  year={2023}
}
@inproceedings{
    schuster2021programming,
    title={Programming Puzzles},
    author={Tal Schuster and Ashwin Kalyan and Alex Polozov and Adam Tauman Kalai},
    booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
    year={2021},
    url={https://arxiv.org/abs/2106.05784}
}
```