task_categories:text-retrieval task_categories:text-generation task_ids:document-retrieval language_creators:found multilinguality:multilingual source_datasets:original language:code license:cc-by-nc-nd-4.0 code arxiv:2306.03091 region:us
RepoBench is a dataset that benchmarks repository-level code auto-completion systems.

RepoBench-P denotes RepoBench for pipeline, 
which is subtask of RepoBench including both relevant code retrieval and next-line code prediction.
@misc{liu2023repobench,
      title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems}, 
      author={Tianyang Liu and Canwen Xu and Julian McAuley},
      year={2023},
      eprint={2306.03091},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
---
language_creators:
- found
language:
- code
license:
- cc-by-nc-nd-4.0
multilinguality:
- multilingual
pretty_name: RepoBench-Pipeline
source_datasets:
- original
task_categories:
- text-retrieval
- text-generation
task_ids:
- document-retrieval
tags:
- code
---

# Dataset Card for RepoBench-P

## Dataset Description

- **Homepage:** https://github.com/Leolty/repobench
- **Paper:** https://arxiv.org/abs/2306.03091

## Dataset Summary

**RepoBench-P (Pipeline)** is a subtask of **RepoBench**([GitHub](https://github.com/Leolty/repobench), [arXiv](https://arxiv.org/abs/2306.03091)), combinig the retrieval and code completion tasks. Specifically, the retrieval task is used to retrieve the most relevant code snippet first, and then do the code completion task with retrieved code snippet as cross-file context for next-line prediction, which mirrors complex real-world scenarios that a practical auto-completion system would face.

## Settings

- `cff`: short for cross_file_first, indicating the cross-file module in next line is first used in the current file.

- `cfr`: short for cross_file_random, indicating the cross-file module in next line is not first used in the current file.

- `if`: short for in_file, indicating the next line does not contain any cross-file module.

## Supported Languages

- `python` and `java`

## Loading Data

For example, to load the `python` dataset, and you can provide the `split` argument to choose the specific setting.

```python
from datasets import load_dataset

dataset = load_dataset("tianyang/repobench-p", "python", split="cff")
```

> Note: The `split` argument is optional. If not provided, the entire dataset will be loaded.

## Dataset Structure

```json
{
    "repo_name": "repository name of the data point",
    "file_path": "path/to/current_file",
    "context": [
        {
            "path": "path/to/cross_file_1",
            "identifier": "identifier of the cross-file module",
            "snippet": "the code snippet of the cross-file module",
            "tokenized_snippet": "tokenized code snippet of the cross-file module"
        },
        // ...
        {
            "path": "path/to/cross_file_k",
            "identifier": "identifier of the cross-file module",
            "snippet": "the code snippet of the cross-file module",
            "tokenized_snippet": "tokenized code snippet of the cross-file module"
        },
    ],
    "import_statement": "all import statements in current file",
    "code": "the code for next-line prediction",
    "next_line": "the next line of the code",
    "gold_snippet_index": 2 // NOTE: Only for "cross_file_first" and "cross_file_random" settings, for "in_file" setting, we set it to -1.

}
```

## Licensing Information

CC BY-NC-ND 4.0

## Citation Information

```bibtex
@misc{liu2023repobench,
      title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems}, 
      author={Tianyang Liu and Canwen Xu and Julian McAuley},
      year={2023},
      eprint={2306.03091},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

## Contributions

Thanks to [@Leolty](https://github.com/Leolty) for adding this dataset.