task_categories:text2text-generation size_categories:n<1K language:en license:mit code-generation arxiv:2308.01861 region:us
FudanSELab ClassEval
@misc{du2023classeval,
      title={ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation}, 
      author={Xueying Du and Mingwei Liu and Kaixin Wang and Hanlin Wang and Junwei Liu and Yixuan Chen and Jiayi Feng and Chaofeng Sha and Xin Peng and Yiling Lou},
      year={2023},
      eprint={2308.01861},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
---
license: mit
language:
- en
size_categories:
- n<1K
tags:
  - code-generation
task_categories:
  - text2text-generation
pretty_name: ClassEval
configs:
- config_name: default
  data_files:
  - split: test
    path: "ClassEval_data.json"
---

# Dataset Card for FudanSELab ClassEval

## Dataset Description

- **Repository:** [GitHub Repository](https://github.com/FudanSELab/ClassEval)
- **Paper:** [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://arxiv.org/abs/2308.01861)

### Dataset Summary

We manually build ClassEval of 100 class-level Python coding tasks, consists of 100 classes and 412 methods, and average 33.1 test cases per class.

For 100 class-level tasks, diversity is maintained by encompassing these tasks over a wide spectrum of topics, including Management Systems, Data Formatting, Mathematical Operations, Game Development, File Handing, Database Operations and Natural Language Processing.

For 412 methods, they have been constructed with diverse dependencies, including (i) Library Dependency, where the methods rely on specific external libraries; (ii) Field Dependency, in which the methods are contingent on class instance variables, or fields; (iii) Method Dependency, where the methods are dependent on other methods within the same class; and (iv) Standalone, wherein the methods operate independently without reliance on fields, other methods, or external libraries.

### Languages

The programming language is Python. The natural language used in the comments and docstrings is English.

## Dataset Structure

```python
from datasets import load_dataset
dataset = load_dataset("FudanSELab/ClassEval")

DatasetDict({
    test: Dataset({
        features: ['task_id', 'skeleton', 'test', 'solution_code', 'import_statement', 'class_description', 'methods_info', 
'class_name', 'test_classes', 'class_constructor', 'fields'],
        num_rows: 100
    })
})
```

### Data Fields

The specific data fields for each task are delineated as follows:

* task_id: the unique identifier for each task.

* skeleton: the class skeleton, including all input descriptions in our class-level coding tasks. 

* test: all test cases for the whole class.

* solution_code: the ground-truth class-level code for each task.

More fine-grained class-level information from the class skeleton, including:

* import_statement: the import statements for each task.

* class_name: the name of the class.

* class_description: a concise description of the purpose and functionality of the class.

* class_constructor: the whole constructor of the class.

* fields: the fields defined in the class_constructor.

Detailed information for each method in the "methods_info" field, including:

* method_name: the method signature.

* method_input: the method contract design, including all input descriptions in the method.

* test_code: the test cases for the method.

* solution_code: the ground-truth method-level code.

* dependencies: the dependency information of the method.

### Data Splits

The dataset only consists of a test split with 100 samples.

## Dataset Creation

### Source Data

Manually-crafted

## Additional Information

### Licensing Information

This repository is under [MIT](https://github.com/FudanSELab/ClassEval/blob/master/LICENSE) license. But the data is distributes through [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license.

### Citation Information

```
@misc{du2023classeval,
      title={ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation}, 
      author={Xueying Du and Mingwei Liu and Kaixin Wang and Hanlin Wang and Junwei Liu and Yixuan Chen and Jiayi Feng and Chaofeng Sha and Xin Peng and Yiling Lou},
      year={2023},
      eprint={2308.01861},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

### Contributions

Xueying Du xueyingdu21@m.fudan.edu.cn

Mingwei Liu liumingwei@fudan.edu.cn

Kaixin Wang kxwang23@m.fudan.edu.cn

Hanlin Wang wanghanlin23@m.fudan.edu.cn

Junwei Liu jwliu22@m.fudan.edu.cn

Yixuan Chen 23212010005@m.fudan.edu.cn

Jiayi Feng 23210240148@m.fudan.edu.cn

Chaofeng Sha cfsha@fudan.edu.cn

Xin Peng pengxin@fudan.edu.cn

Yiling Lou yilinglou@fudan.edu.cn
