task_categories:text-generation size_categories:100M<n<1B language:en license:cc-by-nc-sa-4.0 region:us
---
dataset_info:
  features:
  - name: problem
    dtype: string
  - name: solution
    dtype: string
  splits:
  - name: train
    num_bytes: 1121418005
    num_examples: 1468146
  download_size: 486193162
  dataset_size: 1121418005
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
task_categories:
- text-generation
language:
- en
size_categories:
- 100M<n<1B
license: cc-by-nc-sa-4.0
---
# Dataset Card for "code_exercises"

# Code exercise
This dataset is composed of a diverse set of \~120k Python code exercises (~120m total tokens) generated by ChatGPT 3.5. It is designed to distill ChatGPT 3.5 knowledge about Python coding tasks into other (potentially smaller) models. The exercises have been generated by following the steps described in the [related GitHub repository](https://github.com/jina-ai/textbook).

The generated exercises follow the format of the [Human Eval benchmark](https://github.com/openai/human-eval). Each training sample is split into a Python function signature with a descriptive docstring, and a solution to the exercise.

This approach is inspired by several works on synthetic dataset generation, especially by _Textbooks Are All You Need_ [(Gunasekar et al. 2023)](https://doi.org/10.48550/arXiv.2306.11644).

## Disclaimer

* This dataset has been generated using ChatGPT 3.5, and you should check the legal status of AI-generated content in your jurisdiction before use. We cannot guarantee that it is free of IP restrictions. You should also make sure that your usage complies with the [OpenAI Terms of Use](https://openai.com/policies/terms-of-use), in so far as legally applicable.

* This dataset focuses narrowly on improving performance on the kinds of tasks described in the Human Eval benchmark. The Human Eval benchmark has limitations and does not necessarily fully represent the coding abilities of a large language model, and there is no way to guarantee that an improvement on this benchmark represents an overall improvement in programming performance. We present this data as is, without any guarantee of its usefulness in any specific context, to encourage research that might be inspired by our method.

## Synthetic exercise creation

Model distillation is the process of transferring some of the skilled performance of large models on specific classes of tasks to significantly smaller models. The purpose is to get performance comparable to the larger model, but at a fraction of the cost and at vastly quicker speed. The general outline of this strategy is described (without technical implementation details) in [Textbooks Are All You Need](https://doi.org/10.48550/arXiv.2306.11644).

Key to the distillation process is the creation of synthetic data, generated by the larger AI model, to train the smaller model. We have applied this approach to Python programming tasks and are publishing a summary of our methods here along with the synthetic dataset.

For fuller details and implementation code, see the [related GitHub repository](https://github.com/jina-ai/textbook).

### Diversity

The main problem with model-generated synthetic data is its diversity. If we had constructed this dataset by giving ChatGPT 3.5 the same prompt several hundred thousand times, we would get many very similar, if not functionally identical, results. This would reduce the usefulness of the dataset for training. In principle, one might solve the problem by filtering the results for near duplicates, but this is a non-trivial problem, and even if it could be solved, it would be a wasteful and potentially expensive use of the larger model.

And even then, we could not be sure the examples adequately covered the topic. To solve this problem, we introduced a novel scheme for systematically prompting large language models to produce diverse examples.

### Using a topic tree to build diverse prompts

We constructed a hierarchical model of subjects in Python programming, i.e. a topic tree. First, we manually identified 42 general topic areas in Python knowledge, for example, _data structures_ and _sorting algorithms_. We asked an LLM to propose 10 subtopics for each, and then for each of those 420 fine-grained topics, we asked the LLM to generate 5 even more fine-grained sub-subtopics. This resulted in roughly 2000 very fine-grained topics.

We generated prompts by randomly selecting two of those roughly two thousand topics and combining them:

```
Create a code completion exercise on the intersection of {topic 1} and {topic 2}.
```

To increase randomness and diversity in the results, we also constructed a list of 40 professions, like _economist_, _engineer_, and _social worker_, and added them to the prompt:

```
Create a code completion exercise on the intersection of {topic 1} and {topic 2}.
Write it for a {profession}. 
```

In principle, there are approximately two million possible pairs of topics, and with 40 possible professions, this yields 80 million unique prompts. If the response to each prompt averages 100 tokens, this means our method can generate an 8 billion token synthetic dataset while maintaining a high degree of diversity. The roughly 120,000 published here is a small random subset of what is possible.

## Credits

This dataset was developed at [Jina.ai](https://jina.ai/)