license:mit region:us
---
license: mit
---
# Massive Multitask Language Understanding (MMLU) in German

This dataset is to be used for the evaluation of LLM German language understanding. 
It is based on the hendrycksTest dataset ([here](https://huggingface.co/datasets/cais/mmlu) and [here](https://huggingface.co/datasets/tasksource/mmlu)) and was created
by using the GPT-3.5 API to translate the entire test set and a few examples of the validation set. To make sure the answer options follow the intended sentence structure
and are always of the correct format, GPT was prompted to output in a JSON format. This came with some complications that were later manually fixed.

The prompt used to translate a single example was the following:
```
insert prompt here @TODO
```

This translation cost a total of ~13€ including iterating on the prompt and fixing broken examples.

