task_categories:question-answering task_categories:text-generation size_categories:10K<n<100K language:nl license:apache-2.0 baize stack overflow stackoverflow chat doi:10.57967/hf/0529 region:us
---
license: apache-2.0
task_categories:
- question-answering
- text-generation
language:
- nl
tags:
- baize
- stack overflow
- stackoverflow
- chat
pretty_name: Stack Overflow Chat Dutch
size_categories:
- 10K<n<100K
---

# Dataset Card for Stack Overflow Chat Dutch

## Dataset Description

- **Homepage:** N/A
- **Repository:** N/A
- **Paper:** N/A
- **Leaderboard:** N/A
- **Point of Contact:** Bram Vanroy

### Dataset Summary

This dataset contains 56,964 conversations between een AI assistant and a (fake) "Human" (generated) in Dutch, specifically in the domain of programming (Stack Overflow). They are translations of [Baize](https://github.com/project-baize/baize-chatbot/tree/6790946f638d60fcaf397574189124f15792f35a/data)'s machine-generated answers to the [Stack Overflow](https://huggingface.co/datasets/pacovaldez/stackoverflow-questions) dataset. 

☕ [**Want to help me out?**](https://www.buymeacoffee.com/bramvanroy) Translating the data with the OpenAI API, and prompt testing, cost me 💸$133.60💸. If you like this dataset, please consider [buying me a coffee](https://www.buymeacoffee.com/bramvanroy) to offset a portion of this cost, I appreciate it a lot! ☕

### Languages

- Dutch

## Dataset Structure

### Data Instances

```python
 {'id': 9,
  'input': 'Het gesprek tussen de mens en de AI-assistent.\n'
           '[|Human|] Ik heb wat hulp nodig met Spark SQL union en het ordenen '
           'van kolommen.\n'
           '[|AI|] Natuurlijk, ik help je graag. Waar heb je specifiek hulp '
           'bij nodig?\n'
           '[|Human|] Ik wil de output van een UNION-query ordenen op een '
           'kolom die niet aanwezig is in de SELECT-instructie. Is dat '
           'mogelijk?\n'
           '[|AI|] Ja, het is mogelijk met Spark SQL. Je kunt de ordinale '
           'positie van de kolom in de ORDER BY-clausule gebruiken in plaats '
           'van de kolomnaam. Als je bijvoorbeeld wilt ordenen op de derde '
           'kolom in de SELECT-instructie, kun je ORDER BY 3 gebruiken.\n'
           '[|Human|] Dat is nuttig. Kan ik ook een alias gebruiken in de '
           'ORDER BY-clausule?\n'
           '[|AI|] Ja, je kunt een alias gebruiken in de ORDER BY-clausule. De '
           'alias moet worden gedefinieerd in de SELECT-instructie. Als je '
           'bijvoorbeeld de derde kolom als "col3" alias geeft, kun je ordenen '
           'met ORDER BY col3.\n'
           '[|Human|] Oké, ik denk dat ik het begrijp. Kan ik dezelfde alias '
           'gebruiken voor kolommen uit beide tabellen in de UNION?\n'
           '[|AI|] Nee, je kunt dezelfde alias niet gebruiken voor kolommen '
           'uit beide tabellen in de UNION. Elke tabel in de UNION moet een '
           'unieke alias hebben voor de kolommen.\n'
           '[|Human|] ',
  'topic': 'Spark SQL UNION - ORDER BY kolom niet in SELECT'},
```

### Data Fields

- **id**: the ID of the item. The following 82 IDs are not included because they could not be translated: `[1713, 1937, 1960, 4326, 4356, 8357, 8542, 8827, 9137, 9782, 11560, 11961, 12244, 12362, 12488, 13259, 13621, 14445, 14835, 15006, 17746, 18808, 19285, 19426, 19491, 21270, 21661, 22098, 23352, 23840, 23869, 25148, 25928, 27102, 27856, 28387, 29942, 30041, 30251, 32396, 32742, 32941, 33628, 34116, 34648, 34859, 35977, 35987, 36035, 36456, 37028, 37238, 37640, 38107, 38735, 39015, 40984, 41115, 41567, 42397, 43219, 43783, 44599, 44980, 45239, 47676, 48922, 49534, 50282, 50683, 50804, 50919, 51076, 51211, 52000, 52183, 52489, 52595, 53884, 54726, 55795, 56992]`
- **input**: the machine-generated conversation between AI and "Human". Always starts with `Het gesprek tussen de mens en de AI-assistent.\n` and has at least one occurrence of both `[|AI|]` and `[|Human|]`.
- **topic**: the topic description


## Dataset Creation

Both the translations and the topics were translated with OpenAI's API for `gpt-3.5-turbo`. `max_tokens=1024, temperature=0` as parameters.

The prompt template to translate the input is (where `src_lang` was English and `tgt_lang` Dutch):

```python
CONVERSATION_TRANSLATION_PROMPT = """You are asked to translate a conversation between an AI assistant and a human from {src_lang} into {tgt_lang}.

Here are the requirements that you should adhere to:
1. maintain the format: the conversation consists of the AI (marked as `[|AI|]`) and the human (`[|Human|]`) talking in turns and responding to each other;
2. do not translate the speaker identifiers `[|AI|]` and `[|Human|]` but always copy them into the translation in appropriate places;
3. ensure accurate translation and keep the correctness of the conversation;
4. make sure that text is fluent to read and does not contain grammatical errors. Use standard {tgt_lang} without regional bias;
5. translate the human's text using informal, but standard, language;
6. make sure to avoid biases (such as gender bias, grammatical bias, social bias);
7. if the human asks to correct grammar mistakes or spelling mistakes then you have to generate a similar mistake in {tgt_lang}, and then also generate a corrected output version for the AI in {tgt_lang};
8. if the human asks to translate text from one to another language, then you only translate the human's question to {tgt_lang} but you keep the translation that the AI provides in the language that the human requested;
9. do not translate code fragments but copy them as they are. If there are English examples, variable names or definitions in code fragments, keep them in English.

Now translate the following conversation with the requirements set out above. Do not provide an explanation and do not add anything else.\n\n"""
```

The prompt to translate the topic is:

```python
TOPIC_TRANSLATION_PROMPT = "Translate the following title of a conversation from {src_lang} to {tgt_lang} in a succinct," \
                           " summarizing manner. Translate accurately and formally. Do not provide any explanation" \
                           " about the translation and do not include the original title.\n\n"
```

The system message was:

```
You are a helpful assistant that translates English to Dutch to the requirements that are given to you.
```

Note that 82 items (0.1%) were not successfully translated. The translation was missing the AI identifier `[|AI|]` and/or the human one `[|Human|]`. The IDs for the missing items are `[1713, 1937, 1960, 4326, 4356, 8357, 8542, 8827, 9137, 9782, 11560, 11961, 12244, 12362, 12488, 13259, 13621, 14445, 14835, 15006, 17746, 18808, 19285, 19426, 19491, 21270, 21661, 22098, 23352, 23840, 23869, 25148, 25928, 27102, 27856, 28387, 29942, 30041, 30251, 32396, 32742, 32941, 33628, 34116, 34648, 34859, 35977, 35987, 36035, 36456, 37028, 37238, 37640, 38107, 38735, 39015, 40984, 41115, 41567, 42397, 43219, 43783, 44599, 44980, 45239, 47676, 48922, 49534, 50282, 50683, 50804, 50919, 51076, 51211, 52000, 52183, 52489, 52595, 53884, 54726, 55795, 56992]`.

### Source Data

#### Initial Data Collection and Normalization

Initial data generation by [Baize](https://github.com/project-baize/baize-chatbot), which in turn relied on [Stack Overflow Questions](https://huggingface.co/datasets/pacovaldez/stackoverflow-questions).

#### Who are the source language producers?

Quora users and OpenAI's GPT models (via [Baize](https://github.com/project-baize/baize-chatbot)).

## Considerations for Using the Data

Note that neither the translations in this new dataset, nor the machine-generated dialogues in the original [Baize](https://github.com/project-baize/baize-chatbot) dataset, have been verified by humans.

### Discussion of Biases

As with any machine-generated texts, users should be aware of potential biases that are included in this dataset. Although the prompt specifically includes `make sure to avoid biases (such as gender bias, grammatical bias, social bias)`, of course the impact of such command is not known. It is likely that biases remain in the dataset so use with caution.

### Other Known Limitations

The translation quality has not been verified. Use at your own risk!

### Licensing Information

Licensing info for [`Stack Overflow Questions`](https://huggingface.co/datasets/pacovaldez/stackoverflow-questions) is listed as Apache 2.0. If you use the current dataset, you should also adhere to the original license. 

This text was generated (either in part or in full) with GPT-3 (`gpt-3.5-turbo`), OpenAI’s large-scale language-generation model. Upon generating draft language, the author reviewed, edited, and revised the language to their own liking and takes ultimate responsibility for the content of this publication.

If you use this dataset, you must also follow the [Sharing](https://openai.com/policies/sharing-publication-policy) and [Usage](https://openai.com/policies/usage-policies) policies. 

As clearly stated in their [Terms of Use](https://openai.com/policies/terms-of-use), specifically 2c.iii, "[you may not] use output from the Services to develop models that compete with OpenAI". That means that you cannot use this dataset to build models that are intended to commercially compete with OpenAI. [As far as I am aware](https://law.stackexchange.com/questions/93308/licensing-material-generated-with-chatgpt), that is a specific restriction that should serve as an addendum to the current license.


### Citation Information

If you use this data set, please cite :


Vanroy, B. (2023). Stack Overflow Chat Dutch [Data set]. Hugging Face. https://doi.org/10.57967/HF/0529

```bibtex
@misc{https://doi.org/10.57967/hf/0529,
  doi = {10.57967/hf/0529},
  url = {https://huggingface.co/datasets/BramVanroy/stackoverflow-chat-dutch},
  author = {Vanroy, Bram},
  title = {{S}tack {O}verflow {C}hat {D}utch},
  publisher = {Hugging Face},
  year = {2023}
}
```


### Contributions

Thanks to [Baize](https://github.com/project-baize/baize-chatbot) for the initial machine-generated dialogues, based on [Stack Overflow Questions](https://huggingface.co/datasets/pacovaldez/stackoverflow-questions).