license:cc-by-nc-4.0 region:us
Data for alpaca_eval, which aims to help automatic evaluation of instruction-following models
@misc{alpaca_eval,
  author = {Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {AlpacaEval: An Automatic Evaluator of Instruction-following Models},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\\url{https://github.com/tatsu-lab/alpaca_eval}}
}
---
license: cc-by-nc-4.0



---

### Usage 

```
load_dataset("krenerd/alpaca_eval_multilingual", "alpaca_eval") # or alpaca_eval_en
load_dataset("krenerd/alpaca_eval_multilingual", "alpaca_eval_ko")
load_dataset("krenerd/alpaca_eval_multilingual", "alpaca_eval_ja")
```


### Method

The dataset was translated by GPT-4 API using the following prompt. 

```
ja = ChatPromptTemplate.from_messages(
    [
        SystemMessagePromptTemplate.from_template(
            "You are a helpful assistant fluent in English and Japanese."
        ),
        HumanMessagePromptTemplate.from_template(
            "Translate the following text to Japanese. Show the answer only. このテキストを直訳するのではなく、その意味を保持しつつ、より自然なリクエストに言い換えて翻訳してください text=```{instruction}```"
        ),
    ]
)

ko = ChatPromptTemplate.from_messages(
    [
        SystemMessagePromptTemplate.from_template(
            "You are a helpful assistant fluent in English and Korean."
        ),
        HumanMessagePromptTemplate.from_template(
            "Translate the following text to Korean. Show the answer only. 말 그대로 번역하지 말고, 의미가 유지되는 한에서 자연스러운 요청으로 번역해줘. text=```{instruction}```"
        ),
    ]
)
```


Script: https://gist.github.com/sieu-n/88542733914f80f780359f5c82c99a62