task_categories:multiple-choice language:ko license:cc-by-nc-nd-4.0 arxiv:2309.02706 region:us
    HAE-RAE Bench
    @article{son2023hae,
  title={HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models},
  author={Son, Guijin and Lee, Hanwool and Kim, Suwan and Lee, Jaecheol and Yeom, Je Won and Jung, Jihyu and Kim, Jung Woo and Kim, Songseong},
  journal={arXiv preprint arXiv:2309.02706},
  year={2023}
}
---
license: cc-by-nc-nd-4.0
extra_gated_prompt: >-
  To request access to the dataset, please fill out this form, and we'll review
  and let you know if your use case is approved.
extra_gated_fields:
  First Name: text
  Last Name: text
  Institution: text
  Intended Use: text
  I agree to use this dataset for non-commercial research ONLY: checkbox
task_categories:
- multiple-choice
language:
- ko
---

HAE-RAE Bench is an evaluation suite specifically curated to challenge models that lack Korean cultural and contextual depth.
For a comprehensive overview, refer to our [paper](https://arxiv.org/abs/2309.02706).   

The HAE-RAE Bench team is constantly working to broaden its coverage and regularly introduces new tasks to the benchmark.
For detailed information on the tasks included, please refer to our release notes.

### Release Notes
__2023.09.28__:  [LM-Eval-Harness](https://github.com/EleutherAI/lm-evaluation-harness) support added for the following 8 tasks: 
Loan Words, Rare Words, Standard Nomenclature, History, General Knowledge,correct_definition_matching, date_understanding,reading_comprehension. 
Refer to the following [document](https://github.com/guijinSON/HAE-RAE-Bench.v2/blob/main/HAE_RAE_Bench_Evaluation.ipynb) to run the evaluation yourself.

__2023.09.16__: 10 tasks added, 5 from original HAE-RAE Bench(Loan Words, Rare Words, Standard Nomenclature, History, General Knowledge),
 5 new tasks (correct_definition_matching, date_understanding, lyrics_denoising, proverbs_denoising, reading_comprehension)