task_categories:multiple-choice task_categories:question-answering size_categories:1M<n<10M language:zh license:cc-by-nc-4.0 llm Ancient Chinese Evaluation chinese region:us
The Ancient Chinese Language Understanding Evaluation (ACLUE) is an evaluation benchmark focused on ancient Chinese language comprehension. It aims to assess the performance of large-scale language models on understanding ancient Chinese.
\
---
license: cc-by-nc-4.0
task_categories:
- multiple-choice
- question-answering
language:
- zh
tags:
- llm
- Ancient Chinese
- Evaluation
- chinese
pretty_name: ACLUE
size_categories:
- 1M<n<10M
---
# Dataset Card for ACLUE

- **Homepage:** [https://github.com/isen-zhang/ACLUE](https://github.com/isen-zhang/ACLUE)
- **Repository:** [https://huggingface.co/datasets/tyouisen/aclue](https://huggingface.co/datasets/tyouisen/aclue)
- **Paper:** []()
- **Leaderboard:** [https://github.com/isen-zhang/ACLUE](https://github.com/isen-zhang/ACLUE)

### 简介 （Introduction）

Ancient Chinese Language Understanding Evaluation (ACLUE) 是一个面向古代汉语的评估基准，旨在帮助评估大型语言模型在古代汉语上的表现。

The Ancient Chinese Language Understanding Evaluation (ACLUE) is an evaluation benchmark focused on ancient Chinese language comprehension. It aims to assess the performance of large-scale language models (LLMs) on understanding ancient Chinese. 

### 数据 （Data）
该基准测试包含15个任务，涵盖了各个领域，包括词汇、句法、语义、推理和知识。我们为这15个任务提供了开发集和测试集数据，开发集中有5个问题，而测试集中则有100多个问题。我们鼓励研究人员使用ACLUE来测试和提升其模型在古代汉语语言理解方面的能力。ACLUE的任务取自人工挑选的公开资源和自动生成的古代汉语语料库。这些问题涵盖了从夏朝（公元前2070年）到明朝（公元1368年）的广泛时间范围。ACLUE对所有任务都采用了多项选择题的形式。

The benchmark comprises 15 tasks spanning various domains, including lexical, syntactic, semantic, inference, and knowledge. We provide development and test dataset for each of 15 tasks, with 5 questions in development set and 100+ quesitons in test set. We encourage researchers to use ACLUE to test and enhance their models' abilities in ancient Chinese language understanding. ACLUE's tasks are derived from a combination of manually curated questions from publicly available resources, and automatic generated questions from classical Chinese language corpora. The range of questions span from the Xia dynasty (2070 BCE) to the Ming dynasty (1368 CE). ACLUE employs a multiple-choice question format for all tasks.


### 数据实例（ Data Instances）
数据集中的每个问题都是一个包含4个选项的多项选择题，其中只有一个选项是正确答案。以下是两个示例：

Each question in the dataset is a multiple-choice questions with 4 choices and only one choice as the correct answer.  Here are two examples:

```
以下是关于{古诗词曲鉴赏}的单项选择题，请直接给出正确答案的选项。
    题目：《木兰诗--北朝民歌》唧唧复唧唧,木兰当户织。不闻机杼声,唯闻女叹息。问女何所思,问女何所忆。女亦无所思,女亦无所忆。昨夜见军帖,可汗大点兵,军书十二卷,卷卷有爷名。阿爷无大儿,木兰无长兄,愿为市鞍马,从此替爷征。东市买骏马,西市买鞍鞯,南市买辔头,北市买长鞭。旦辞爷娘去,暮宿黄河边,不闻爷娘唤女声,但闻黄河流水鸣溅溅。旦辞黄河去,暮至黑山头,不闻爷娘唤女声,但闻燕山胡骑鸣啾啾。万里赴戎机,关山度若飞。朔气传金柝,寒光照铁衣。将军百战死,壮士十年归。归来见天子,天子坐明堂。策勋十二转,赏赐百千强。可汗问所欲,木兰不用尚书郎,愿驰千里足,送儿还故乡。爷娘闻女来,出郭相扶将;阿姊闻妹来,当户理红妆;小弟闻姊来,磨刀霍霍向猪羊。开我东阁门,坐我西阁床。脱我战时袍,著我旧时裳。当窗理云鬓,对镜帖花黄。出门看火伴,火伴皆惊忙:同行十二年,不知木兰是女郎。雄兔脚扑朔,雌兔眼迷离;双兔傍地走,安能辨我是雄雌?下列对这首诗的理解和分析,不正确的一项是 ()
    A.  《木兰诗》是南北朝时期的一首长篇叙事民歌,风格刚健质朴。全诗以“木兰是女郎”来构思木兰的传奇故事,富有浪漫色彩。
    B.  “愿为市鞍马”的“市”是“市场”的意思,“万里赴戎机”的“戎机”是“战事”的意思。
    C.  木兰“不用尚书郎”而愿“还故乡”固然有对家乡的眷恋,但也有自己女儿身秘密的因素。
    D.  “朔气传金柝,寒光照铁衣”运用对偶手法,描写了木兰在边塞艰苦的军旅生活。
    答案是：B
```
```
题目：《虞美人》李煜。春花秋月何时了？往事知多少。小楼昨夜又东风，故国不堪回首月明中。雕栏玉砌应犹在，只是朱颜改。问君能有几多愁？恰似一江春水向东流。对《虞美人》的赏析,不恰当的一项是（）
    A. 词作从眼前景物入手,生发联想和想像,追怀昔日帝王生活,描摹了一幅幅鲜活的画面,隐晦地表达出叛逆之情,惹恼了宋太宗,铸成了词人悲惨结局。
    B. 词作以实虚相间的手法来绘景、抒情、达意,忽而写眼前,忽而写想像。
    C. 《虞美人》乃李煜绝笔词
    D. 《虞美人》以其形式别致给人美感愉悦。
    答案是：
```
以下列出了任务的类别、实例数量、问题平均长度以及任务的来源：

The category, number of instances, average length of the question, and origin of the tasks are provided below:

| Task                           | Total Q. | Avg. len |Task (zh)                    | Category     | Origin    |
|-------------------------------|------|------|-----------------------------------|----------|-----------|
| Named entity recognition      | 500  | 138  | 古汉语命名体识别                   | lexical  | generated |
| Polysemy resolution           | 500  | 116  | 古文单字多义                       | lexical  | generated |
| Homographic character resolution | 500 | 137  | 通假字                             | lexical  | generated |
| Sentence segmentation         | 500  | 210  | 古文断句                           | syntactic| generated |
| Couplet prediction            | 500  | 62   | 对联预测                           | semantic | generated |
| Poetry context prediction     | 500  | 77   | 古诗词上下句预测                   | semantic | generated |
| Poetry sentiment analysis    | 500  | 60   | 诗词情感分类                       | inference| generated |
| Poem quality estimation      | 406  | 118  | 古诗词质量评估                     | inference| generated |
| Ancient Chinese medical      | 211  | 38   | 医古文                             | knowledge| collected |
| Ancient Chinese literature   | 160  | 44   | 古代文学知识                       | knowledge| collected |
| Traditional Chinese culture  | 136  | 59   | 国学常识                           | knowledge| collected |
| Poetry appreciation          | 103  | 258  | 古诗词曲鉴赏                       | inference| collected |
| Basic ancient Chinese        | 249  | 52   | 基础古汉语知识                     | knowledge| collected |
| Reading comprehension        | 101  | 982  | 古文阅读理解                       | inference| collected |
| Ancient Chinese phonetics    | 101  | 50   | 古音学                             | knowledge| collected |

#### 加载数据 （Load data）

```python
task_list = ['polysemy_resolution',
     'poetry_sentiment_analysis',
     'named_entity_recognition',
     'basic_ancient_chinese',
     'poetry_context_prediction',
     'sentence_segmentation',
     'couplet_prediction',
     'poetry_appreciate',
     'ancient_chinese_culture',
     'ancient_phonetics',
     'homographic_character_resolution',
     'ancient_literature',
     'ancient_medical',
     'poetry_quality_assessment',
     'reading_comprehension']

from datasets import load_dataset
dataset = {k: load_dataset(r"tyouisen/aclue", k) for k in task_list}

# Print an example:
print(dataset['polysemy_resolution']['test'][0])

```


### 引用 （Citation）
```
```

### 许可证 （License）
ACLUE数据集采用：(The ACLUE dataset is licensed under a：)

[Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).

