@inproceedings{
mao2023wamp,
title={{WAMP}: A Competition-Level Dataset for Assessing the Mathematical Reasoning Capabilities of {LLM}s},
author={Yujun Mao and Yoon Kim and Yilun Zhou},
booktitle={The 3rd Workshop on Mathematical Reasoning and AI at NeurIPS'23},
year={2023},
url={https://openreview.net/forum?id=loDIYsNLbw}
}
Recent large language models (LLMs) have shown indications of math deduction abilities. However, it is unclear that for challenging math problems, what information about the problem helps (or hurts). In this paper, we propose a challenging benchmark dataset for such analyses. The Concept and Hint-Annotated Math Problems, or CHAMP, consists of competition-level math problems annotated with "concepts," or general math facts, and "hints," or problem-specific tricks. These entities and their interconnections allow us to explore the effects of additional information, such as relevant hints, misleading concepts, or related problems. We conduct 12 preliminary studies with 4 models, summarize our findings and discuss how CHAMP supports general discussions around LLMs' capabilities to understand and use contexts. The dataset, code and an extended version of the paper are available on the project website at https://yujunmao1.github.io/CHAMP.
CHAMP: A Competition-Level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities
Yujun Mao Yoon Kim Yilun Zhou