MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data
Numerical reasoning over hybrid data containing both textual and tabular
content (e.g., financial reports) has recently attracted much attention in the
NLP community. However, existing question answering (QA) benchmarks over hybrid
data only include a single flat table in each document and thus lack examples
of multi-step numerical reasoning across multiple hierarchical tables. To
facilitate data analytical progress, we construct a new large-scale benchmark,
MultiHiertt, with QA pairs over Multi Hierarchical Tabular and Textual data.
MultiHiertt is built from a wealth of financial reports and has the following
unique characteristics: 1) each document contain multiple tables and longer
unstructured texts; 2) most of tables contained are hierarchical; 3) the
reasoning process required for each question is more complex and challenging
than existing benchmarks; and 4) fine-grained annotations of reasoning
processes and supporting facts are provided to reveal complex numerical
reasoning. We further introduce a novel QA model termed MT2Net, which first
applies facts retrieving to extract relevant supporting facts from both tables
and text and then uses a reasoning module to perform symbolic reasoning over
retrieved facts. We conduct comprehensive experiments on various baselines. The
experimental results show that MultiHiertt presents a strong challenge for
existing baselines whose results lag far behind the performance of human
experts. The dataset and code are publicly available at
https://github.com/psunlpgroup/MultiHiertt.