Task-aware Retrieval with Instructions
Akari Asaiyz, Timo Schicky, Patrick Lewisy, Xilun Cheny, Gautier Izacardy,
Sebastian Rie
Task-aware Retrieval with Instructions
We study the problem of retrieval with instructions, where users of a
retrieval system explicitly describe their intent along with their queries. We
aim to develop a general-purpose task-aware retrieval system using multi-task
instruction tuning, which can follow human-written instructions to find the
best documents for a given query. We introduce the first large-scale collection
of approximately 40 retrieval datasets with instructions, BERRI, and present
TART, a multi-task retrieval system trained on BERRI with instructions. TART
shows strong capabilities to adapt to a new retrieval task via instructions and
advances the state of the art on two zero-shot retrieval benchmarks, BEIR and
LOTTE, outperforming models up to three times larger. We further introduce a
new evaluation setup, X^2-Retrieval to better reflect real-world scenarios,
where diverse domains and tasks are pooled and a system needs to find documents
aligning users' intents. In this setup, TART significantly outperforms
competitive baselines, further demonstrating the effectiveness of guiding
retrieval with instructions.
Akari Asai Timo Schick Patrick Lewis Xilun Chen Gautier Izacard Sebastian Riedel Hannaneh Hajishirzi Wen-tau Yih