Papers
arxiv:2401.01943

Generalist embedding models are better at short-context clinical semantic search than specialized embedding models

Published on Jan 3
Authors:
,
,
,
,

Abstract

The increasing use of tools and solutions based on Large Language Models (LLMs) for various tasks in the medical domain has become a prominent trend. Their use in this highly critical and sensitive domain has thus raised important questions about their robustness, especially in response to variations in input, and the reliability of the generated outputs. This study addresses these questions by constructing a textual dataset based on the ICD-10-CM code descriptions, widely used in US hospitals and containing many clinical terms, and their easily reproducible rephrasing. We then benchmarked existing embedding models, either generalist or specialized in the clinical domain, in a semantic search task where the goal was to correctly match the rephrased text to the original description. Our results showed that generalist models performed better than clinical models, suggesting that existing clinical specialized models are more sensitive to small changes in input that confuse them. The highlighted problem of specialized models may be due to the fact that they have not been trained on sufficient data, and in particular on datasets that are not diverse enough to have a reliable global language understanding, which is still necessary for accurate handling of medical documents.

Community

Really cool to see this kind of evaluation! cc @katielink @tomaarsen

Such an analysis would at least need to include biomedical models which are trained on semantic similarity (surch as BioLORD) or search tasks (such as MedCPT).

@FremyCompany there is an open issue on the papers GitHub repo about extending the eval to MedCPT https://github.com/kaduceo/ICD10CM_Embedding_Benchmark/issues/1. Agree this would be a more valuable comparison.

Arent most of the medical models compared, models that havent been trained for semantic search? If you dont give a model the right objective and training I wouldnt expect it to do well on semantic search.

@davanstrien did I miss something obvious?

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2401.01943 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2401.01943 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2401.01943 in a Space README.md to link it from this page.

Collections including this paper 1