A-PROOF Binary Sentence Classification

Description

A fine-tuned binary text classification model that determines whether a sentence is relevant for WHO-ICF category classification.

Since 95% of the sentences in clinical notes is not relevant for ICF classification, it makes sense to filter the relevant sentences before applying other classification processes. Using the binary classification, the processing of large volumes of data can be optimised as only 5% of the sentences needs to be classified for the level of functioning. For further classification of relevant sentences, you can use the multilabel classifier: https://huggingface.co/CLTL/icf-domains and the any of the relevant regression classifiers for obtaining a level score.

Relevant sentences are likely to be express patient's functioning for the following 9 ICF categories:

ICF code	Domain	name in repo
b440	Respiration functions	ADM
b140	Attention functions	ATT
d840-d859	Work and employment	BER
b1300	Energy level	ENR
d550	Eating	ETN
d450	Walking	FAC
b455	Exercise tolerance functions	INS
b530	Weight maintenance functions	MBW
b152	Emotional functions	STM

Intended use and limitations

The model was fine-tuned (trained, validated and tested) on medical records from the Amsterdam UMC (the two academic medical centers of Amsterdam). It might perform differently on text from a different hospital or text from non-hospital sources (e.g. GP records).
The model only distinguishes sentences on the basis of the 9 ICF categories.

How to use

To generate predictions with the model, use the Transformers library:

# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline('text-classification', model='CLTL/binary_icf_classifier')
result = pipe('De patient is erg moe')
print(result)
[{'label': 'pos', 'score': 0.9977788329124451}]

# load model directly
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer. from_pretrained ('CLTL/binary_icf_classifier')
model = AutoModelForSequenceClassification.from_pretrained('CLTL/binary_icf_classifier')

Training data

The training data consists of clinical notes from medical records (in Dutch) of the Amsterdam UMC. Due to privacy constraints, the data cannot be released.
The annotation guidelines used for the project can be found here.

Evaluation results

The evaluation is done on a sentence-level (the classification unit): .97 precision, .96 recall, .97 f1.

Contact

Piek Vossen, [email protected]

References

https://github.com/cltl-students/Cecilia_Kuan_data_augmentation Cecilia Kuan, 2023, Generative Approach of Data Augmentation for Pre-Trained Clinical NLP System, MA Thesis, Vrije Universiteit Amsterdam