GERNERMED++: Transfer Learning in German Medical NLP

We present a statistical model for German medical natural language processing trained for named entity recognition (NER) as an open, publicly available model. The work serves as a refined successor to our first GERNERMED model which is substantially outperformed by our work. We demonstrate the effectiveness of combining multiple techniques in order to achieve strong results in entity recognition performance by the means of transfer-learning on pretrained deep language models (LM), word-alignment and neural machine translation. Due to the sparse situation on open, public medical entity recognition models for German texts, this work offers benefits to the German research community on medical NLP as a baseline model. Since our model is based on public English data, its weights are provided without legal restrictions on usage and distribution. The sample code and the statistical model is available at: https://github.com/frankkramer-lab/GERNERMED-pp


Introduction
Extraction and processing of key information from medical notes and doctors' letters poses a common challenge in advanced digitization of health care systems. In particular, research-oriented data mining of non-research-centric data sources (often referred to as second use) often requires expensive data harmonization processes in order to transform unstructured or semi-structured data into strictly structured, uniform data representations such as HL7 or FHIR. While manually solving these processes can be carried out for document analysis on certain studies, it is rendered impractical for large scale text analysis on legacy data or processing day-to-day clinical data.
Handling heterogeneous data from text-based documents is a central subject of natural language processing. In recent years deep learning-inspired approaches have been applied successfully to tackle various NLP tasks effectively. However, training deep language models requires proper datasets in regards to aspects like corpus size, annotation work, data diversity and overall dataset quality, in order to retrieve well-performing models. In medical NLP, obtaining such annotated datasets remains rather difficult because the use and publication of medical data is highly restricted for the reasons of privacy and countrydependent data protection legislation. Even though medical datasets have been published in English, such datasets for German texts in contrast are still frequently unavailable to external researchers.
In this paper, we propose an approach of combining multiple ideas to obtain a German medical NLP model: • Translation: The state of German medical corpora is limited and the use of internal datasets for training and publication of models is legally disputed in regards to the fact that the possibility of privacy-concerned training data extraction cannot be excluded. In contrast, medical datasets in English have already been published and therefore, neural machine translation (NMT) can be applied to obtain German data from English datasets.
• Annotation Projection: Annotation of large corpora is crucial for supervised learning and determines the quality of the final performance of the model. However the cost of obtaining gold-standard annotations from scratch opposes the practical feasibility. Given our set of NMT-based German data, word alignment estimation can be used to project token-level annotations from English data to German data without manual intervention.
• Transfer-Learning through Model Fine-Tuning: To further improve the downstream performance of the NLP model under the constraints of our small, task-specific dataset, a larger, pre-trained German LM is used for advanced semantic, context-aware feature extraction and further finetuning.

Related Work
In the recent decade, in particular in the last five years, the field of natural language processing has been radically transformed by the use of data-driven, neural methods that are able to surpass previous state-of-the-art performances. The introduction of the attention-based transformer model [36] in the field of NLP lead to various follow-up works such as BERT [7] and similar deep language models that are trained and applied on domain-specific contexts [28,30,22,1,2,23]. All these domain-specific works works share in common that their research focus lies primarily on English application and use. The training of novel transformer-based German NLP models requires proper datasets with respect to size and quality. In purely supervised scenarios, this also includes the need for gold-standard annotation labels. While several works with internal datasets exist, their datasets are not shared among the research community and remain undisclosed [37,10,4,35,24,20,32,6,19,13,25,21,34], and thus this presents major hurdles for open research and independent reproducibility. The situation on public, English datasets is more convenient and several large datasets like MIMIC-III [29] or the i2b2 challenges with datasets such as the n2c2 2018 dataset [14] have been published, as well as the multilingual Mantra GSC [18] dataset from the biomedical domain. Only in recent few years has the German medical NLP research community addressed this issue and developed novel German medical datasets that are publicly accessible as foundation for future NLP work. [3,17].
With regards to novel German medical NLP systems, commercial software like Averbis Health Discovery [12] 1 and German Spark NLP for Healthcare [15] 2 are proprietary and require licenses. As an exception, mEx [31] is freely available, but the model weights must be requested and legal restrictions on the models' usage are imposed. For German medical NER tasks, no public and open neural model is available to the best of our knowledge, except for GERMERMED [11].

Dataset Acquisition
The dataset retrieval pipeline for German texts follows the approach proposed in GERNERMED [11]: As a starting point, the 2018 n2c2 shared task on ADE and medication extraction in EHR dataset serves as English source dataset of medical entities from anonymized electronic health records. The source dataset is decomposed into sentences as initial preprocessing step. During that process, text spans that have been replaced with an anonymized identifier text bracket by the editors of the source dataset are detected and replaced with randomized synthetic data from the Faker Python module in order to reduce the number of irregular text occurrences while updating the initial annotation span indices accordingly. For instance, this includes text entities like first and family name, dates and postal addresses.
We apply the publicly available FAIRseq transformer.wmt19.en-de [27] NMT model for sentence-wise automatic translation, which features a transformerbased neural model for translating sentences from English to German. Since the annotation information from the source dataset cannot be directly preserved for German sentences, the reconstruction of the annotation spans for the translated sentences can be estimated by the means of bitext word alignment as a postprocessing step. In contrast to the approach in GERNERMED, we refine the word alignment estimation step in regards to the following aspects: • Improved Tokenization: The tokenization of sentences for word alignment differs from modern tokenizers that generate sub-word-level tokens optimized through techniques such as byte pair encoding schemes. Most word alignment methods operate on word-level tokenization with whitespacebased token splitting. In order to reduce the number of misaligned words, we further refined the word-level tokenization by separating punctuation from words instead of only relying on tokenization splits on whitespace characters.
• Word Alignment Technique: In NLP Bitext word alignment is the task of determining the semantic correspondence between words from a bilingual sentence pair consisting of the source and translated sentence.
In previous work, the Fast Align [9] implementation has been used for establishing such correspondences. It uses the IBM 2 alignment model for alignment estimation in a purely unsupervised fashion. While there are also other models inspired from statistical machine translation [26,38], recent work has been done towards neural approaches [16,8]. For this work, we use the pre-trained model from Awesome-Align [8]. In short, the model tackles the task by encoding both sentences through a pretrained crosslingual language model in order to obtain contextualized word vector embeddings. Although the words of the sentence pairs largely differ with respect to their syntactic and linguistic features, the implementation makes use of the assumption that corresponding words are similar in terms of their word vectors in embedding space in order to find the word correlations in each sentence.
After the translation of the sentences, applying the word alignment estimation on the set of sentence pairs given the refinements for tokenizer and word alignment yields essential information on the relationship between the annotation spans of the English entity labels and their German counterparts. As a minor disadvantage of the common Pharaoh alignment format, the difference in annotation granularity cannot be preserved completely on character level. Even though the annotation spans of the source dataset are provided as character-level indices, the word-level tokenization restricts the ability to reconstruct sub-wordlevel annotation spans in the German target data when the backprojection of the word-level indices from the word alignment estimation onto the character-level indices of the target sentence text string is evaluated.

Entity Recognition Training
The training of our entity recognition model employs the entity recognition parser from the SpaCy library which follows an transducer-based parsing approach with an BILOU-scheme (Begin,Inside,Last,Outside,Unit ) instead of a state-agnostic token tagging approach.
Slim model: Without the use of a transfer-learning-based approach, in SpaCy the transformation from discrete tokens into a dense vector representation is implemented by a model that is usually trained from scratch. Such model includes the embedding of the tokens into vectors via Bloom embeddings and further uses convolutional and dense layers to establish context-awareness and feature abstraction.
Transfer-learning: Inspired by the success of transformer-based neural networks and their effectiveness on language modeling through pre-training on large scale text corpora, transfer-learning-based methods using deep transformer models can also contribute to stronger entity recognition performance by providing contextualized token embeddings through earlier pre-training without the need to train such large models from scratch. As one instance, the masked language model BERT and several descendants have been released with pre-trained weights for various different languages including German, making it well-suited for transfer-learning.
Entity Parsing: The entity parsing is trained on top of the token vectors from a slim model or a pre-trained language model. First, an additional feature vector per token is predicted by a dense layer. During document parsing in NER, each token is processed depending on the parsing state. The state vector at a certain position is composed of the indices of the current token, the first token of the last entity and the previous token. The state-specific feature vector is compiled by adding up the token-specific feature vectors from the indices of the state vector. Given this feature vector, the next action can be estimated by a dense layer while invalid actions are masked by setting their probabilities to zero. The action determines the annotation of the current token and the next state when parsing the next token.

Dataset Acquisition
The source dataset from the 2018 n2c2 shared task on ADE and medication extraction in EHR consists of 404 annotated text documents. The annotation includes the labels Strength, Form, Dosage, Route, Frequency, Drug, Duration, Reason, ADE. The documents are split into sentences using the SpaCy sentencizer for English texts. After translation and word alignment estimation we obtain our raw dataset of 17938 German sentences. The annotation distribution of the raw dataset is shown in table 1. For further clean-up of the raw dataset, sentences that do not contain any entity label are discarded from the set of sentences resulting in a total of 16632 sentence samples. Due to the nature of the labels ADE and Reason of being rather weakly defined and not of broad use for general applicability, we decided to drop these annotation labels from the dataset. As additional clean-up step, the annotation label Route was removed as well since we concluded that the label samples lack of textual diversity after a swift investigation on the actual label occurrences.

Entity Recognition Training
Before the entity recognition model is trained, we split the previously described, filtered dataset into training, validation and test set (80%,10%,10%). The split statistics is provided in table 2. Since the IOB-based entity recognition parser requires the annotated dataset to contain only non-overlapping annotation spans, annotation overlaps are resolved by removing the annotation span of shorter length while only preserving the longest span.  We investigate the ability of improving the entity recognition performance by the means of transfer-learning on deep language models on the basis of two German models: • German BERT [5] 3 (bert-base-german-cased ): The model from Deepset AI follows the default architecture of BERT and has been specifically pretrained on German data. The pre-training dataset stems from German Wikipedia, OpenLegalData, and German news articles.
• GottBERT [33]: The model is based on the RoBERTa architecture and has been trained on the OSCAR dataset using the fairseq implementation. OSCAR is a German subset of CommonCrawl.
Both language models are publicly available. We retrieved both models from the Huggingface platform. For fine-tuning the entity recognizer on top of the language model, we utilize SpaCy for training. In this context, the modelspecific tokenizer is inherited from the language model.
The training was performed on a single Nvidia Titan RTX. The training took 8-47 minutes (German BERT : 47m, GottBERT : 26m, Slim: 8m). In order to measure the differences in performance scores, we also compare the SpaCy Slim model using the same training and test set as baseline reference model, as well as the publicly available GERNERMED model as static model evaluated on the test set. It should be noted that the GERNERMED model scores must be considered as tainted because its weights are trained on a dataset that might partially contain samples from our test set. For the evaluation, the NER procedure is considered as a token-wise multi-class classification problem. We computed the precision (Pr ), recall (Re) and F1-score (F1 ) for each individual label class as well as its respective weighted average score (Total ). The final results on the test set are depicted in table 3.

Scores on Test Set
NER Tags   Both transfer-learning-based approaches exhibit strong performance in absolute numbers. Though to our surprise, German BERT achieves significantly inferior performance scores in direct comparison to GottBERT. We attribute this performance gap to the differences in pre-training dataset sizes for German BERT (12GB) and GottBERT (145GB).

Out-of-Distribution Evaluation
The evaluation on the test set does not provide valuable information on how a model can maintain its scores beyond the scope of the train and test set. A known property of neural networks as statistical models is their ability to overfit to the training dataset. While strong performance on the test set indicates the ability to abstract from individual samples without blunt sample memorization, it cannot measure the model's reliance on the inherent bias of the dataset and its ability to generalize to out-of-distribution(OoD) samples. To investigate the OoD generalization ability, we retrieved 30 text samples provided from independent physicians annotated with equivalent labels to our dataset and evaluated the models' performance on this separated dataset. The results are shown in table 4.

Scores on OoD Dataset
NER Tags   The results highlight the impact of the transfer-learning-based NER models in order to preserve strong performance on OoD data samples. However similar to the results from the test set, German BERT performs inferior to the GottBERT-based model by an increased margin according to the weighted F1score. In contrast, the basic reference models suffer from substantially degraded scores in comparison to their scores on the test set.
Due to the sparseness and independent origin of the OoD dataset, the number of labels is imbalanced across individual class labels and explains that the evaluation scores can yield 1.0 or 0.0 in several situations.

Discussion
Considering the current poor availability of open medical NLP systems for non-English natural language as well as for German in particular, our refined approach demonstrates a powerful opportunity to build a strong medical NER model solely on the use of a public English dataset. Another notable factor of the use of public datasets is its inherent immunity against privacy threats from adversarial training data extraction, which allows us to publish our resulting models for third party usage. In contrast, NLP systems trained on internal data from proprietary sources like local clinics are often kept unavailable due to legal reasons. However, our approach does not solve the need for languagespecific datasets in general since it is required for measuring the effects of biases and artifacts introduced by the translation and alignment process. Beyond this, more diverse datasets are beneficial in order to estimate the out-of-distribution generalization capabilities and intra-medical domain shifts of the NER model more accurately.
With regards to the entity recognition performance on the test set and on the out-of-distribution dataset in particular, the substantial gains in scores by using a pre-trained deep language model indicate the usefulness of transfer-learning for small datasets in comparison to basic NER models trained from scratch. However, using a general pre-trained model for transfer-learning imposes practical limitations like inefficient tokenization in case of a large domain-specific vocabulary or when re-training is too expensive. Training a medical-specific German language model requires sufficient amount of data and is considered as future work.
Despite of the fact that the performance of different (future) models could be compared in theory, a fair comparison requires all competing models to support equivalent label classes with consistent semantic definitions across all models. This poses a challenge since the underlying, independent datasets largely differ in annotation labels and label definition.