Memory-aligned Knowledge Graph for Clinically Accurate Radiology Image Report Generation

Automatic generating the clinically accurate radiology report from X-ray images is important but challenging. The identification of multi-grained abnormal regions in image and corresponding abnormalities is difficult for data-driven neural models. In this work, we introduce a Memory-aligned Knowledge Graph (MaKG) of clinical abnormalities to better learn the visual patterns of abnormalities and their relationships by integrating it into a deep model architecture for the report generation. We carry out extensive experiments and show that the proposed MaKG deep model can improve the clinical accuracy of the generated reports.


Introduction
Medical images are complex and hard to understand without specialized expertise. Given that the volume of radiology images is large, automatically generating the reports by the computer-aided system can alleviate the radiologists from the timeconsuming reporting task. Recently, many deep learning models are studied in the automated radiology report generation (Han et al., 2018;Xie et al., 2019;Yang et al., 2021;Chen et al., 2020).
The deep encoder-decoder architecture has been commonly adopted in the report generation, where visual features were extracted from the input medical images using a convolutional neural network and fed to a recurrent neural network to generate the report. Different from image captioning which inputs one image and output one sentence, the report has much longer length while the correctness of medical entities generated in the report is the core requirement. More than the requirement of detecting abnormalities accurately like classification, the report is expected to provide the support details of present abnormalities. Thus, generating accurate report with readable and logical descriptions by natural language generation model is the key challenge in the report generation task.
Generating correct reports is impossible if the pathology of abnormal regions and corresponding abnormalities cannot be identified at first. Most existing studies (Liu et al., 2021a;Chen et al., 2020Chen et al., , 2021 proposed the attention and memory mechanism to enhance the identification of abnormal regions. However, different status of the same abnormality may have their specifics and the correlations of these visual patterns are ignored. In addition, identifying the actual abnormalities from abnormal regions is also challenging since the complex and rare abnormalities are hard to determined without professional knowledge. To incorporate the prior medical knowledge, several research Zhang et al., 2020;Liu et al., 2021b) applied medical knowledge graph of certain abnormalities in the report generation aiming to learn the abnormality relationships. The corresponding representations, i.e., graph embedding, are computed by graph neural network given the input images. However, such representations are affected by the inner-connections of abnormalities for each input where the general characteristic of abnormalities are missing. For example, the representations of "Effusion", computed as graph embedding, are different when "Effusion" appears with or without "Atelectasis". But the general characteristic of "Effusion" over all relevant observations, e.g., density or shapes, are only determined by itself independently. This general but independent characteristic is still missed to model by existing approaches which limits the effectiveness the knowledge graph.
To alleviate the above challenges, in this work, we propose to learn the memory-aligned graph model, aiming to enhances the pathology identification and prior medical knowledge incorporation. The memory features of possible abnormal regions are first aligned by the input visual feature in an alternative manner, and concatenated with a universal memory embedding before feeding to the graph attention network to compute the graph embedding. The graph embedding are later learned by the classification and fine-tuned in the report generation. We evaluate the proposed approach using two publicly accessible datasets. The evaluation results show the effectiveness of utilizing memory-aligned knowledge graph in generating the clinically accurate radiology report.

Problem Formulation
Given the radiology image with extracted visual features as V , the model aims to generate a radiology report R = {y 1 , y 2 , ...}. We introduce a Memory-aligned Knowledge Graph (MaKG) to explore multi-grained features of the abnormalities and their relationships. The multi-grained memory featuresM are first aligned from the memory slots M by V , and concatenated with a meshed memory embedding E to learn the abnormality graph embedding G for generating radiology report R. This process can be formulated as, (1) Implementation. Following (Chen et al., 2020(Chen et al., , 2021Liu et al., 2021b), we adopt a memory slots M ∈ R M×D to record the information of abnormal regions which would indicate the potential abnormalities. The memory slots are initialized as plain learnable vectors and updated together with other modules. The M stands for the total number of the knowledge corresponding to the abnormality identification. We also adopt a E ∈ R N×D embedding to model the universal features of each abnormality. The N is equal to the number of the abnormalities. We follow (Zhang et al., 2020) to construct and initialize the abnormality knowledge graph G = (V, E); |V| = N which is a universal structure in the training. The nodes V cover the common chest abnormalities and grouped by their organ or body part appearances as edges E. The graph embedding G ∈ R N×D is computed by the graph attention network. A overview of this framework is shown in Fig. 1.

Memory-aligned Graph Embedding
To learn the visual patterns of possible abnormal regions, we apply Multi-Head Attention (MHA) (Vaswani et al., 2017) to query the responding memory features from the memory slots M . The MHA computes the associated weighted between different features which allows the abnormality-related memory features to be distilled from original M . To align different level of the alignment, we can perform the alignment attention alternatively as, where V 0 = V , M 0 = M , V ′ i and M ′ i denote i-th step aligned visual and memory features, respectively. As observed, the patterns of abnormal regions should be learned in different fine-grained ways due to their variable shapes and sizes. Thus, we follow  to repeat the alignment K times and obtain multi-grained memory features We then aggregate the multi-grained memory features aŝ To model the prior knowledge on the global characteristic of each abnormality which may not depend on the current input V , we add an meshed memory embedding E ∈ R N×D of which each row represent one particular abnormality. We compute the graph embedding G ∈ R N×D using graph attentional layer GAT(·) (Veličković et al., 2017) as, where FFN(x) = ReLU(xW ff 1 + b ff 1 )W ff 2 + b ff 2 , W ff 1 , W ff 2 ∈ R D×D and W G ∈ R M×N are learnable parameters, b ff 1 , b ff 2 are learnable bias vectors. We learn G by adding a fully-connected layer with Sigmoid activation for each node and serving it as a binary classifier. Each node embedding is used to predict the existence probability of corresponding abnormality, and the classifier is trained using weighted binary cross entropy loss. The details can be found in (Zhang et al., 2020).

Report Generation by Graph embedding
For each decoding step t, the hidden stats h t is encoded from the input word features x t by the standard encoder from Transformer, where w t and e t are the word embedding and positional embedding, respectively. A L layers Transformer decoder is employed to generate the proper report by the attending MaKG embeddings G as,  Both encoder and decoder are trained by minimizing the cross-entropy loss L gen (θ) = − T t=1 log(p t |p 1:t−1 ).

Datasets, Metrics and Settings
We use two publicly available datasets IU For report quality, we adopt the language generation metrics including BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), ROUGE (Lin, 2004) and CIDEr (Vedantam et al., 2015). To measure the clinical accuracy, we adopt the Clinical Efficacy (CE) (Chen et al., 2020) and Clinical Metrics (CM) (Miura et al., 2021) for common and critical observation accuracy, and MIRQI (Zhang et al., 2020) to evaluate accuracy of 14 1 observations and their associated attributes. The micro-avg F1-measure scores are reports.
To compare with the proposed model TRANS.+MAKG, we employ the basic vanilla We adopt DenseNet121 (Huang et al., 2017) to extract the visual features. The dimensions of hidden state and number of heads in MHA are set as 512 and 8. K and M are set as 3 and 20. The model is trained with the learning rate 5e-5 in the end-to-end manner.

Results on Multi-label Classification
For performance comparisons on image classification, we evaluate the proposed MAKG with the base DENSENET (Huang et al., 2017) and integrating with KG (Zhang et al., 2020) embedded with different graph neural network. Higher or equivalent scores are obtained for most of the classes as shown in Table 1. A possible explanation is that the alignment mechanism of MAKG enhances the learning of the abnormality patterns by distilling the irrelevant regions from the images.

Results on Report Generation
The main focus of this experiment is to evaluate the effectiveness of applying memory alignment knowledge graph (MaKG) in enhancing the clinical accuracy of the report generation.  Clinical Accuray Metric As shown in Table 3, TRANS.+MAKG achieves the first and second best performances over all clinical accuracy related metrics, and outperforms TRANS+KG with significantly improvement in MIRQI score which evaluates the accuracy of both abnormalities and their associated attributes. It indicates integrating MaKG is able to enhance the generation of clinically accurate report by providing correct attribute descriptions in the fine-grained level. This observation is important because the correctness of the associated attributes is necessary for the correctness of the abnormality descriptions. The incomplete or incorrect attributes of the same abnormalities would result different or even incorrect follow-up treatments. Noted that TRANS.+MAKG does not obtain the first best score in CE which measures the accuracy of 13 clinical observations and normality observation. However, the best scores of CM and Hits are observed shows that TRANS.+MAKG is able to identify the most critical abnormalities and cover most of the abnormalities that are frequently mentioned in the report repositories. As observed from Table. 3, no model could detect all evaluated abnormalities for IU XRay dataset. Thus, we further study the detailed results as shown in Table. 2. As observed, there are some abnormalities of which appearance ratio is around 5% in the whole training set which is relatively rare.
The failed detection could be caused by different reasons, such too few training data (e.g., "Fracture") or too hard to learning (e.g., "Pneumothorax" which is also very hard for clinicians to determine).

Class (%)
Integration  Natural Language Generation Metrics As the experimental results show, the higher NLG scores do not always indicate the clinically accurate reports are generated. While the clinical accuracy is a mission-critical requirement for radiology report generation, the generated report is expected to be clinically accurate using relatively readable sentences. The TRANS.+MAKG achieves similar NLG scores which indicates that the integration of MaKG is able to generate more reasonable descriptions of the abnormalities without decreasing the informativeness from TRANS. much. More powerful decoders (e.g., MemroyTrans. (Chen et al., 2020) or AlignTrans. ) should be able to enhance the overall performances.
Qualitative Results As shown in Fig. 2, two cases of ground truth and generated reports are visualized. The extracted clinical findings and the associated modifications are also attached. As observed, TRANS.+MAKG is able to detect more correct abnormalities in such cases than TRANS.+KG. It is believed to assistant clinicians to detect the abnormalities which are easy to ignored, thus increases the usability of applying the MaKG in improving the clinical accuracy in the report generation task.  (14) IU XRay WORDSAT (Xu et al., 2015) 0.262 0.383 0.369 0.317 0.094 0.215 0.463 5.6 ADAATTN (Lu et al., 2017) 0.269 0.379 0.367 0.358 0.240 0.338 0.474 6.6 SENTSAT (Krause et al., 2017) 0.274 0.372 0.365 0.318 0.106 0.241 0.451 4.8 COATTN (Jing et al., 2018) 0.256 0.367 0.357 0.307 0.061 0.245 0.438 5.2 SENTKG (Zhang et al., 2020) 0  Table 3: Performance comparison of report generation models. The best scores are in bold face and the second best are underlined."B.", "M." "R." and "C." stand for BLEU, METEOR, ROUGE and CIDEr scores, respectively. The maximum number of "Hits" is 14 which is defined by CheXpert labeling toolkit.
The heart and mediastinal contours are stable. Aorta is calcified and tortuous, compatible with atherosclerotic disease. Since the prior study, there's been interval development of left lower lobe airspace disease. The right lung is clear. 1. Interval development of left lower lobe airspace disease. This may be due to atelectasis or infiltrate.
The heart size is moderately enlarged. There is evidence of previous aortic valve replacement. XXXX sternotomy XXXX are grossly intact. The pulmonary XXXX and mediastinum are within normal limits. There is no pleural effusion or pneumothorax. There are chronically increased interstitial lung markings without superimposed focal airspace disease identified. There are degenerative changes of the spine. Cardiomegaly without superimposed acute disease noted.

Ground Truth
Trans.+KG

Conclusions
In this work, we propose a memory-aligned knowledge graph (MaKG) to enhance the clinically accurate report generation by modeling the relationship between abnormal regions and particular abnormalities. The experiments prove the effectiveness of integrating MaKG with the generation model is able to generate descriptive report with both correct abnormalities and associated attributes. In ad-dition, the proposed MaKG is not limited to the specific knowledge graph structure which give the opportunities on incorporating different professional knowledge for specific medical applications.