It is observed that the data-driven RNNs designed in [1]} and [2]} could be easily misled by the rare and diverse abnormalities, disabling them from efficiently modeling the abnormal sentences.
Different from them, our idea is mainly inspired by radiologists’ working patterns [3]}, to explore and distill the posterior and prior knowledge for accurate radiology report generation, which is missing in their approaches.
For the network structure, we first explore the posterior knowledge of input radiology image by proposing to explicitly extract the abnormal regions; Next, inspired by [2]} and [5]} which proved the effectiveness of retrieval module and medical knowledge graph, we leverage the retrieved reports and medical knowledge graph to model the prior working experience and prior medical knowledge. In particular, instead of only retrieving some sentences in previous works [2]}, [3]}, we propose to retrieve a large amount of similar reports. Besides, since the templates may change over time, which was ignored in [2]}, using fixed templates will introduce inevitable errors. As a result, we treat the retrieved reports as latent guidance. In all, we combine the merits of retrieval module and knowledge graph in a single model. Finally, by distilling the useful prior and posterior knowledge, our approach could generate accurate reports.
<FIGURE>