In recent years, automatic radiology report generation has attracted extensive research interests [1]}, [2]}, [3]}, [4]}, [5]}, [6]}. Most existing methods, like [7]}, [8]}, [9]} follow the standard image captioning approaches and employ the encoder-decoder framework, e.g., CNN-HRNN [7]}, [11]}. In the encoding stage, the image features are extracted by CNN from the entire image; In the decoding stage, the whole report is generated by HRNN. However, directly applying image captioning approaches to radiology images has the following problems: 1) Visual data deviation: the appearance of normal images dominate the dataset over that of abnormal images [12]}.
As a result, this unbalanced visual distribution would distract the model from accurately capturing the features of rare and diverse abnormal regions. 2) Textual data deviation: as shown in Figure REF , in a report, radiologists tend to describe all the items in an image, making the descriptions of normal regions dominate the entire report. Besides, many similar sentences are used to describe the same normal regions. With this unbalanced textual distribution, training with such dataset makes the generation of normal sentences dominant [4]}, [8]}, [9]}, [6]}, disabling the model to describe specific crucial abnormalities. In brief, as shown in Figure REF , the widely-used HRNN [17]} generates some repeated sentences of normalities and fails to depict some rare but important abnormalities.
