Beyond the traditional image captioning task, image paragraph generation that produces a long and semantic-coherent paragraph to describe the input image has recently attracted increasing research interests [1]}, [2]}, [3]}. To this end, a hierarchical recurrent network (HRNN) [1]}, [2]} is proposed. In particular, the HRNN uses a two-level RNN model to generate the paragraph based on the image features extracted by a CNN. The two-level RNN includes a paragraph RNN and a sentence RNN, where the paragraph RNN is used to generate topic vectors and each topic vector is used by the sentence RNN to produce a sentence to describe the image. However, the correctness of generating abnormalities should be emphasized more than other normalities in a radiology report, while in a natural image paragraph each sentence has equal importance.
