The task of image captioning [1]}, [2]} has received extensive research interests. These approaches mainly adopt the encoder-decoder framework which translates the image to a single descriptive sentence. Such framework have achieved great success in advancing the state-of-the-arts [3]}, [4]}, [5]}, [6]}, [7]}, [8]}. However, rather than only generating one single sentence, radiology report generation aims to generate a long paragraph, which consists of multiple structural sentences with each one focusing on a specific medical observation for a specific region in the radiology image.
