Our contributions are in four-fold: (1) we describe an integrated image interpretation framework for disease annotation and medical report generation, (2) we transfer knowledge from large image data sets (ChestX-ray8 <cite>[16]</cite> and ImageNet) to enhance medical image interpretation using a small number of reports for training (IU X-ray [2] ), (3) we evaluate suitability of the NLP evaluation metrics for medical report generation, and (4) we demonstrate the functionality of localizing the key finding in an X-ray with a heatmap.