Demonstrating alternatives to supervised pretraining, several recent studies leverage the unlabeled radiology reports for weak supervision. Clinical reports capture physicians' impressions in the form of unstructured text, yet contain rich semantic information. Prior work has capitalized on these natural annotations by using contrastive learning to encourage similar feature representations between image and text [1]}, [2]}, [3]}. While these methods attempt to learn representations from images and text jointly, we take a different approach, learning representations by predicting text from images. The report generation task is an active area of research [4]}, [5]}, [6]}, and Wang et al. [7]} even demonstrated that pretrained LSTM captioning models improve downstream classification of medical images, but efficiency of learned representations for transfer is yet unknown.
