We aim to use radiology reports to provide weak supervision for learning transferable radiograph representations. Compared to labels or segmentation annotations, radiology reports provide semantically dense information about the contents of a radiograph. Not only do reports note presence or absence of phenomena, but they also describe inter-relatedness between conditions and provide supporting evidence for diagnoses. The recent work of Desai et al. [1]} exploits the semantic density of text, showing that transferable representations of images can be learned from corresponding textual captions by jointly training an image encoder and text decoder. We adapt the VirTex model to the radiology domain, training an encoder-decoder architecture on MIMIC-CXR v2.0 X-rays and radiology reports [2]}. TieNet [3]} similarly applied image-captioning to the radiology domain, but in contrast, we train our model from scratch, use transformer decoders for captioning, and explore efficiency of learned representations.
