Our model, RadTex, closely mirrors the architecture of VirTex with two main components: a visual backbone and textual head (encoder and decoder, respectively). Augmented X-rays are input to a visual backbone, consisting of a ResNet-50 convolutional network and a linear projection layer, which flattens embeddings to allow for decoder attention over visual features. Corresponding radiology reports are normalized, tokenized, and embedded using a learned token and positional embedding. After element-wise sum, layer normalization and dropout, these vector embeddings are passed to the textual head, which uses transformer decoders [1]} to predict masked tokens. The decoders use the context of visual features along with prior token predictions to predict the next token in a sequence. Like VirTex, we use bidirectional prediction of tokens during training. An overview of the RadTex model architecture is shown in Figure REF .
