Pretraining was done separately for Pathology9 and EdemaSeverity to ensure that the test sets were not seen during pretraining. We pretrain using the same hyperparameters as Desai et al. [1]}, including SGD, LookAhead, and Weight decay. However, we train with a batch size of 128 images across 2 NVIDIA Volta V100 GPUs and do not use an early stopping criterion, but instead pretrain for a full 50 epochs. For pretraining and all downstream training, we use a linear learning rate warmup [2]} for 5% of epochs followed by cosine decay for the remaining 95% of training. We pretrain our models and train all downstream models five times to estimate uncertainty in performance.
<TABLE>