Image Encoders & Pre-processing We carry out our vision experiments using CNN-based classifiers, i.e. the LeNet architecture [1]} when working with \(x\) -MNIST data and the ResNet18 architecture [2]} when working with CIFAR10/100/SVHN data. With the \(x\) -MNIST data, we normalize the inputs using the mean and standard deviation calculated from the training set for both training and inference, as is standard. With the CIFAR10/100/SVHN data, we apply the normalization transform along with augmentation strategies (random crop and horizontal flips) during training and only normalization during inference. \(x\) -MNIST data have dimensionality \(28 \times 28 = 784\)  while CIFAR-like data have dimensionality \(32 \times 32 \times 3 = 3072\) .
