
ConceptMNIST: The network \(g\)  consists of 2 convolutional layers with 32 channels each, along with a maxpool layer in between followed by a fully connected layer. The network \(f\)  consists of a fully connected layer. The fine-tuning is done end-to-end with the weight 0.1 given to the concept loss. We train the model with batch size set to 64, learning rate set to 1e-4 for a total of 20 epochs.

CUB: The network \(g\)  consists of a Inception V3 pre-trained on Imagenet and the network \(f\)  consists of one layer MLP. The fine-tuning is done end-to-end with the weight 0.001 given to the concept loss. The rest of the hyperparameters are unchanged from [1]}.

OAI: For the OAI dataset, we utilize the architecture provided in [1]} which consists of a ResNet-18 with last 12 layers unfrozen. The prediction model consists of a 3 layer MLP with 50 neurons each in the first 2 layers while the last layer provides a regression score. The training hyperparameters are used unchanged.

