For MNIST ([1]}), we use a one-hidden layer MLP with 4096 neurons in the hidden layer trained with a 0.06 learning rate on 200 epochs and no momentum, except for Figure REF , in which we switch to a learning rate of 0.01 for clearer result.
For CIFAR-10 ([2]}), we use a VGG-11 network ([3]}) without batch-norm, and it is trained on 200 epochs, with the learning rate starts at 0.01 (with 0.9 momentum) and divided by 10 at epoch 100 and epoch 150 respectively. Additionally, spatial turn-over dropout is applied to VGG-11 on its convolutional layers. The batch size is 256 for training and all the models use the ReLU activation.
