Autoencoding. The generator networks in the Autoencoding task follow an encoder-decoder structure in Pix2Pix [1]}, where the encoders are the searched backbones and the decoders contain 14 layers of convolution and deconvolution. We train the generator network using conditional GAN [2]} with a discriminator containing 7 convolution layers. We apply spectral normalization [3]} to stabilize the discriminator. The generator is trained with the L1 loss with weight 0.99 and GAN loss with weight 0.01. We use structural similarity index measure (SSIM) [4]} as the metric for network performance evaluation. The data augmentations applied for the generator are random flip and color jittering. Both the generator and discriminator use Adam [5]} optimizer to stabilize training with an initial learning rate of 0.0005. Similar to the settings in Taskonomy, each network is trained for 30 epochs.
