Deep Infomax~(DIM) is an unsupervised representation learning framework by maximizing the mutual information between the inputs and the outputs of an encoder, while probabilistic constraints are imposed on the outputs. In this paper, we propose Supervised Deep InfoMax~(SDIM), which introduces supervised probabilistic constraints to the encoder outputs. The supervised probabilistic constraints are equivalent to a generative classifier on high-level data representations, where class conditional log-likelihoods of samples can be evaluated. Unlike other works building generative classifiers with conditional generative models, SDIMs scale on complex datasets, and can achieve comparable performance with discriminative counterparts.   With SDIM, we could perform \emph{classification with rejection}.
Instead of always reporting a class label, SDIM only makes predictions when test samples' largest logits surpass some pre-chosen thresholds, otherwise they will be deemed as out of the data distributions, and be rejected.   Our experiments show that SDIM with rejection policy can effectively reject illegal inputs including out-of-distribution samples and adversarial examples. Non-robustness of neural network models emerges as a pressing concern since they are observed to be vulnerable to adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014) . Many attack methods have been developed to find imperceptible perturbations to fool the target classifiers (Moosavi-Dezfooli et al., 2016; Carlini & Wagner, 2017; Brendel et al., 2017) . Meanwhile, many defense schemes have also been proposed to improve the robustnesses of the target models (Goodfellow et al., 2014; Tramèr et al., 2017; Madry et al., 2017; Samangouei et al., 2018 ). An important fact about these works is that they focus on discriminative classifiers, which directly model the conditional probabilities of labels given samples. Another promising direction, which is almost neglected so far, is to explore robustness of generative classifiers (Ng & Jordan, 2002) . A generative classifier explicitly model conditional distributions of inputs given the class labels. During inference, it evaluates all the class conditional likelihoods of the test input, and outputs the class label corresponding to the maximum. Conditional generative models are powerful and natural choices to model the class conditional distributions, but they suffer from two big problems: (1) it is hard to scale generative classifiers on high-dimensional tasks, like natural images classification, with comparable performance to the discriminative counterparts. Though generative classifiers have shown promising results of adversarial robustness, they hardly achieve acceptable classification performance even on CIFAR10 Schott et al., 2018; Fetaya et al., 2019) . (2) The behaviors of likelihood-based generative models can be counter-intuitive and brittle. They may assign surprisingly higher likelihoods to out-of-distribution (OoD) samples (Nalisnick et al., 2018; Choi & Jang, 2018) . Fetaya et al. (2019) discuss the issues of likelihood as a metric for density modeling, which may be the reason of non-robust classification, e.g. OoD samples detection. In this paper, we propose supervised deep infomax (SDIM) by introducing supervised statistical constraints into deep infomax (DIM, Hjelm et al. (2018) ), an unsupervised learning framework by maximizing the mutual information between representations and data. SDIM is trained by optimizing two objectives: (1) maximizing the mutual information (MI) between the inputs and the high-level data representations from encoder; (2) ensuring that the representations satisfy the supervised statistical constraints. The supervised statistical constraints can be interpreted as a generative classifier on high-level data representations giving up the full generative process. Unlike full generative models making implicit manifold assumptions, the supervised statistical constraints of SDIM serve as explicit enforcement of manifold assumption: data representations (low-dimensional) are trained to form clusters corresponding to their class labels. With SDIM, we could perform classification with rejection (Nalisnick et al., 2019; Geifman & El-Yaniv, 2017) . SDIMs reject illegal inputs based on off-manifold conjecture (Samangouei et al., 2018; Gu & Rigazio, 2014) , where illegal inputs, e.g. adversarial examples, lie far away from the data manifold. Samples whose class conditionals are smaller than the pre-chosen thresholds will be deemed as off-manifold, and prediction requests on them will be rejected. The contributions of this paper are : • We propose Supervised Deep Infomax (SDIM), an end-to-end framework whose probabilistic constraints are equivalent to a generative classifier. SDIMs can achieve comparable classification performance with similar discrinimative counterparts at the cost of small over-parameterization. • We propose a simple but novel rejection policy based on off-manifold conjecture: SDIM outputs a class label only if the test sample's largest class conditional surpasses the prechosen class threshold, otherwise outputs rejection. The choice of thresholds relies only on training set, and takes no additional computations. • Experiments show that SDIM with rejection policy can effectively reject illegal inputs, including OoD samples and adversarial examples generated by a comprehensive group of adversarial attacks. We introduce supervised probabilistic constraints to DIM. Giving up the full generative process, SDIMs are equivalent to generative classifiers on high-level data representations. Unlike full conditional generative models which achieve poor classification performance even on CIFAR10, SDIMs attain comparable performance as the discriminative counterparts on complex datasets. The training of SDIM is also computationally similar to discriminative classifiers, and does not require prohibitive computational resources. Our proposed rejection policy based on off-manifold conjecture, a built-in property of SDIM, can effectively reject illegal inputs including OoD samples and adversarial examples. We demonstrate that likelihoods modeled on high-level data representations, rather than raw pixel intensities, are more robust on downstream tasks without the requirement of generating real samples.
scale generative classifiers  on complex datasets, and evaluate their effectiveness to reject illegal inputs including out-of-distribution samples and adversarial examples.
{"src": "scitldr-AIC_train", "paper_id": "rkg98yBFDr"}