MixUp is a data augmentation scheme in which pairs of training samples and their corresponding labels are mixed using linear coefficients. Without label mixing, MixUp becomes a more conventional scheme: input samples are moved but their original labels are retained. Because samples are preferentially moved in the direction of other classes \iffalse -- which are typically clustered in input space -- \fi we refer to this method as directional adversarial training, or DAT. We show that under two mild conditions, MixUp asymptotically convergences to a subset of DAT. We define untied MixUp (UMixUp), a superset of MixUp wherein training labels are mixed with different linear coefficients to those of their corresponding samples. We show that under the same mild conditions, untied MixUp converges to the entire class of DAT schemes. Motivated by the understanding that UMixUp is both a generalization of MixUp and a form of adversarial training, we experiment with different datasets and loss functions to show that UMixUp provides improved performance over MixUp. In short, we present a novel interpretation of MixUp as belonging to a class highly analogous to adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp. Deep learning applications often require complex networks with a large number of parameters (He et al., 2016; Zagoruyko & Komodakis, 2016; Devlin et al., 2018) . Although neural networks perform so well that their ability to generalize is an area of study in itself (Zhang et al., 2017a; Arpit et al., 2017) , their high complexity nevertheless causes them to overfit their training data (Kukacka et al., 2017) . For this reason, effective regularization techniques are in high demand. There are two flavors of regularization: complexity curtailing and data augmentation 1 . Complexity curtailing methods constrain models to learning in a subset of parameter space which has a higher probability of generalizing well. Notable examples are weight decay (Krogh & Hertz, 1991) and dropout (Srivastava et al., 2014) . Data augmentation methods add transformed versions of training samples to the original training set. Conventionally, transformed samples retain their original label, so that models effectively see a larger set of data-label training pairs. Commonly applied transformations in image applications include flips, crops and rotations. A recently devised family of augmentation schemes called adversarial training has attracted active research interest (Szegedy et al., 2013; Goodfellow et al., 2014; Miyato et al., 2016; Athalye et al., 2018; Shaham et al., 2018; He et al., 2018) . Adversarial training seeks to reduce a model's propensity to misclassify minimally perturbed training samples, or adversarials. While attack algorithms used for testing model robustness may search for adversarials in unbounded regions of input space, adversarial training schemes generally focus on perturbing training samples within a bounded region, while retaining the sample's original label (Goodfellow et al., 2015; Shaham et al., 2018) . Another recently proposed data augmentation scheme is MixUp (Zhang et al., 2017b) , in which new samples are generated by mixing pairs of training samples using linear coefficients. Despite its well established generalization performance (Zhang et al., 2017b; Guo et al., 2018; Verma et al., 2018) , the working mechanism of MixUp is not well understood. Guo et al. (2018) suggest viewing MixUp as imposing local linearity on the model using points outside of the data manifold. While this perspective is insightful, we do not believe it paints a full picture of how MixUp operates. A recent study (Lamb et al., 2019) provides empirical evidence that MixUp improves adversarial robustness, but does not present MixUp as a form of adversarial training. We build a framework to understand MixUp in a broader context: we argue that adversarial training is a central working principle of MixUp. To support this contention, we connect MixUp to a MixUplike scheme which does not perform label mixing, and we relate this scheme to adversarial training. Without label mixing, MixUp becomes a conventional augmentation scheme: input samples are moved, but their original labels are retained. Because samples are moved in the direction of other samples -which are typically clustered in input space -we describe this method as 'directional'. Because this method primarily moves training samples in the direction of adversarial classes, this method is analogous to adversarial training. We thus refer to MixUp without label mixing as directional adversarial training (DAT). We show that MixUp converges to a subset of DAT under mild conditions, and we thereby argue that adversarial training is a working principle of MixUp. Inspired by this new understanding of MixUp as a form of adversarial training, and upon realizing that MixUp is (asymptotically) a subset of DAT, we introduce Untied MixUp (UMixUp), a simple enhancement of MixUp which converges to the entire family of DAT schemes, as depicted in Figure  1 . Untied Mixup mixes data-label training pairs in a similar way to MixUp, with the distinction that the label mixing ratio is an arbitrary function of the sample mixing ratio. We perform experiments to show that UMixUp's classification performance improves upon MixUp. In short, this research is motivated by a curiosity to better understand the working of MixUp. In-sodoing we aim to: 1. Establish DAT as analogous to adversarial training. This is discussed in section 4. 2. Establish UMixUp as a superset of MixUp, and as converging to the entire family of DAT schemes. In-so-doing, a) establish MixUp's convergence to a subset of DAT, and thereby that it operates analogously to adversarial training; and b) establish UMixUp as a broader class of MixUp-like schemes that operate analogously to adversarial training. This is discussed in 5. 3. Establish empirically that UMixUp's classification performance improves upon MixUp. This is discussed in section 6. Finally we note that this paper has another contribution. Conventionally, MixUp is only applicable to baseline models that use cross entropy loss. All analytical results we develop in this paper are applicable to a wider family of models using any loss function which we term target-linear. We define target-linearity and experiment with a new loss function called negative cosine-loss to show its potential. Regular (non-calligraphic) capitalized letters such as X will denote random variables, and their lowercase counterparts, e.g., x, will denote realizations of a random variable. Any sequence, (a 1 , a 2 , . . . , a n ) will be denoted by a n 1 . Likewise (A 1 , A 2 , . . . , A n ) will be denoted by A n 1 , and a sequence of sample pairs ((x 1 , x 1 ), (x 2 , x 2 ), . . . , (x n , x n )) denoted by (x, x ) n 1 . For any value a ∈ [0, 1], we will use a as a short notation for 1 − a. Classification Setting Consider a standard classification problem, in which one wishes to learn a classifier that predicts the class label for a sample. Formally, let X be a vector space in which the samples of interest live and let Y be the set of all possible labels associated with these samples. The set of training samples will be denoted by D, a subset of X . We will use t(x ) to denote the true label of x. Let F be a neural network function, parameterized by θ, which maps X to another vector space Z. Let ϕ : Y → Z be a function that maps a label in Y to an element in Z such that for any y, y ∈ Y, if y = y , then ϕ(y) = ϕ(y ). In the space Z, we refer to F (x) as the model's prediction. With slight abuse of language, we will occasionally refer to both t(x) and ϕ(t(x)) as the "label" of x. Let : Z ×Z → R be a loss function, using which one defines an overall loss function as Here we have taken the notational convention that the first argument of represents the model's prediction and the second represents the target label. In this setting, the learning problem is formulated as minimizing L with respect to its characterizing parameters θ.
We present a novel interpretation of MixUp as belonging to a class highly analogous to adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp
{"src": "scitldr-AIC_train", "paper_id": "SkgjKR4YwH"}