Nonlinearity is crucial to the performance of a deep (neural) network (DN).
 To date there has been little progress understanding the menagerie of available  nonlinearities, but recently progress has been made on understanding the r\^{o}le played by piecewise affine and convex nonlinearities like the ReLU and absolute value activation functions and max-pooling.
 In particular, DN layers constructed from these operations can be interpreted as {\em max-affine spline operators} (MASOs) that have an elegant link to vector quantization (VQ) and $K$-means.
 While this is good theoretical progress, the entire MASO approach is predicated on the requirement that the nonlinearities be piecewise affine and convex, which precludes important activation functions like the sigmoid, hyperbolic tangent, and softmax.
 {\em This paper extends the MASO framework to these and an infinitely large class of new nonlinearities by linking deterministic MASOs with probabilistic Gaussian Mixture Models (GMMs). }
 We show that, under a GMM, piecewise affine, convex nonlinearities like ReLU, absolute value, and max-pooling can be interpreted as solutions to certain natural ``hard'' VQ inference problems, while sigmoid, hyperbolic tangent, and softmax can be interpreted as solutions to corresponding ``soft'' VQ inference problems.
 We further extend the framework by hybridizing the hard and soft VQ optimizations to create a $\beta$-VQ inference that interpolates between hard, soft, and linear VQ inference.
 A prime example of a $\beta$-VQ DN nonlinearity is the {\em swish} nonlinearity, which offers state-of-the-art performance in a range of computer vision tasks but was developed ad hoc by experimentation.
 Finally, we validate with experiments an important assertion of our theory, namely that DN performance can be significantly improved by enforcing orthogonality in its linear filters.
 Deep (neural) networks (DNs) have recently come to the fore in a wide range of machine learning tasks, from regression to classification and beyond. A DN is typically constructed by composing a large number of linear/affine transformations interspersed with up/down-sampling operations and simple scalar nonlinearities such as the ReLU, absolute value, sigmoid, hyperbolic tangent, etc. BID13 . Scalar nonlinearities are crucial to a DN's performance. Indeed, without nonlinearity, the entire network would collapse to a simple affine transformation. But to date there has been little progress understanding and unifying the menagerie of nonlinearities, with few reasons to choose one over another other than intuition or experimentation.Recently, progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU, leaky ReLU, and absolute value activations and downsampling operations like max-, average-, and channel-pooling BID1 . In particular, these operations can be interpreted as max-affine spline operators (MASOs) BID16 ; BID14 that enable a DN to find a locally optimized piecewise affine approximation to the prediction operator given training data. A spline-based prediction is made in two steps. First, given an input signal x, we determine which region of the spline's partition of the domain (the input signal space) it falls into. Second, we apply to x the fixed (in this case affine) function that is assigned to that partition region to obtain the prediction y = f (x).The key result of BID1 is any DN layer constructed from a combination of linear and piecewise affine and convex is a MASO, and hence the entire DN is merely a composition of MASOs.MASOs have the attractive property that their partition of the signal space (the collection of multidimensional "knots") is completely determined by their affine parameters (slopes and offsets). This provides an elegant link to vector quantization (VQ) and K-means clustering. That is, during learning, a DN implicitly constructs a hierarchical VQ of the training data that is then used for splinebased prediction. This is good progress for DNs based on ReLU, absolute value, and max-pooling, but what about DNs based on classical, high-performing nonlinearities that are neither piecewise affine nor convex like the sigmoid, hyperbolic tangent, and softmax or fresh nonlinearities like the swish BID20 that has been shown to outperform others on a range of tasks?Contributions . In this paper , we address this gap in the DN theory by developing a new framework that unifies a wide range of DN nonlinearities and inspires and supports the development of new ones. The key idea is to leverage the yinyang relationship between deterministic VQ/K-means and probabilistic Gaussian Mixture Models (GMMs) BID3 . Under a GMM, piecewise affine, convex nonlinearities like ReLU and absolute value can be interpreted as solutions to certain natural hard inference problems, while sigmoid and hyperbolic tangent can be interpreted as solutions to corresponding soft inference problems. We summarize our primary contributions as follows:Contribution 1: We leverage the well-understood relationship between VQ, K-means, and GMMs to propose the Soft MASO (SMASO) model, a probabilistic GMM that extends the concept of a deterministic MASO DN layer. Under the SMASO model, hard maximum a posteriori (MAP) inference of the VQ parameters corresponds to conventional deterministic MASO DN operations that involve piecewise affine and convex functions, such as fully connected and convolution matrix multiplication; ReLU, leaky-ReLU, and absolute value activation; and max-, average-, and channelpooling. These operations assign the layer's input signal (feature map) to the VQ partition region corresponding to the closest centroid in terms of the Euclidean distance, Contribution 2: A hard VQ inference contains no information regarding the confidence of the VQ region selection, which is related to the distance from the input signal to the region boundary. In response, we develop a method for soft MAP inference of the VQ parameters based on the probability that the layer input belongs to a given VQ region. Switching from hard to soft VQ inference recovers several classical and powerful nonlinearities and provides an avenue to derive completely new ones. We illustrate by showing that the soft versions of ReLU and max-pooling are the sigmoid gated linear unit and softmax pooling, respectively. We also find a home for the sigmoid, hyperbolic tangent, and softmax in the framework as a new kind of DN layer where the MASO output is the VQ probability.Contribution 3: We generalize hard and soft VQ to what we call β-VQ inference, where β ∈ (0, 1) is a free and learnable parameter. This parameter interpolates the VQ from linear (β → 0), to probabilistic SMASO (β = 0.5), to deterministic MASO (β → 1). We show that the β-VQ version of the hard ReLU activation is the swish nonlinearity, which offers state-of-the-art performance in a range of computer vision tasks but was developed ad hoc through experimentation BID20 .Contribution 4: Seen through the MASO lens, current DNs solve a simplistic per-unit (per-neuron), independent VQ optimization problem at each layer. In response, we extend the SMASO GMM to a factorial GMM that that supports jointly optimal VQ across all units in a layer. Since the factorial aspect of the new model would make naïve VQ inference exponentially computationally complex, we develop a simple sufficient condition under which a we can achieve efficient, tractable, jointly optimal VQ inference. The condition is that the linear "filters" feeding into any nonlinearity should be orthogonal. We propose two simple strategies to learn approximately and truly orthogonal weights and show on three different datasets that both offer significant improvements in classification per-formance. Since orthogonalization can be applied to an arbitrary DN, this result and our theoretical understanding are of independent interest. This paper is organized as follows. After reviewing the theory of MASOs and VQ for DNs in Section 2, we formulate the GMM-based extension to SMASOs in Section 3. Section 4 develops the hybrid β-VQ inference with a special case study on the swish nonlinearity. Section 5 extends the SMASO to a factorial GMM and shows the power of DN orthogonalization. We wrap up in Section 6 with directions for future research. Proofs of the various results appear in several appendices in the Supplementary Material.
Reformulate deep networks nonlinearities from a vector quantization scope and bridge most known nonlinearities together.
{"src": "scitldr-AIC_train", "paper_id": "Syxt2jC5FX"}