Model compression has become necessary when applying neural networks (NN) into many real application tasks that can accept slightly-reduced model accuracy with strict tolerance to model complexity. Recently, Knowledge Distillation, which distills the knowledge from well-trained and highly complex teacher model into a compact student model, has been widely used for model compression. However, under the strict requirement on the resource cost, it is quite challenging to achieve comparable performance with the teacher model, essentially due to the drastically-reduced expressiveness ability of the compact student model. Inspired by the nature of the expressiveness ability in Neural Networks, we propose to use multi-segment activation, which can significantly improve the expressiveness ability with very little cost, in the compact student model. Specifically, we propose a highly efficient multi-segment activation, called Light Multi-segment Activation (LMA), which can rapidly produce multiple linear regions with very few parameters by leveraging the statistical information. With using LMA, the compact student model is capable of achieving much better performance effectively and efficiently, than the ReLU-equipped one with same model scale. Furthermore, the proposed method is compatible with other model compression techniques, such as quantization, which means they can be used jointly for better compression performance. Experiments on state-of-the-art NN architectures over the real-world tasks demonstrate the effectiveness and extensibility of the LMA.
A piecewise linear function is composed of multiple linear segments. Some piecewise functions are continuous when the boundary value calculated by two adjacent intervals function is the same, whereas some may not be continuous. Benefit from its simplicity and the fitting ability to any function with enough segments, it is widely-used in machine learning models , especially as activations in Neural Networks . Theoretically, @cite_21 @cite_19 studied the number of linear regions in Neural Networks produced by piecewise linear activation functions (PLA), which can be used to measure the expressiveness ability of the networks.
@cite_19 @cite_21
2951603627 1931837783 2963334980 1887046642
We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models compute functions that react equally to complicated patterns of different inputs. The compositional structure of these functions enables them to re-use pieces of computation exponentially often in terms of the network's depth. This paper investigates the complexity of such compositional maps and contributes new theoretical results regarding the advantage of depth for neural networks with piecewise linear activation functions. In particular, our analysis is not specific to a single family of models, and as an example, we employ it for rectifier and maxout networks. We improve complexity bounds from pre-existing work and investigate the behavior of units in higher layers. A new piecewise linear method is presented for the approximation of digitized curves. This method produces a sequence of consecutive line segments and has the following characteristics: (i) it approximates the digitized curve with the minimum number of line segments, (ii) the Euclidean distance between each point of the digitized curve and the line segment that approximates it, does not exceed a boundary value spl epsiv and (iii) the vertices of the produced line are not (necessarily) points of the input curve. In this work, the reachable set estimation and safety verification problems for a class of piecewise linear systems equipped with neural network controllers are addressed. The neural network is considered to consist of Rectified Linear Unit (ReLU) activation functions. A layer-by-layer approach is developed for the output reachable set computation of ReLU neural networks. The computation is formulated in the form of a set of manipulations for a union of polytopes. Based on the output reachable set for neural network controllers, the output reachable set for a piecewise linear feedback control system can be estimated iteratively for a given finite-time interval. With the estimated output reachable set, the safety verification for piecewise linear systems with neural network controllers can be performed by checking the existence of intersections of unsafe regions and output reach set. A numerical example is presented to illustrate the effectiveness of our approach. Abstract The analysis and detection of edges is a central problem in applied mathematics and image processing. A number of results in recent years have shown that directional multiscale methods such as continuous curvelet and shearlet transforms offer a powerful theoretical framework to capture the geometry of edge singularities, going far beyond the capabilities of the conventional wavelet transform. The continuous shearlet transform, in particular, provides a precise geometric characterization of edges in piecewise constant functions in R 2 and R 3 , including corner points. However, a question has been raised frequently: What happens if the function is piecewise smooth and not just piecewise constant? Clearly, a piecewise smooth function is a much more realistic model of images with edges. In this paper, we extend the characterization results previously known and show that, also in the case of piecewise smooth functions, the continuous shearlet transform can detect the location and orientation of edge points, including corner points, through its asymptotic decay at fine scales. The new proof introduces innovative technical constructions to deal with the more challenging problem. The new results set the theoretical groundwork for the application of the shearlet framework to a wider class of problems from image processing.