It is well known that a neural network with piecewise linear activation functions generates a finite number of linear regions in the input space [1]}, [2]}, [3]}. This property is called the piecewise linear property and is demonstrated in Fig. REF . The number of linear regions as well as the density of such regions can usually reflect the effective complexity. Therefore, a series of studies on effective complexity starts from piecewise linear activation functions (e.g., ReLU, Maxout) or are based on the piecewise linear property.
