The Role of Depth, Width, and Activation Complexity in the Number of Linear Regions of Neural Networks
Many feedforward neural networks generate continuous and piecewise-linear
(CPWL) mappings. Specifically, they partition the input domain into regions on
which the mapping is an affine function. The number of these so-called linear
regions offers a natural metric to characterize the expressiveness of CPWL
mappings. Although the precise determination of this quantity is often out of
reach, bounds have been proposed for specific architectures, including the
well-known ReLU and Maxout networks. In this work, we propose a more general
perspective and provide precise bounds on the maximal number of linear regions
of CPWL networks based on three sources of expressiveness: depth, width, and
activation complexity. Our estimates rely on the combinatorial structure of
convex partitions and highlight the distinctive role of depth which, on its
own, is able to exponentially increase the number of regions. We then introduce
a complementary stochastic framework to estimate the average number of linear
regions produced by a CPWL network architecture. Under reasonable assumptions,
the expected density of linear regions along any 1D path is bounded by the
product of depth, width, and a measure of activation complexity (up to a
scaling factor). This yields an identical role to the three sources of
expressiveness: no exponential growth with depth is observed anymore.