On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vector
It is an elementary fact in the scientific literature that the Lipschitz norm
of the realization function of a feedforward fully-connected rectified linear
unit (ReLU) artificial neural network (ANN) can, up to a multiplicative
constant, be bounded from above by sums of powers of the norm of the ANN
parameter vector. Roughly speaking, in this work we reveal in the case of
shallow ANNs that the converse inequality is also true. More formally, we prove
that the norm of the equivalence class of ANN parameter vectors with the same
realization function is, up to a multiplicative constant, bounded from above by
the sum of powers of the Lipschitz norm of the ANN realization function (with
the exponents $ 1/2 $ and $ 1 $). Moreover, we prove that this upper bound only
holds when employing the Lipschitz norm but does neither hold for H\"older
norms nor for Sobolev-Slobodeckij norms. Furthermore, we prove that this upper
bound only holds for sums of powers of the Lipschitz norm with the exponents $
1/2 $ and $ 1 $ but does not hold for the Lipschitz norm alone.