Proposition 2
For any Multi-Layer Perceptron (MLP) implementing \(\theta (z)\)  with 1-Lipschitz activation functions (e.g., ReLU,
Leaky ReLU, SoftPlus, Tanh, Sigmoid, ArcTan or Softsign) [1]}, the upper bound of Eq. (REF ) is
\(\hat{\mathcal {L}}_{\theta } = \Vert  \theta (z)J - \nabla _z f(z) \Vert _2 + \lambda \sqrt{d} \prod _{k=1}^K\Vert W_k\Vert _2\) 
