Note that, unlike the result of Thorpe and van Gennip [1]}, this result is obtained without regularisation. So in a sense the \(1/2-\) Hölder regularity corresponds to the natural regularity of the weights obtained by gradient descent.
