We compare the performance of three commonly used nonlinear optimization algorithms on the problem of linear model improvement via smooth hyperparameter tuning. Using the above analysis we are able to apply a second-order optimization method, which offers to dramatically reduce the number of lower function evaluations in BLPP and significantly speed up optimization problems where the lower level constraint evaluation dominates. To compare, we optimize the hyperparameters of 3 linear models on MNIST [1]} using three commonly used optimization algorithms, two gradient-only: (i) Adam [2]}, (ii) L-BFGS [3]} and one using second-order information (enabled by this work) (iii) SQP [4]}. The three linear models we choose all employ a least-squares lower level loss where the target vector is the one-hot encoding of the ten MNIST digits. We train on 1000 randomly selected MNIST examples in the train set and evaluate the upper level (test) loss on all examples in the test set. The upper level loss is the cross-entropy classification loss, \(f_U({z^\star }, p) = -\sum _j^{N_\text{test}}\sum _{i=1}^{10} t_{i,j} \log \big (q_{i,j}\big )\)  where \(t_{i,j} = \delta _{i,y_j}\)  and \(q_{i,j} = e^{x_j^T {z^\star }_i} / \left( \sum _{i=1}^{10} e^{x_j^T {z^\star }_i} \right)\) .\(t_{i,j} = \delta _{i,y_j}\)  indicates that \(t_{i,j}\)  is equal to 1 if example \(j\)  is of the class \(i\)  and 0 otherwise. For all models \(z \in \mathbb {R}^{n d}\)  where \(n = 10\)  is the number of MNIST classes and \(d\)  is the number of features in the data vector.
