The most popular quasi-Newton update is bfgs for which \(P_k\)  approximates \([H_n(\theta _k)]^{-1}\)  iteratively using only gradients. Although not as fast as nr, convergence for quasi-Newton with an appropriate choice of \(\gamma _k\)  can be superlinear.See [1]} and [2]} for an overview of quasi-Newton updates and their properties. Quasi-Newton methods bypass Hessian computations, but analytical convergence results are more difficult to obtain.
The gradient-descent algorithm (gd) sets \(P_k=I_d\) , it only requires computing a single gradient \(G_n(\theta _k)\) . Convergence, however, requires choosing \(\gamma _k\)  small enough that all eigenvalues of \(I_d-\gamma _k H_n(\tilde{\theta }_k)\)  lie in the restricted region \((-1,1)\) , for some intermediate value \(\tilde{\theta }_k\) . This results in very slow convergence when the ratio of the largest to smallest eigenvalue of \(H_n\)  is large.
