Another popular way to compute \( \widehat{\theta }_\text{ols} \)  is to use the first-order optimization methods in an iterative way.
Arguably, the most basic first-order optimization method is the gradient descent (GD) algorithm.
For the OLS problem, a standard GD algorithm should update the estimates iteratively as \(\widehat{\theta }^{(t)} = \widehat{\theta }^{(t-1)} - \alpha \dot{\mathcal {L}}(\widehat{\theta }^{(t-1)})\) , where \( \dot{\mathcal {L}}(\theta ) \)  denotes the gradient of \( \mathcal {L}\)  at \( \theta \)  and \( \alpha > 0 \)  is the learning rate controlling the step size.
As shown by [1]}, with an appropriately selected \(\alpha \) , the estimates \( \lbrace \widehat{\theta }^{(t)} \rbrace  \)  generated by the GD algorithm should satisfy \( \Vert \widehat{\theta }^{(t)} - \widehat{\theta }_\text{ols}\Vert  \le \rho ^t \Vert \widehat{\theta }^{(0)} - \widehat{\theta }_\text{ols}\Vert \)  for some convergence factor \( \rho \in [0,1) \)  and an arbitrary initial value \( \widehat{\theta }^{(0)} \) .
The optimal convergence factor can be achieved if we choose \( \alpha = 2 / \lbrace \lambda _{\max }(\widehat{\Sigma }_{xx})+ \lambda _{\min }(\widehat{\Sigma }_{xx}) \rbrace  \) .
In this case, the optimal convergence factor should be \( \rho = (\kappa - 1)/ (\kappa +1) \)  , where \( \kappa = \lambda _{\max }(\widehat{\Sigma }_{xx}) / \lambda _{\min }(\widehat{\Sigma }_{xx}) \ge 1\)  is the condition number of \( \widehat{\Sigma }_{xx} \) .
