Generalized minimal residual method

In mathematics, the generalized minimal residual method (GMRES) is an iterative method for the numerical solution of a nonsymmetric system of linear equations. The method approximates the solution by the vector in a Krylov subspace with minimal residual. The Arnoldi iteration is used to find this vector.
The GMRES method was developed by Yousef Saad and Martin H. Schultz in 1986.
GMRES is a generalization and improvement of the MINRES method developed by Chris Paige and Michael Saunders in 1975. GMRES also is a special case of the DIIS method developed by Peter Pulay in 1980. DIIS is also applicable to non-linear systems.
The method.
Denote the Euclidean norm of any vector v by formula_1. Denote the (square) system of linear equations to be solved by
The matrix "A" is assumed to be invertible of size "m"-by-"m". Furthermore, it is assumed that b is normalized, i.e., that formula_3.
The "n"-th Krylov subspace for this problem is
where formula_5 is the initial error given an initial guess formula_6. Clearly formula_7 if formula_8.
GMRES approximates the exact solution of formula_9 by the vector formula_10 that minimizes the Euclidean norm of the residual formula_11.
The vectors formula_12 might be close to linearly dependent, so instead of this basis, the Arnoldi iteration is used to find orthonormal vectors formula_13 which form a basis for formula_14. In particular, formula_15. 
Therefore, the vector formula_10 can be written as formula_17 with formula_18, where formula_19 is the "m"-by-"n" matrix formed by formula_20.
The Arnoldi process also produces an (formula_21)-by-formula_22 upper Hessenberg matrix formula_23 with
Because columns of formula_25 are orthonormal, we have
where
is the first vector in the standard basis of formula_28, and 
formula_30 being the first trial vector (usually zero). Hence, formula_31 can be found by minimizing the Euclidean norm of the residual
This is a linear least squares problem of size "n".
This yields the GMRES method. On the formula_22-th iteration:
At every iteration, a matrix-vector product formula_38 must be computed. This costs about formula_39 floating-point operations for general dense matrices of size formula_40, but the cost can decrease to formula_41 for sparse matrices. In addition to the matrix-vector product, formula_42 floating-point operations must be computed at the "n" -th iteration.
Convergence.
The "n"th iterate minimizes the residual in the Krylov subspace formula_14. Since every subspace is contained in the next subspace, the residual does not increase. After "m" iterations, where "m" is the size of the matrix "A", the Krylov space "K""m" is the whole of R"m" and hence the GMRES method arrives at the exact solution. However, the idea is that after a small number of iterations (relative to "m"), the vector "x""n" is already a good approximation to the 
exact solution.
This does not happen in general. Indeed, a theorem of Greenbaum, Pták and Strakoš states that for every nonincreasing sequence "a"1, …, "a""m"−1, "a""m" = 0, one can find a matrix "A" such that the ||"r""n"|| = "a""n" for all "n", where "r""n" is the residual defined above. In particular, it is possible to find a matrix for which the residual stays constant for "m" − 1 iterations, and only drops to zero at the last iteration.
In practice, though, GMRES often performs well. This can be proven in specific situations. If the symmetric part of "A", that is formula_44, is positive definite, then
where formula_46 and formula_47 denote the smallest and largest eigenvalue of the matrix formula_48, respectively.
If "A" is symmetric and positive definite, then we even have
where formula_50 denotes the condition number of "A" in the Euclidean norm.
In the general case, where "A" is not positive definite, we have
where "P""n" denotes the set of polynomials of degree at most "n" with "p"(0) = 1, "V" is the matrix appearing in the spectral decomposition of "A", and "σ"("A") is the spectrum of "A". Roughly speaking, this says that fast convergence occurs when the eigenvalues of "A" are clustered away from the origin and "A" is not too far from normality.
All these inequalities bound only the residuals instead of the actual error, that is, the distance between the current iterate "x""n" and the exact solution.
Extensions of the method.
Like other iterative methods, GMRES is usually combined with a preconditioning method in order to speed up convergence.
The cost of the iterations grow as O("n"2), where "n" is the iteration number. Therefore, the method is sometimes restarted after a number, say "k", of iterations, with "x""k" as initial guess. The resulting method is called GMRES("k") or Restarted GMRES. For non-positive definite matrices, this method may suffer from stagnation in convergence as the restarted subspace is often close to the earlier subspace.
The shortcomings of GMRES and restarted GMRES are addressed by the recycling of Krylov subspace in the GCRO type methods such as GCROT and GCRODR.
Recycling of Krylov subspaces in GMRES can also speed up convergence when sequences of linear systems need to be solved.
Comparison with other solvers.
The Arnoldi iteration reduces to the Lanczos iteration for symmetric matrices. The corresponding Krylov subspace method is the minimal residual method (MinRes) of Paige and Saunders. Unlike the unsymmetric case, the MinRes method is given by a three-term recurrence relation. It can be shown that there is no Krylov subspace method for general matrices, which is given by a short recurrence relation and yet minimizes the norms of the residuals, as GMRES does.
Another class of methods builds on the unsymmetric Lanczos iteration, in particular the BiCG method. These use a three-term recurrence relation, but they do not attain the minimum residual, and hence the residual does not decrease monotonically for these methods. Convergence is not even guaranteed.
The third class is formed by methods like CGS and BiCGSTAB. These also work with a three-term recurrence relation (hence, without optimality) and they can even terminate prematurely without achieving convergence. The idea behind these methods is to choose the generating polynomials of the iteration sequence suitably.
None of these three classes is the best for all matrices; there are always examples in which one class outperforms the other. Therefore, multiple solvers are tried in practice to see which one is the best for a given problem.
Solving the least squares problem.
One part of the GMRES method is to find the vector formula_52 which minimizes
Note that formula_23 is an ("n" + 1)-by-"n" matrix, hence it gives an over-constrained linear system of "n"+1 equations for "n" unknowns.
The minimum can be computed using a QR decomposition: find an ("n" + 1)-by-("n" + 1) orthogonal matrix Ω"n" and an ("n" + 1)-by-"n" upper triangular matrix formula_55 such that
The triangular matrix has one more row than it has columns, so its bottom row consists of zero. Hence, it can be decomposed as
where formula_58 is an "n"-by-"n" (thus square) triangular matrix.
The QR decomposition can be updated cheaply from one iteration to the next, because the Hessenberg matrices differ only by a row of zeros and a column:
where "h""n+1" = ("h"1,"n+1", …, "h""n+1,n+1")T. This implies that premultiplying the Hessenberg matrix with Ω"n", augmented with zeroes and a row with multiplicative identity, yields almost a triangular matrix:
This would be triangular if σ is zero. To remedy this, one needs the Givens rotation
where
With this Givens rotation, we form
Indeed,
is a triangular matrix.
Given the QR decomposition, the minimization problem is easily solved by noting that
Denoting the vector formula_66 by
with "g""n" ∈ R"n" and γ"n" ∈ R, this is
The vector "y" that minimizes this expression is given by 
Again, the vectors formula_70 are easy to update.
Example code.
Regular GMRES (MATLAB / GNU Octave).
function [x, e] = gmres( A, b, x, max_iterations, threshold)
 n = length(A);
 m = max_iterations;
 % use x as the initial vector
 r = b - A * x;
 b_norm = norm(b);
 error = norm(r) / b_norm;
 % initialize the 1D vectors
 sn = zeros(m, 1);
 cs = zeros(m, 1);
 %e1 = zeros(n, 1);
 e1 = zeros(m+1, 1);
 e1(1) = 1;
 e = [error];
 r_norm = norm(r);
 Q(:,1) = r / r_norm;
 beta = r_norm * e1; %Note: this is not the beta scalar in section "The method" above but the beta scalar multiplied by e1
 for k = 1:m
 % run arnoldi
 [H(1:k+1, k) Q(:, k+1)] = arnoldi(A, Q, k);
 % eliminate the last element in H ith row and update the rotation matrix
 [H(1:k+1, k) cs(k) sn(k)] = apply_givens_rotation(H(1:k+1,k), cs, sn, k);
 % update the residual vector
 beta(k + 1) = -sn(k) * beta(k);
 beta(k) = cs(k) * beta(k);
 error = abs(beta(k + 1)) / b_norm;
 % save the error
 e = [e; error];
 if (error <= threshold)
 break;
 end
 end
 % if threshold is not reached, k = m at this point (and not m+1) 
 % calculate the result
 y = H(1:k, 1:k) \ beta(1:k);
 x = x + Q(:, 1:k) * y;
end
% Arnoldi Function %
function [h, q] = arnoldi(A, Q, k)
 q = A*Q(:,k); % Krylov Vector
 for i = 1:k % Modified Gram-Schmidt, keeping the Hessenberg matrix
 h(i) = q' * Q(:, i);
 q = q - h(i) * Q(:, i);
 end
 h(k + 1) = norm(q);
 q = q / h(k + 1);
end
% Applying Givens Rotation to H col %
function [h, cs_k, sn_k] = apply_givens_rotation(h, cs, sn, k)
 % apply for ith column
 for i = 1:k-1
 temp = cs(i) * h(i) + sn(i) * h(i + 1);
 h(i+1) = -sn(i) * h(i) + cs(i) * h(i + 1);
 h(i) = temp;
 end
 % update the next sin cos values for rotation
 [cs_k sn_k] = givens_rotation(h(k), h(k + 1));
 % eliminate H(i + 1, i)
 h(k) = cs_k * h(k) + sn_k * h(k + 1);
 h(k + 1) = 0.0;
end
%%----Calculate the Given rotation matrix----%%
function [cs, sn] = givens_rotation(v1, v2)
% if (v1 == 0)
% cs = 0;
% sn = 1;
% else
 t = sqrt(v1^2 + v2^2);
% cs = abs(v1) / t;
% sn = cs * v2 / v1;
 cs = v1 / t; % see http://www.netlib.org/eispack/comqr.f
 sn = v2 / t;
% end
end