Semidefinite programming

Semidefinite programming (SDP) is a subfield of convex optimization concerned with the optimization of a linear objective function (a user-specified function that the user wants to minimize or maximize)
over the intersection of the cone of positive semidefinite matrices with an affine space, i.e., a spectrahedron.
Semidefinite programming is a relatively new field of optimization which is of growing interest for several reasons. Many practical problems in operations research and combinatorial optimization can be modeled or approximated as semidefinite programming problems. In automatic control theory, SDPs are used in the context of linear matrix inequalities. SDPs are in fact a special case of cone programming and can be efficiently solved by interior point methods.
All linear programs can be expressed as SDPs, and via hierarchies of SDPs the solutions of polynomial optimization problems can be approximated. Semidefinite programming has been used in the optimization of complex systems. In recent years, some quantum query complexity problems have been formulated in terms of semidefinite programs.
Motivation and definition.
Initial motivation.
A linear programming problem is one in which we wish to maximize or minimize a linear objective function of real variables over a polytope. In semidefinite programming, we instead use real-valued vectors and are allowed to take the dot product of vectors; nonnegativity constraints on real variables in LP ("linear programming") are replaced by semidefiniteness constraints on matrix variables in SDP ("semidefinite programming"). Specifically, a general semidefinite programming problem can be defined as any mathematical programming problem of the form
where the formula_2, and the formula_3 are real numbers and formula_4 is the dot product of formula_5 and formula_6.
Equivalent formulations.
An formula_7 matrix formula_8 is said to be positive semidefinite if it is the Gramian matrix of some vectors (i.e. if there exist vectors formula_9 such that formula_10 for all formula_11). If this is the case, we denote this as formula_12. Note that there are several other equivalent definitions of being positive semidefinite, for example, positive semidefinite matrices are self-adjoint matrices that have only non-negative eigenvalues.
Denote by formula_13 the space of all formula_7 real symmetric matrices. The space is equipped with the inner product (where formula_15 denotes the trace)
formula_16
We can rewrite the mathematical program given in the previous section equivalently as
where entry formula_11 in formula_19 is given by formula_20 from the previous section and formula_21 is a symmetric formula_7 matrix having formula_11th entry formula_24 from the previous section. Thus, the matrices formula_19 and formula_21 are symmetric and the above inner products are well-defined.
Note that if we add slack variables appropriately, this SDP can be converted to one of the form
For convenience, an SDP may be specified in a slightly different, but equivalent form. For example, linear expressions involving nonnegative scalar variables may be added to the program specification. This remains an SDP because each variable can be incorporated into the matrix formula_28 as a diagonal entry (formula_29 for some formula_30). To ensure that formula_31, constraints formula_32 can be added for all formula_33. As another example, note that for any positive semidefinite matrix formula_28, there exists a set of vectors formula_35 such that the formula_30, formula_37 entry of formula_28 is formula_39 the scalar product of formula_40 and formula_41. Therefore, SDPs are often formulated in terms of linear expressions on scalar products of vectors. Given the solution to the SDP in the standard form, the vectors formula_35 can be recovered in formula_43 time (e.g., by using an incomplete Cholesky decomposition of X).
Duality theory.
Definitions.
Analogously to linear programming, given a general SDP of the form
(the primal problem or P-SDP), we define the "dual" semidefinite program (D-SDP) as
where for any two matrices formula_46 and formula_47, formula_48 means formula_49.
Weak duality.
The weak duality theorem states that the value of the primal SDP is at least the value of the dual SDP. Therefore, any feasible solution to the dual SDP lower-bounds the primal SDP value, and conversely, any feasible solution to the primal SDP upper-bounds the dual SDP value. This is because
where the last inequality is because both matrices are positive semidefinite, and the result of this function is sometimes referred to as duality gap.
Strong duality.
Under a condition known as Slater's condition, the value of the primal and dual SDPs are equal. This is known as strong duality. Unlike for linear programs, however, not every SDP satisfies strong duality; in general, the value of the dual SDP may lie strictly below the value of the primal.
(i) Suppose the primal problem (P-SDP) is bounded below and strictly
feasible (i.e., there exists
formula_51 such that formula_52, formula_53).
Then there is an optimal solution formula_54 to (D-SDP) and
(ii) Suppose the dual problem (D-SDP) is bounded above and strictly
feasible (i.e.,
formula_56 for some formula_57).
Then there is an optimal solution formula_58 to (P-SDP) and
the equality from (i) holds.
Examples.
Example 1.
Consider three random variables formula_59, formula_60, and formula_19. By definition, their correlation coefficients formula_62 are valid if and only if
in which case this matrix is called the correlation matrix. Suppose that we know from some prior knowledge (empirical results of an experiment, for example) that formula_64 and formula_65. The problem of determining the smallest and largest values that formula_66 can take is given by:
We set formula_71 to obtain the answer. This can be formulated by an SDP. We handle the inequality constraints by augmenting the variable matrix and introducing slack variables, for example
formula_72
Solving this SDP gives the minimum and maximum values of formula_73 as formula_74 and formula_75 respectively.
Example 2.
Consider the problem
where we assume that formula_78 whenever formula_79.
Introducing an auxiliary variable formula_80 the problem can be reformulated:
In this formulation, the objective is a linear function of the variables formula_83.
The first restriction can be written as
where the matrix formula_85 is the square matrix with values in the diagonal equal
to the elements of the vector formula_86.
The second restriction can be written as
Defining formula_88 as follows
We can use the theory of Schur Complements to see that
The semidefinite program associated with this problem is
Example 3 (Goemans–Williamson max cut approximation algorithm).
Semidefinite programs are important tools for developing approximation algorithms for NP-hard maximization problems. The first approximation algorithm based on an SDP is due to Michel Goemans and David P. Williamson (JACM, 1995). They studied the max cut problem: Given a graph "G" = ("V", "E"), output a partition of the vertices "V" so as to maximize the number of edges crossing from one side to the other. This problem can be expressed as an integer quadratic program:
Unless P = NP, we cannot solve this maximization problem efficiently. However, Goemans and Williamson observed a general three-step procedure for attacking this sort of problem:
For max cut, the most natural relaxation is
This is an SDP because the objective function and constraints are all linear functions of vector inner products. Solving the SDP gives a set of unit vectors in formula_99; since the vectors are not required to be collinear, the value of this relaxed program can only be higher than the value of the original quadratic integer program. Finally, a rounding procedure is needed to obtain a partition. Goemans and Williamson simply choose a uniformly random hyperplane through the origin and divide the vertices according to which side of the hyperplane the corresponding vectors lie. Straightforward analysis shows that this procedure achieves an expected "approximation ratio" (performance guarantee) of 0.87856 - ε. (The expected value of the cut is the sum over edges of the probability that the edge is cut, which is proportional to the angle formula_100 between the vectors at the endpoints of the edge over formula_101. Comparing this probability to formula_102, in expectation the ratio is always at least 0.87856.) Assuming the unique games conjecture, it can be shown that this approximation ratio is essentially optimal.
Since the original paper of Goemans and Williamson, SDPs have been applied to develop numerous approximation algorithms. Recently, Prasad Raghavendra has developed a general framework for constraint satisfaction problems based on the unique games conjecture.
Algorithms.
There are several types of algorithms for solving SDPs. These algorithms output the value of the SDP up to an additive error formula_95 in time that is polynomial in the program description size and formula_104.
There are also facial reduction algorithms that can be used to preprocess SDPs problems by inspecting the constraints of the problem. These can be used to detect lack of strict feasibility, to delete redundant rows and columns, and also to reduce the size of the variable matrix.
Interior point methods.
Most codes are based on interior point methods (CSDP, MOSEK, SeDuMi, SDPT3, DSDP, SDPA). Robust and efficient for general linear SDP problems. Restricted by the fact that the algorithms are second-order methods and need to store and factorize a large (and often dense) matrix.
First-order methods.
First-order methods for conic optimization avoid computing, storing and factorizing a large Hessian matrix and scale to much larger problems than interior point methods, at some cost in accuracy. A first-order method is implemented in the Splitting Cone Solver (SCS). Another first-order method is the alternating direction method of multipliers (ADMM). This method requires in every step projection on the cone of semidefinite matrices.
Bundle method.
The code ConicBundle formulates the SDP problem as a nonsmooth optimization problem and solves it by the Spectral Bundle method of nonsmooth optimization. This approach is very efficient for a special class of linear SDP problems.
Other solving methods.
Algorithms based on Augmented Lagrangian method (PENSDP) are similar in behavior to the interior point methods and can be specialized to some very large scale problems. Other algorithms use low-rank information and reformulation of the SDP as a nonlinear programming problem (SDPLR).
Approximate methods.
Algorithms that solve SDPs approximately have been proposed as well. The main goal of such methods is to achieve lower complexity in applications where approximate solutions are sufficient and complexity must be minimal. A prominent method that has been used for data detection in multiple-input multiple-output (MIMO) wireless systems is Triangular Approximate SEmidefinite Relaxation (TASER), which operates on the Cholesky decomposition factors of the semidefinite matrix instead of the semidefinite matrix. This method calculates approximate solutions for a max-cut-like problem that are often comparable to solutions from exact solvers but in only 10-20 algorithm iterations.
Applications.
Semidefinite programming has been applied to find approximate solutions to combinatorial optimization problems, such as the solution of the max cut problem with an approximation ratio of 0.87856. SDPs are also used in geometry to determine tensegrity graphs, and arise in control theory as LMIs.