Non-convex optimization problems arise from various areas in science and engineering. Although many numerical methods and theories have been developed for unconstrained non-convex problems, the parallel development for constrained non-convex problems remains limited. That restricts the practices of mathematical modeling and quantitative decision making in many disciplines. In this paper, an inexact proximal-point penalty method is proposed for constrained optimization problems where both the objective function and the constraint can be non-convex. The proposed method approximately solves a sequence of subproblems, each of which is formed by adding to the original objective function a proximal term and quadratic penalty terms associated to the constraint functions. Under a weak-convexity assumption, each subproblem is made strongly convex and can be solved effectively to a required accuracy by an optimal gradient-type method. The theoretical property of the proposed method is analyzed in two different cases. In the first case, the objective function is non-convex but the constraint functions are assumed to be convex, while in the second case, both the objective function and the constraint are non-convex. For both cases, we give the complexity results in terms of the number of function value and gradient evaluations to produce near-stationary points. Due to the different structures, different definitions of near-stationary points are given for the two cases. The complexity for producing a nearly @math -stationary point is @math for the first case while it becomes @math for the second case.
When all constraint functions in are affine, a primal-dual Frank-Wolfe method is proposed in @cite_72 , and it finds an @math -stationary point with a complexity of @math in general and @math when there exists a strictly feasible solution. Compared to @cite_72 , this paper uses a different notion of @math -stationary point and our constraint functions can be nonlinear and non-convex.
@cite_72
2157959686 2460087882 905619 1877183374
In this paper we develop a new affine-invariant primal–dual subgradient method for nonsmooth convex optimization problems. This scheme is based on a self-concordant barrier for the basic feasible set. It is suitable for finding approximate solutions with certain relative accuracy. We discuss some applications of this technique including fractional covering problem, maximal concurrent flow problem, semidefinite relaxations and nonlinear online optimization. For all these problems, the rate of convergence of our method does not depend on the problem’s data. We give a simple proof that the Frank-Wolfe algorithm obtains a stationary point at a rate of @math on non-convex objectives with a Lipschitz continuous gradient. Our analysis is affine invariant and is the first, to the best of our knowledge, giving a similar rate to what was already proven for projected gradient methods (though on slightly different measures of stationarity). We provide stronger and more general primal-dual convergence results for Frank-Wolfe-type algorithms (a.k.a. conditional gradient) for constrained convex optimization, enabled by a simple framework of duality gap certificates. Our analysis also holds if the linear subproblems are only solved approximately (as well as if the gradients are inexact), and is proven to be worst-case optimal in the sparsity of the obtained solutions. On the application side, this allows us to unify a large variety of existing sparse greedy methods, in particular for optimization over convex hulls of an atomic set, even if those sets can only be approximated, including sparse (or structured sparse) vectors or matrices, low-rank matrices, permutation matrices, or max-norm bounded matrices. We present a new general framework for convex optimization over matrix factorizations, where every Frank-Wolfe iteration will consist of a low-rank update, and discuss the broad application areas of this approach. We propose a randomized block-coordinate variant of the classic Frank-Wolfe algorithm for convex optimization with block-separable constraints. Despite its lower iteration cost, we show that it achieves a similar convergence rate in duality gap as the full Frank-Wolfe algorithm. We also show that, when applied to the dual structural support vector machine (SVM) objective, this yields an online algorithm that has the same low iteration complexity as primal stochastic subgradient methods. However, unlike stochastic subgradient methods, the block-coordinate Frank-Wolfe algorithm allows us to compute the optimal step-size and yields a computable duality gap guarantee. Our experiments indicate that this simple algorithm outperforms competing structural SVM solvers.