The framework of Integral Quadratic Constraints (IQC) introduced by (2014) reduces the computation of upper bounds on the convergence rate of several optimization algorithms to semi-definite programming (SDP). In particular, this technique was applied to Nesterov's accelerated method (NAM). For quadratic functions, this SDP was explicitly solved leading to a new bound on the convergence rate of NAM, and for arbitrary strongly convex functions it was shown numerically that IQC can improve bounds from Nesterov (2004). Unfortunately, an explicit analytic solution to the SDP was not provided. In this paper, we provide such an analytical solution, obtaining a new general and explicit upper bound on the convergence rate of NAM, which we further optimize over its parameters. To the best of our knowledge, this is the best, and explicit, upper bound on the convergence rate of NAM for strongly convex functions.
The work of @cite_4 focuses only on a convex quadratic function @math , and obtain @math for @math and @math , basically re-deriving .
@cite_4
2520966671
We investigate the minimization of Newton's functional for the problem of the body of minimal resistance with maximal height @math cite butt in the class of convex developable functions defined in a disc. This class is a natural candidate to find a (non-radial) minimizer in accordance with the results of cite lrp2. We prove that the minimizer in this class has a minimal set in the form of a regular polygon with @math sides centered in the disc, and numerical experiments indicate that the natural number @math is a non-decreasing function of @math . The corresponding functions all achieve a lower value of the functional than the optimal radially symmetric function with the same height @math . We develop and analyze a variant of Nesterov's accelerated gradient descent (AGD) for minimization of smooth non-convex functions. We prove that one of two cases occurs: either our AGD variant converges quickly, as if the function was convex, or we produce a certificate that the function is "guilty" of being non-convex. This non-convexity certificate allows us to exploit negative curvature and obtain deterministic, dimension-free acceleration of convergence for non-convex functions. For a function @math with Lipschitz continuous gradient and Hessian, we compute a point @math with @math in @math gradient and function evaluations. Assuming additionally that the third derivative is Lipschitz, we require only @math evaluations. We introduce a notion of quasi regularity for points with respect to the inclusion @math , where @math is a nonlinear Frechet differentiable function from @math to @math . When @math is the set of minimum points of a convex real-valued function @math on @math and @math satisfies the @math -average Lipschitz condition of Wang, we use the majorizing function technique to establish the semilocal linear quadratic convergence of sequences generated by the Gauss-Newton method (with quasi-regular initial points) for the convex composite function @math . Results are new even when the initial point is regular and @math is Lipschitz. This paper analyzes the iteration-complexity of a quadratic penalty accelerated inexact proximal point method for solving linearly constrained nonconvex composite programs. More specifically, the objective function is of the form @math where @math is a differentiable function whose gradient is Lipschitz continuous and @math is a closed convex function with bounded domain. The method, basically, consists of applying an accelerated inexact proximal point method for solving approximately a sequence of quadratic penalized subproblems associated to the linearly constrained problem. Each subproblem of the proximal point method is in turn approximately solved by an accelerated composite gradient method. It is shown that the proposed scheme generates a @math -approximate stationary point in at most @math . Finally, numerical results showing the efficiency of the proposed method are also given. Let @math XźA2r be a real curve embedded into an even-dimensional affine space. We characterise when the @math rth secant variety to @math X is an irreducible component of the algebraic boundary of the convex hull of the real points @math X(R) of @math X. This fact is then applied to @math 4-dimensional @math SO(2)-orbitopes and to the so called Barvinok---Novik orbitopes to study when they are basic closed semi-algebraic sets. In the case of @math 4-dimensional @math SO(2)-orbitopes, we find all irreducible components of their algebraic boundary. In this paper we consider a system of quadratic equations @math , where @math is unknown while normal random vectors @math and quadratic measurements @math are known. The system is assumed to be underdetermined, i.e., @math . We prove that if there exists a sparse solution @math , i.e., at most @math components of @math are nonzero, then by solving a convex optimization program, we can solve for @math up to a multiplicative constant with high probability, provided that @math . On the other hand, we prove that @math is necessary for a class of natural convex relaxations to be exact. We consider the adversarial convex bandit problem and we build the first @math -time algorithm with @math -regret for this problem. To do so we introduce three new ideas in the derivative-free optimization literature: (i) kernel methods, (ii) a generalization of Bernoulli convolutions, and (iii) a new annealing schedule for exponential weights (with increasing learning rate). The basic version of our algorithm achieves @math -regret, and we show that a simple variant of this algorithm can be run in @math -time per step at the cost of an additional @math factor in the regret. These results improve upon the @math -regret and @math -time result of the first two authors, and the @math -regret and @math -time result of Hazan and Li. Furthermore we conjecture that another variant of the algorithm could achieve @math -regret, and moreover that this regret is unimprovable (the current best lower bound being @math and it is achieved with linear functions). For the simpler situation of zeroth order stochastic convex optimization this corresponds to the conjecture that the optimal query complexity is of order @math . We propose a novel stochastic gradient method---semi-stochastic coordinate descent (S2CD)---for the problem of minimizing a strongly convex function represented as the average of a large number of smooth convex functions: @math . Our method first performs a deterministic step (computation of the gradient of @math at the starting point), followed by a large number of stochastic steps. The process is repeated a few times, with the last stochastic iterate becoming the new starting point where the deterministic step is taken. The novelty of our method is in how the stochastic steps are performed. In each such step, we pick a random function @math and a random coordinate @math ---both using nonuniform distributions---and update a single coordinate of the decision vector only, based on the computation of the @math partial derivative of @math at two different points. Each random step of the method constitutes an unbiased estimate of the gradient of @math and moreover, the squared norm of the steps goes to zero in expectation, meaning that the stochastic estimate of the gradient progressively improves. The complexity of the method is the sum of two terms: @math evaluations of gradients @math and @math evaluations of partial derivatives @math , where @math is a novel condition number. We consider the minimization of non-convex quadratic forms regularized by a cubic term, which exhibit multiple saddle points and poor local minima. Nonetheless, we prove that, under mild assumptions, gradient descent approximates the @math to within @math accuracy in @math steps for large @math and @math steps for small @math (compared to a condition number we define), with at most logarithmic dependence on the problem dimension. When we use gradient descent to approximate the Nesterov-Polyak cubic-regularized Newton step, our result implies a rate of convergence to second-order stationary points of general smooth non-convex functions. Consider a convex relaxation @math of a pseudo-boolean function @math . We say that the relaxation is totally half-integral if @math is a polyhedral function with half-integral extreme points @math , and this property is preserved after adding an arbitrary combination of constraints of the form @math , @math , and @math where @math is a constant. A well-known example is the roof duality relaxation for quadratic pseudo-boolean functions @math . We argue that total half-integrality is a natural requirement for generalizations of roof duality to arbitrary pseudo-boolean functions. Our contributions are as follows. First, we provide a complete characterization of totally half-integral relaxations @math by establishing a one-to-one correspondence with bisubmodular functions . Second, we give a new characterization of bisubmodular functions. Finally, we show some relationships between general totally half-integral relaxations and relaxations based on the roof duality. Convex and submodular functions play an important role in many applications, and in particular in combinatorial optimization. Here we study two special cases: convexity in one dimension and submodularity in two dimensions. The latter type of functions are equivalent to the well-known Monge matrices. A matrix @math is called a Monge matrix if for every @math and @math we have @math . If inequality holds in the opposite direction, then V is an inverse Monge matrix (supermodular function). Many problems, such as the traveling salesperson problem and various transportation problems, can be solved more efficiently if the input is a Monge matrix. In this work we present testing algorithms for the above properties. A testing algorithm for a predetermined property @math is given query access to an unknown function f and a distance parameter @math . The algorithm should accept f with high probability if it has the property @math and reject it with high probability if more than an @math -fraction of the function values should be modified so that f obtains the property. Our algorithm for testing whether a 1-dimensional function @math is convex (concave) has query complexity and running time of @math . Our algorithm for testing whether an n1 × n2 matrix V is a Monge (inverse Monge) matrix has query complexity and running time of @math . We prove lower bounds on the complexity of finding @math -stationary points (points @math such that @math ) of smooth, high-dimensional, and potentially non-convex functions @math . We consider oracle-based complexity measures, where an algorithm is given access to the value and all derivatives of @math at a query point @math . We show that for any (potentially randomized) algorithm @math , there exists a function @math with Lipschitz @math th order derivatives such that @math requires at least @math queries to find an @math -stationary point. Our lower bounds are sharp to within constants, and they show that gradient descent, cubic-regularized Newton's method, and generalized @math th order regularization are worst-case optimal within their natural function classes. We propose a fast methodology for encoding graphs with information-theoretically minimum numbers of bits. Specifically, a graph with property @math is called a @math -graph . If @math satisfies certain properties, then an n-node m-edge @math -graph G can be encoded by a binary string X such that (1) G and X can be obtained from each other in O(n log n) time, and (2) X has at most @math bits for any continuous superadditive function @math so that there are at most @math distinct @math -node @math -graphs. The methodology is applicable to general classes of graphs; this paper focuses on planar graphs. Examples of such @math include all conjunctions over the following groups of properties: (1) G is a planar graph or a plane graph; (2) @math is directed or undirected; (3) @math is triangulated, triconnected, biconnected, merely connected, or not required to be connected; (4) the nodes of G are labeled with labels from @math for @math ; (5) the edges of G are labeled with labels from @math for @math ; and (6) each node (respectively, edge) of G has at most @math self-loops (respectively, @math multiple edges). Moreover, @math and @math are not required to be O(1) for the cases of @math being a plane triangulation. These examples are novel applications of small cycle separators of planar graphs and are the only nontrivial classes of graphs, other than rooted trees, with known polynomial-time information-theoretically optimal coding schemes. We consider global efficiency of algorithms for minimizing a sum of a convex function and a composition of a Lipschitz convex function with a smooth map. The basic algorithm we rely on is the prox-linear method, which in each iteration solves a regularized subproblem formed by linearizing the smooth map. When the subproblems are solved exactly, the method has efficiency @math , akin to gradient descent for smooth minimization. We show that when the subproblems can only be solved by first-order methods, a simple combination of smoothing, the prox-linear method, and a fast-gradient scheme yields an algorithm with complexity @math . The technique readily extends to minimizing an average of @math composite functions, with complexity @math in expectation. We round off the paper with an inertial prox-linear method that automatically accelerates in presence of convexity. We propose a novel geometric structure induced by any given convex polygon @math , called @math , which is an arrangement of @math segments, each of which is parallel to an edge of @math , where @math denotes the number of edges of @math . This structure enjoys six simple yet nontrivial properties, which can be derived from two elementary geometric properties -- convexity and parallelism. We then apply @math to solve the geometric optimization problem of computing the parallelogram(s) in @math with the maximum area. We show that the essential nature of the locally maximal area parallelograms is captured by @math and that computing all these parallelograms reduces to answering @math location queries on @math . We answer each of these queries in @math time, and thus obtain an @math time algorithm for this optimization problem. Our algorithm avoids an explicit construction of @math , which would take @math time. We study the minimization of a convex function @math over the set of @math positive semi-definite matrices, but when the problem is recast as @math , with @math and @math . We study the performance of gradient descent on @math ---which we refer to as Factored Gradient Descent (FGD)---under standard assumptions on the original function @math . We provide a rule for selecting the step size and, with this choice, show that the local convergence rate of FGD mirrors that of standard gradient descent on the original @math : i.e., after @math steps, the error is @math for smooth @math , and exponentially small in @math when @math is (restricted) strongly convex. In addition, we provide a procedure to initialize FGD for (restricted) strongly convex objectives and when one only has access to @math via a first-order oracle; for several problem instances, such proper initialization leads to global convergence guarantees. FGD and similar procedures are widely used in practice for problems that can be posed as matrix factorization. To the best of our knowledge, this is the first paper to provide precise convergence rate guarantees for general convex functions under standard convex assumptions. The subspace approximation problem Subspace( @math , @math ) asks for a @math -dimensional linear subspace that fits a given set of points optimally, where the error for fitting is a generalization of the least squares fit and uses the @math norm instead. Most of the previous work on subspace approximation has focused on small or constant @math and @math , using coresets and sampling techniques from computational geometry. In this paper, extending another line of work based on convex relaxation and rounding, we give a polynomial time algorithm, , with the approximation guarantee roughly @math , where @math is the @math -th moment of a standard normal random variable N(0,1). We show that the convex relaxation we use has an integrality gap (or "rank gap") of @math , for any constant @math . Finally, we show that assuming the Unique Games Conjecture, the subspace approximation problem is hard to approximate within a factor better than @math , for any constant @math . For an infinite family of @math -regular LPS Ramanujan graphs, we show that the diameter of these graphs is greater than or equal to @math , where @math is an odd prime number and @math is the number of vertices. On the other hand, for any @math -regular Ramanujan graph we show that the distance of only tiny fraction of all pairs of vertices is greater than @math . We also have some numerical experiments for LPS Ramanujan graphs and random Cayley graphs which suggest that the diameters are asymptotically @math and @math , respectively. These are consistent with Sarnak's expectation on the covering exponent of universal quantum gates and our conjecture for the optimal strong approximation for quadratic forms in 4 variables. Finite convex geometries are combinatorial structures. It follows from a recent result of M. Richter and L.G. Rogers that there is an infinite set @math of planar convex polygons such that @math with respect to geometric convex hulls is a locally convex geometry and every finite convex geometry can be represented by restricting the structure of @math to a finite subset in a natural way. An @math is a differentiable convex simple closed curve @math in the plane having an inscribed circle of radius @math and a circumscribed circle of radius @math such that the ratio @math is at least @math . Motivated by Richter and Rogers' result, we construct a set @math such that (1) @math contains all points of the plane as degenerate singleton circles and all of its non-singleton members are differentiable convex simple closed planar curves; (2) @math with respect to the geometric convex hull operator is a locally convex geometry; (3) as opposed to @math , @math is closed with respect to non-degenerate affine transformations; and (4) for every (small) positive @math and for every finite convex geometry, there are continuum many pairwise affine-disjoint finite subsets @math of @math such that each @math consists of almost-circles of accuracy @math and the convex geometry in question is represented by restricting the convex hull operator to @math . The affine-disjointness of @math and @math means that, in addition to @math , even @math is disjoint from @math for every non-degenerate affine transformation @math . We consider the fundamental problem in non-convex optimization of efficiently reaching a stationary point. In contrast to the convex case, in the long history of this basic problem, the only known theoretical results on first-order non-convex optimization remain to be full gradient descent that converges in @math iterations for smooth objectives, and stochastic gradient descent that converges in @math iterations for objectives that are sum of smooth functions. We provide the first improvement in this line of research. Our result is based on the variance reduction trick recently introduced to convex optimization, as well as a brand new analysis of variance reduction that is suitable for non-convex optimization. For objectives that are sum of smooth functions, our first-order minibatch stochastic method converges with an @math rate, and is faster than full gradient descent by @math . We demonstrate the effectiveness of our methods on empirical risk minimizations with non-convex loss functions and training neural nets.