This article presents a new search algorithm for the NP-hard problem of optimizing functions of binary variables that decompose according to a graphical model. It can be applied to models of any order and structure. The main novelty is a technique to constrain the search space based on the topology of the model. When pursued to the full search depth, the algorithm is guaranteed to converge to a global optimum, passing through a series of monotonously improving local optima that are guaranteed to be optimal within a given and increasing Hamming distance. For a search depth of 1, it specializes to Iterated Conditional Modes. Between these extremes, a useful tradeoff between approximation quality and runtime is established. Experiments on models derived from both illustrative and real problems show that approximations found with limited search depth match or improve those obtained by state-of-the-art methods based on message passing and linear programming.
Third, lazy flipping with a limited search depth as a means of approximate optimization competes with message passing algorithms @cite_29 @cite_3 @cite_18 @cite_28 and with algorithms based on convex programming relaxations of the optimization problem @cite_18 @cite_7 @cite_0 @cite_8 , in particular with Tree-reweighted Belief Propagation (TRBP) @cite_13 @cite_28 @cite_21 and sub-gradient descent @cite_35 @cite_1 .
@cite_35 @cite_18 @cite_7 @cite_8 @cite_28 @cite_29 @cite_21 @cite_1 @cite_3 @cite_0 @cite_13
2069278600 2895571900 2146989110 562372631
We present a new efficient algorithm for the search version of the approximate Closest Vector Problem with Preprocessing (CVPP). Our algorithm achieves an approximation factor of O(n sqrt log n ), improving on the previous best of O(n^ 1.5 ) due to Lag arias, Lenstra, and Schnorr hkzbabai . We also show, somewhat surprisingly, that only O(n) vectors of preprocessing advice are sufficient to solve the problem (with the slightly worse approximation factor of O(n)). We remark that this still leaves a large gap with respect to the decisional version of CVPP, where the best known approximation factor is O(sqrt n log n ) due to Aharonov and Regev AharonovR04 . To achieve these results, we show a reduction to the same problem restricted to target points that are close to the lattice and a more efficient reduction to a harder problem, Bounded Distance Decoding with preprocessing (BDDP). Combining either reduction with the previous best-known algorithm for BDDP by Liu, Lyubashevsky, and Micciancio LiuLM06 gives our main result. In the setting of CVP without preprocessing, we also give a reduction from (1+eps)gamma approximate CVP to gamma approximate CVP where the target is at distance at most 1+1 eps times the minimum distance (the length of the shortest non-zero vector) which relies on the lattice sparsification techniques of Dadush and Kun DadushK13 . As our final and most technical contribution, we present a substantially more efficient variant of the LLM algorithm (both in terms of run-time and amount of preprocessing advice), and via an improved analysis, show that it can decode up to a distance proportional to the reciprocal of the smoothing parameter of the dual lattice MR04 . We show that this is never smaller than the LLM decoding radius, and that it can be up to an wide tilde Omega (sqrt n ) factor larger. We consider the problem of finding an approximate second-order stationary point of a constrained non-convex optimization problem. We first show that, unlike the unconstrained scenario, the vanilla projected gradient descent algorithm may converge to a strict saddle point even when there is only a single linear constraint. We then provide a hardness result by showing that checking ( , )-second order stationarity is NP-hard even in the presence of linear constraints. Despite our hardness result, we identify instances of the problem for which checking second order stationarity can be done efficiently. For such instances, we propose a dynamic second order Frank--Wolfe algorithm which converges to ( , )-second order stationary points in O ( ^ -2 , ^ -3 ) iterations. The proposed algorithm can be used in general constrained non-convex optimization as long as the constrained quadratic sub-problem can be solved efficiently. A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum. Here we argue, based on results from statistical physics, random matrix theory, neural network theory, and empirical evidence, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest. Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum. Motivated by these arguments, we propose a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods. We apply this algorithm to deep or recurrent neural network training, and provide numerical evidence for its superior optimization performance. 1. Automatic code generation for real-time convex optimization J. Mattingley and S. Boyd 2. Gradient-based algorithms with applications to signal recovery problems A. Beck and M. Teboulle 3. Graphical models of autoregressive processes J. Songsiri, J. Dahl and L. Vandenberghe 4. SDP relaxation of homogeneous quadratic optimization Z. Q. Luo and T. H. Chang 5. Probabilistic analysis of SDR detectors for MIMO systems A. Man-Cho So and Y. Ye 6. Semidefinite programming, matrix decomposition, and radar code design Y. Huang, A. De Maio and S. Zhang 7. Convex analysis for non-negative blind source separation with application in imaging W. K. Ma, T. H. Chan, C. Y. Chi and Y. Wang 8. Optimization techniques in modern sampling theory T. Michaeli and Y. C. Eldar 9. Robust broadband adaptive beamforming using convex optimization M. Rubsamen, A. El-Keyi, A. B. Gershman and T. Kirubarajan 10. Cooperative distributed multi-agent optimization A. Nenadic and A. Ozdaglar 11. Competitive optimization of cognitive radio MIMO systems via game theory G. Scutari, D. P. Palomar and S. Barbarossa 12. Nash equilibria: the variational approach F. Facchinei and J. S. Pang.