In this paper, we study a family of non-convex and possibly non-smooth inf-projection minimization problems, where the target objective function is equal to minimization of a joint function over another variable. This problem includes difference of convex (DC) functions and a family of bi-convex functions as special cases. We develop stochastic algorithms and establish their first-order convergence for finding a (nearly) stationary solution of the target non-convex function under different conditions of the component functions. To the best of our knowledge, this is the first work that comprehensively studies stochastic optimization of non-convex inf-projection minimization problems with provable convergence guarantee. Our algorithms enable efficient stochastic optimization of a family of non-decomposable DC functions and a family of bi-convex functions. To demonstrate the power of the proposed algorithms we consider an important application in variance-based regularization, and experiments verify the effectiveness of our inf-projection based formulation and the proposed stochastic algorithm in comparison with previous stochastic algorithms based on the min-max formulation for achieving the same effect.
Recently, @cite_17 proposed a min-max formulation based on distributionally robust optimization for variance-based regularization as following: where @math is a hyper-parameter, @math , @math , and @math is called the @math -divergence based on @math . The above problem is convex-concave when the loss function @math is convex in terms of @math . It is was shown in that the above min-max formulation is equivalent to the problem ) with a proper value of @math with high probability under the assumption that the number of training examples @math is sufficiently large (see Theorem 1 and Theorem 2 in @cite_17 ).
@cite_17
2944407464
In this paper the shortest path and the minimum spanning tree problems in a graph with @math nodes and @math cost scenarios (objectives) are discussed. In order to choose a solution the min-max criterion is applied. The minmax versions of both problems are hard to approximate within @math for any @math . The best approximation algorithm for the min-max shortest path problem, known to date, has approximation ratio of @math . On the other hand, for the min-max spanning tree, there is a randomized algorithm with approximation ratio of @math . In this paper a deterministic @math -approximation algorithm for min-max shortest path is constructed. For min-max spanning tree a deterministic @math -approximation algorithm is proposed, which works for a large class of graphs and a randomized @math -approximation algorithm, which can be applied to all graphs, is constructed. It is also shown that the approximation ratios obtained are close to the integrality gaps of the corresponding LP relaxations.