Learning with a convex loss function has been a dominating paradigm for many years. It remains an interesting question how non-convex loss functions help improve the generalization of learning with broad applicability. In this paper, we study a family of objective functions formed by truncating traditional loss functions, which is applicable to both shallow learning and deep learning. Truncating loss functions has potential to be less vulnerable and more robust to large noise in observations that could be adversarial. More importantly, it is a generic technique without assuming the knowledge of noise distribution. To justify non-convex learning with truncated losses, we establish excess risk bounds of empirical risk minimization based on truncated losses for heavy-tailed output, and statistical error of an approximate stationary point found by stochastic gradient descent (SGD) method. Our experiments for shallow and deep learning for regression with outliers, corrupted data and heavy-tailed noise further justify the proposed method.
Several corruption models of input @math have been considered . For example, @cite_4 considered three different corruption models, i.e., additive noise, multiplicative noise, and missing values. They proposed to minimize a non-convex quadratic objective based on estimates of @math and @math using the knowledge of noise distribution . The statistical error of the global optimum to the non-convex problem was established and it was also shown that projected gradient descent will converge in polynomial time to a small neighborhood of global minimizers.
@cite_4
2026523206
Missing and corrupted data are ubiquitous in many science and engineering domains. We analyze the information-theoretic limits of recovering sparse vectors under various models of corrupted and missing data. In particular, consider a high-dimensional linear regression model y = X β∗ + є, where y Є Rn is the response vector, X Є RnXp is a random design matrix with p ≫ n and rows distributed i.i.d. as Ν(0, Σ x ), β∗ є Rp is the unknown regression vector, and є ∼ Ν(0,σ2єΙ) is independent additive noise. Whereas a traditional approach assumes that the covariates X are fully observed, we assume only that a corrupted version Z is observed. Our main contribution is to establish minimax rates of convergence for estimating β∗ in squared l 2 -loss, assuming β∗ is k-sparse. Our upper and lower bounds in both additive noise and missing data cases scale as k log(p k) n, with prefactors depending only on the corruption and or missing pattern of the data.