We address the problem of estimating statistics of hidden units in a neural network using a method of analytic moment propagation. These statistics are useful for approximate whitening of the inputs in front of saturating non-linearities such as a sigmoid function. This is important for initialization of training and for reducing the accumulated scale and bias dependencies (compensating covariate shift), which presumably eases the learning. In batch normalization, which is currently a very widely applied technique, sample estimates of statistics of hidden units over a batch are used. The proposed estimation uses an analytic propagation of mean and variance of the training set through the network. The result depends on the network structure and its current weights but not on the specific batch input. The estimates are suitable for initialization and normalization, efficient to compute and independent of the batch size. The experimental verification well supports these claims. However, the method does not share the generalization properties of BN, to which our experiments give some additional insight.
Works @cite_1 @cite_2 discuss practical and theoretical properties of normalization schemes such as the effect of the reparametrization as a preconditioner, .
@cite_1 @cite_2
2025033760 1981220107 2019588162 1980993650
An algebraic multilevel (ML) preconditioner is presented for the Helmholtz equation in heterogeneous media. It is based on a multilevel incomplete @math factorization and preserves the inherent (complex) symmetry of the Helmholtz equation. The ML preconditioner incorporates two key components for efficiency and numerical stability: symmetric maximum weight matchings and an inverse-based pivoting strategy. The former increases the block-diagonal dominance of the system, whereas the latter controls @math for numerical stability. When applied recursively, their combined effect yields an algebraic coarsening strategy, similar to algebraic multigrid methods, even for highly indefinite matrices. The ML preconditioner is combined with a Krylov subspace method and applied as a “black-box” solver to a series of challenging two- and three-dimensional test problems, mainly from geophysical seismic imaging. The numerical results demonstrate the robustness and efficiency of the ML preconditioner, even at higher frequency regimes. This article surveys preconditioning techniques for the iterative solution of large linear systems, with a focus on algebraic methods suitable for general sparse matrices. Covered topics include progress in incomplete factorization methods, sparse approximate inverses, reorderings, parallelization issues, and block and multilevel extensions. Some of the challenges ahead are also discussed. An extensive bibliography completes the paper. The construction of suitable preconditioners for the solution of linear systems by iterative methods continues to receive a lot of interest. Traditionally, preconditioners are designed to accelerate convergence of iterative methods to the solution of the linear system. However, when truncated iterative methods are used as regularized solvers of ill-posed problems, the rate of convergence is seldom an issue, and traditional preconditioners are of little use. Here, we present a new approach to the design of preconditioners for ill-posed linear systems, suitable when statistical information about the desired solution or a collection of typical solutions is available. The preconditioners are constructed from the covariance matrix of the solution viewed as a random variable. Since the construction is based on available prior information, these preconditioners are called priorconditioners. A statistical truncation index selection is also presented. Computed examples illustrate how effective such priorconditioners can be. High-quality volumetric parameterization of computational domain plays an important role in three-dimensional isogeometric analysis. Reparameterization technique can improve the distribution of isoparametric curves surfaces without changing the geometry. In this paper, using the reparameterization method, we investigate the high-quality construction of analysis-suitable NURBS volumetric parameterization. Firstly, we introduce the concept of volumetric reparameterization, and propose an optimal Mobius transformation to improve the quality of the isoparametric structure based on a new uniformity metric. Secondly, from given boundary NURBS surfaces, we present a two-stage scheme to construct the analysis-suitable volumetric parameterization: in the first step, uniformity-improved reparameterization is performed on the boundary surfaces to achieve high-quality isoparametric structure without changing the shape; in the second step, from a new variational harmonic metric and the reparameterized boundary surfaces, we construct the optimal inner control points and weights to achieve an analysis-suitable NURBS solid. Several examples with complicated geometry are presented to illustrate the effectiveness of proposed methods.