Bayesian optimization (BO) is a powerful paradigm for derivative-free global optimization of a black-box objective function (BOF) that is expensive to evaluate. However, the overhead of BO can still be prohibitive if the maximum number of allowed function evaluations is less than required. In this paper, we investigate how to reduce the required number of function evaluations for BO without compromise in solution quality. We explore the idea of posterior regularization for harnessing low fidelity (LF) data within the Gaussian process upper confidence bound (GP-UCB) framework. The LF data are assumed to arise from previous evaluations of an LF approximation of the BOF. An extra GP expert called LF-GP is trained to fit the LF data. We develop a dynamic weighted product of experts (DW-POE) fusion operator. The regularization is induced from this operator on the posterior of the BOF. The impact of the LF-GP expert on the resulting regularized posterior is adaptively adjusted via Bayesian formalism. Extensive experimental results on benchmark BOF optimization tasks demonstrate the superior performance of the proposed algorithm over state-of-the-art.
Multi-fidelity optimization has recently attracted considerable research interests. Techniques such as hierarchical partitioning @cite_6 , hierarchical modeling @cite_22 and ensemble methods @cite_9 , are used to incorporate multiple fidelities cheap approximations of the BOF. Most relevant to this paper is the line of work on Bayesian optimization with multi-fidelity data such as the MF-GP-UCB method in @cite_0 and the multi-fidelity BO (MFBO) algorithm in @cite_2 . Research topics that are close to MFBO in concept include multi-information source optimization @cite_14 @cite_12 , multi-task BO @cite_8 , multi-output GP @cite_1 @cite_24 , meta-learning based BO @cite_10 @cite_19 .
@cite_14 @cite_22 @cite_8 @cite_9 @cite_1 @cite_6 @cite_0 @cite_24 @cite_19 @cite_2 @cite_10 @cite_12
579827450 2101893470 2171308769 2461743311 1981081578 2963742654 1797876566 2027650277 2951893462 2127292230 2964088293 201974436
1 Model Problem and Its Discretization.- 1.1 Conservation Laws.- 1.2 Finite Volume Methods.- 2 Multiscale Setting.- 2.1 Hierarchy of Meshes.- 2.2 Motivation.- 2.3 Box Wavelet.- 2.3.1 Box Wavelet on a Cartesian Grid Hierarchy.- 2.3.2 Box Wavelet on an Arbitrary Nested Grid Hierarchy.- 2.4 Change of Stable Completion.- 2.5 Box Wavelet with Higher Vanishing Moments.- 2.5.1 Definition and Construction.- 2.5.2 A Univariate Example.- 2.5.3 A Remark on Compression Rates.- 2.6 Multiscale Transformation.- 3 Locally Refined Spaces.- 3.1 Adaptive Grid and Significant Details.- 3.2 Grading.- 3.3 Local Multiscale Transformation.- 3.4 Grading Parameter.- 3.5 Locally Uniform Grids.- 3.6 Algorithms: Encoding, Thresholding, Grading, Decoding.- 3.7 Conservation Property.- 3.8 Application to Curvilinear Grids.- 4 Adaptive Finite Volume Scheme.- 4.1 Construction.- 4.1.1 Strategies for Local Flux Evaluation.- 4.1.2 Strategies for Prediction of Details.- 4.2 A gorithms: Initial data, Prediction, Fluxes and Evolution.- 5 Error Analysis.- 5.1 Perturbation Error.- 5.2 Stability of Approximation.- 5.3 Reliability of Prediction.- 6 Data Structures and Memory Management.- 6.1 Algorithmic Requirements and Design Criteria.- 6.2 Hashing.- 6.3 Data Structures.- 7 Numerical Experiments.- 7.1 Parameter Studies.- 7.1.1 Test Configurations.- 7.1.2 Discretization.- 7.1.3 Computational Complexity and Stability.- 7.1.4 Hash Parameters.- 7.2 Real World Application.- 7.2.1 Configurations.- 7.2.2 Discretization.- 7.2.3 Discussion of Results.- A Plots of Numerical Experiments.- B The Context of Biorthogonal Wavelets.- B.1 General Setting.- B.1.1 Multiscale Basis.- B.1.2 Stable Completion.- B.1.3 Multiscale Transformation.- B.2 Biorthogonal Wavelets of the Box Function.- B.2.1 Haar Wavelets.- B.2.2 Biorthogonal Wavelets on the Real Line.- References.- List of Figures.- List of Tables.- Notation. Despite having various attractive qualities such as high prediction accuracy and the ability to quantify uncertainty and avoid over-fitting, Bayesian Matrix Factorization has not been widely adopted because of the prohibitive cost of inference. In this paper, we propose a scalable distributed Bayesian matrix factorization algorithm using stochastic gradient MCMC. Our algorithm, based on Distributed Stochastic Gradient Langevin Dynamics, can not only match the prediction accuracy of standard MCMC methods like Gibbs sampling, but at the same time is as fast and simple as stochastic gradient descent. In our experiments, we show that our algorithm can achieve the same level of prediction accuracy as Gibbs sampling an order of magnitude faster. We also show that our method reduces the prediction error as fast as distributed stochastic gradient descent, achieving a 4.1 improvement in RMSE for the Netflix dataset and an 1.8 for the Yahoo music dataset. SUMMARY This paper presents a number of algorithms to run the fast multipole method (FMM) on NVIDIA CUDA-capable graphical processing units (GPUs) (Nvidia Corporation, Sta. Clara, CA, USA). The FMM is a class of methods to compute pairwise interactions between N particles for a given error tolerance and with computational cost of O(N). The methods described in the paper are applicable to any FMMs in which the multipole-to-local (M2L) operator is a dense matrix and the matrix is precomputed. This is the case for example in the black-box fast multipole method (bbFMM), which is a variant of the FMM that can handle large class of kernels. This example will be used in our benchmarks. In the FMM, two operators represent most of the computational cost, and an optimal implementation typically tries to balance those two operators. One is the nearby interaction calculation (direct sum calculation, line 29 in Listing 1), and the other is the M2L operation. We focus on the M2L. By combining multiple M2L operations and reordering the primitive loops of the M2L so that CUDA threads can reuse or share common data, these approaches reduce the movement of data in the GPU. Because memory bandwidth is the primary bottleneck of these methods, significant performance improvements are realized. Four M2L schemes are detailed and analyzed in the case of a uniform tree. The four schemes are tested and compared with an optimized, OpenMP parallelized, multi-core CPU code. We consider high and low precision calculations by varying the number of Chebyshev nodes used in the bbFMM. The accuracy of the GPU codes is found to be satisfactory and achieved performance over 200 Gflop s on one NVIDIA Tesla C1060 GPU (Nvidia Corporation, Sta. Clara, CA, USA). This was compared against two quad-core Intel Xeon E5345 processors (Intel Corporation, Sta. Clara, CA, USA) running at 2.33 GHz, for a combined peak performance of 149 Gflop s for single precision. For the low FMM accuracy case, the observed performance of the CPU code was 37 Gflop s, whereas for the high FMM accuracy case, the performance was about 8.5 Gflop s, most likely because of a higher frequency of cache misses. We also present benchmarks on an NVIDIA C2050 GPU (a Fermi processor)(Nvidia Corporation, Sta. Clara, CA, USA) in single and double precision. Copyright © 2011 John Wiley & Sons, Ltd. We consider Multiclass and Multilabel classification with extremely large number of classes, of which only few are labeled to each instance. In such setting, standard methods that have training, prediction cost linear to the number of classes become intractable. State-of-the-art methods thus aim to reduce the complexity by exploiting correlation between labels under assumption that the similarity between labels can be captured by structures such as low-rank matrix or balanced tree. However, as the diversity of labels increases in the feature space, structural assumption can be easily violated, which leads to degrade in the testing performance. In this work, we show that a margin-maximizing loss with l1 penalty, in case of Extreme Classification, yields extremely sparse solution both in primal and in dual without sacrificing the expressive power of predictor. We thus propose a Fully-Corrective Block-Coordinate Frank-Wolfe (FC-BCFW) algorithm that exploits both primal and dual sparsity to achieve a complexity sublinear to the number of primal and dual variables. A bi-stochastic search method is proposed to further improve the efficiency. In our experiments on both Multiclass and Multilabel problems, the proposed method achieves significant higher accuracy than existing approaches of Extreme Classification with very competitive training and prediction time. This paper presents a detailed empirical study of 12 generative approaches to text clustering, obtained by applying four types of document-to-cluster assignment strategies (hard, stochastic, soft and deterministic annealing (DA) based assignments) to each of three base models, namely mixtures of multivariate Bernoulli, multinomial, and von Mises-Fisher (vMF) distributions. A large variety of text collections, both with and without feature selection, are used for the study, which yields several insights, including (a) showing situations wherein the vMF-centric approaches, which are based on directional statistics, fare better than multinomial model-based methods, and (b) quantifying the trade-off between increased performance of the soft and DA assignments and their increased computational demands. We also compare all the model-based algorithms with two state-of-the-art discriminative approaches to document clustering based, respectively, on graph partitioning (CLUTO) and a spectral coclustering method. Overall, DA and CLUTO perform the best but are also the most computationally expensive. The vMF models provide good performance at low cost while the spectral coclustering algorithm fares worse than vMF-based methods for a majority of the datasets. We consider Bayesian methods for multi-information source optimization (MISO), in which we seek to optimize an expensive-to-evaluate black-box objective function while also accessing cheaper but biased and noisy approximations ("information sources"). We present a novel algorithm that outperforms the state of the art for this problem by using a Gaussian process covariance kernel better suited to MISO than those used by previous approaches, and an acquisition function based on a one-step optimality analysis supported by efficient parallelization. We also provide a novel technique to guarantee the asymptotic quality of the solution provided by this algorithm. Experimental evaluations demonstrate that this algorithm consistently finds designs of higher value at less cost than previous approaches. Tasks such as record linkage and multi-target tracking, which involve reconstructing the set of objects that underlie some observed data, are particularly challenging for probabilistic inference. Recent work has achieved efficient and accurate inference on such problems using Markov chain Monte Carlo (MCMC) techniques with customized proposal distributions. Currently, implementing such a system requires coding MCMC state representations and acceptance probability calculations that are specific to a particular application. An alternative approach, which we pursue in this paper, is to use a general-purpose probabilistic modeling language (such as BLOG) and a generic Metropolis-Hastings MCMC algorithm that supports user-supplied proposal distributions. Our algorithm gains flexibility by using MCMC states that are only partial descriptions of possible worlds; we provide conditions under which MCMC over partial worlds yields correct answers to queries. We also show how to use a context-specific Bayes net to identify the factors in the acceptance probability that need to be computed for a given proposed move. Experimental results on a citation matching task show that our general-purpose MCMC engine compares favorably with an application-specific system. Multicell coordinated beamforming (MCBF), where multiple base stations (BSs) collaborate with each other in the beamforming design for mitigating the intercell interference (ICI), has been a subject drawing great attention recently. Most MCBF designs assume perfect channel state information (CSI) of mobile stations (MSs); however CSI errors are inevitable at the BSs in practice. Assuming elliptically bounded CSI errors, this paper studies the robust MCBF design problem that minimizes the weighted sum power of BSs subject to worst-case signal-to-interference-plus-noise ratio (SINR) constraints on the MSs. Our goal is to devise a distributed optimization method to obtain the worst-case robust beamforming solutions in a decentralized fashion with only local CSI used at each BS and limited backhaul information exchange between BSs. However, the considered problem is difficult to handle even in the centralized form. We first propose an efficient approximation method for solving the nonconvex centralized problem, using semidefinite relaxation (SDR), an approximation technique based on convex optimization. Then a distributed robust MCBF algorithm is further proposed, using a distributed convex optimization technique known as alternating direction method of multipliers (ADMM). We analytically show the convergence of the proposed distributed robust MCBF algorithm to the optimal centralized solution. We also extend the worst-case robust beamforming design as well as its decentralized implementation method to a fully coordinated scenario. Simulation results are presented to examine the effectiveness of the proposed SDR method and the distributed robust MCBF algorithm. This work deals with parallel optimization of expensive objective functions which are modeled as sample realizations of Gaussian processes. The study is formalized as a Bayesian optimization problem, or continuous multi-armed bandit problem, where a batch of q > 0 arms is pulled in parallel at each iteration. Several algorithms have been developed for choosing batches by trading off exploitation and exploration. As of today, the maximum Expected Improvement (EI) and Upper Confidence Bound (UCB) selection rules appear as the most prominent approaches for batch selection. Here, we build upon recent work on the multipoint Expected Improvement criterion, for which an analytic expansion relying on Tallis' formula was recently established. The computational burden of this selection rule being still an issue in application, we derive a closed-form expression for the gradient of the multipoint Expected Improvement, which aims at facilitating its maximization using gradient-based ascent algorithms. Substantial computational savings are shown in application. In addition, our algorithms are tested numerically and compared to state-of-the-art UCB-based batch-sequential algorithms. Combining starting designs relying on UCB with gradient-based EI local optimization finally appears as a sound option for batch design in distributed Gaussian Process optimization. Bayesian methods of matrix factorization (MF) have been actively explored recently as promising alternatives to classical singular value decomposition. In this paper, we show that, despite the fact that the optimization problem is non-convex, the global optimal solution of variational Bayesian (VB) MF can be computed analytically by solving a quartic equation. This is highly advantageous over a popular VBMF algorithm based on iterated conditional modes since it can only find a local optimal solution after iterations. We further show that the global optimal solution of empirical VBMF (hyperparameters are also learned from data) can also be analytically computed. We illustrate the usefulness of our results through experiments. Recent works have shown that exploiting multi-scale representations deeply learned via convolutional neural networks (CNN) is of tremendous importance for accurate contour detection. This paper presents a novel approach for predicting contours which advances the state of the art in two fundamental aspects, i.e. multi-scale feature generation and fusion. Different from previous works directly considering multi-scale feature maps obtained from the inner layers of a primary CNN architecture, we introduce a hierarchical deep model which produces more rich and complementary representations. Furthermore, to refine and robustly fuse the representations learned at different scales, the novel Attention-Gated Conditional Random Fields (AG-CRFs) are proposed. The experiments ran on two publicly available datasets (BSDS500 and NYUDv2) demonstrate the effectiveness of the latent AG-CRF model and of the overall hierarchical framework. Multi-view clustering, which seeks a partition of the data in multiple views that often provide complementary information to each other, has received considerable attention in recent years. In real life clustering problems, the data in each view may have considerable noise. However, existing clustering methods blindly combine the information from multi-view data with possibly considerable noise, which often degrades their performance. In this paper, we propose a novel Markov chain method for Robust Multi-view Spectral Clustering (RMSC). Our method has a flavor of lowrank and sparse decomposition, where we firstly construct a transition probability matrix from each single view, and then use these matrices to recover a shared low-rank transition probability matrix as a crucial input to the standard Markov chain method for clustering. The optimization problem of RMSC has a low-rank constraint on the transition probability matrix, and simultaneously a probabilistic simplex constraint on each of its rows. To solve this challenging optimization problem, we propose an optimization procedure based on the Augmented Lagrangian Multiplier scheme. Experimental results on various real world datasets show that the proposed method has superior performance over several state-of-the-art methods for multi-view clustering.