Bayesian optimization (BO) is a powerful paradigm for derivative-free global optimization of a black-box objective function (BOF) that is expensive to evaluate. However, the overhead of BO can still be prohibitive if the maximum number of allowed function evaluations is less than required. In this paper, we investigate how to reduce the required number of function evaluations for BO without compromise in solution quality. We explore the idea of posterior regularization for harnessing low fidelity (LF) data within the Gaussian process upper confidence bound (GP-UCB) framework. The LF data are assumed to arise from previous evaluations of an LF approximation of the BOF. An extra GP expert called LF-GP is trained to fit the LF data. We develop a dynamic weighted product of experts (DW-POE) fusion operator. The regularization is induced from this operator on the posterior of the BOF. The impact of the LF-GP expert on the resulting regularized posterior is adaptively adjusted via Bayesian formalism. Extensive experimental results on benchmark BOF optimization tasks demonstrate the superior performance of the proposed algorithm over state-of-the-art.
Multi-fidelity optimization has recently attracted considerable research interests. Techniques such as hierarchical partitioning @cite_6 , hierarchical modeling @cite_22 and ensemble methods @cite_9 , are used to incorporate multiple fidelities cheap approximations of the BOF. Most relevant to this paper is the line of work on Bayesian optimization with multi-fidelity data such as the MF-GP-UCB method in @cite_0 and the multi-fidelity BO (MFBO) algorithm in @cite_2 . Research topics that are close to MFBO in concept include multi-information source optimization @cite_14 @cite_12 , multi-task BO @cite_8 , multi-output GP @cite_1 @cite_24 , meta-learning based BO @cite_10 @cite_19 .
@cite_14 @cite_22 @cite_8 @cite_9 @cite_1 @cite_6 @cite_0 @cite_24 @cite_19 @cite_2 @cite_10 @cite_12
579827450 2101893470 2171308769 2461743311
1 Model Problem and Its Discretization.- 1.1 Conservation Laws.- 1.2 Finite Volume Methods.- 2 Multiscale Setting.- 2.1 Hierarchy of Meshes.- 2.2 Motivation.- 2.3 Box Wavelet.- 2.3.1 Box Wavelet on a Cartesian Grid Hierarchy.- 2.3.2 Box Wavelet on an Arbitrary Nested Grid Hierarchy.- 2.4 Change of Stable Completion.- 2.5 Box Wavelet with Higher Vanishing Moments.- 2.5.1 Definition and Construction.- 2.5.2 A Univariate Example.- 2.5.3 A Remark on Compression Rates.- 2.6 Multiscale Transformation.- 3 Locally Refined Spaces.- 3.1 Adaptive Grid and Significant Details.- 3.2 Grading.- 3.3 Local Multiscale Transformation.- 3.4 Grading Parameter.- 3.5 Locally Uniform Grids.- 3.6 Algorithms: Encoding, Thresholding, Grading, Decoding.- 3.7 Conservation Property.- 3.8 Application to Curvilinear Grids.- 4 Adaptive Finite Volume Scheme.- 4.1 Construction.- 4.1.1 Strategies for Local Flux Evaluation.- 4.1.2 Strategies for Prediction of Details.- 4.2 A gorithms: Initial data, Prediction, Fluxes and Evolution.- 5 Error Analysis.- 5.1 Perturbation Error.- 5.2 Stability of Approximation.- 5.3 Reliability of Prediction.- 6 Data Structures and Memory Management.- 6.1 Algorithmic Requirements and Design Criteria.- 6.2 Hashing.- 6.3 Data Structures.- 7 Numerical Experiments.- 7.1 Parameter Studies.- 7.1.1 Test Configurations.- 7.1.2 Discretization.- 7.1.3 Computational Complexity and Stability.- 7.1.4 Hash Parameters.- 7.2 Real World Application.- 7.2.1 Configurations.- 7.2.2 Discretization.- 7.2.3 Discussion of Results.- A Plots of Numerical Experiments.- B The Context of Biorthogonal Wavelets.- B.1 General Setting.- B.1.1 Multiscale Basis.- B.1.2 Stable Completion.- B.1.3 Multiscale Transformation.- B.2 Biorthogonal Wavelets of the Box Function.- B.2.1 Haar Wavelets.- B.2.2 Biorthogonal Wavelets on the Real Line.- References.- List of Figures.- List of Tables.- Notation. Despite having various attractive qualities such as high prediction accuracy and the ability to quantify uncertainty and avoid over-fitting, Bayesian Matrix Factorization has not been widely adopted because of the prohibitive cost of inference. In this paper, we propose a scalable distributed Bayesian matrix factorization algorithm using stochastic gradient MCMC. Our algorithm, based on Distributed Stochastic Gradient Langevin Dynamics, can not only match the prediction accuracy of standard MCMC methods like Gibbs sampling, but at the same time is as fast and simple as stochastic gradient descent. In our experiments, we show that our algorithm can achieve the same level of prediction accuracy as Gibbs sampling an order of magnitude faster. We also show that our method reduces the prediction error as fast as distributed stochastic gradient descent, achieving a 4.1 improvement in RMSE for the Netflix dataset and an 1.8 for the Yahoo music dataset. SUMMARY This paper presents a number of algorithms to run the fast multipole method (FMM) on NVIDIA CUDA-capable graphical processing units (GPUs) (Nvidia Corporation, Sta. Clara, CA, USA). The FMM is a class of methods to compute pairwise interactions between N particles for a given error tolerance and with computational cost of O(N). The methods described in the paper are applicable to any FMMs in which the multipole-to-local (M2L) operator is a dense matrix and the matrix is precomputed. This is the case for example in the black-box fast multipole method (bbFMM), which is a variant of the FMM that can handle large class of kernels. This example will be used in our benchmarks. In the FMM, two operators represent most of the computational cost, and an optimal implementation typically tries to balance those two operators. One is the nearby interaction calculation (direct sum calculation, line 29 in Listing 1), and the other is the M2L operation. We focus on the M2L. By combining multiple M2L operations and reordering the primitive loops of the M2L so that CUDA threads can reuse or share common data, these approaches reduce the movement of data in the GPU. Because memory bandwidth is the primary bottleneck of these methods, significant performance improvements are realized. Four M2L schemes are detailed and analyzed in the case of a uniform tree. The four schemes are tested and compared with an optimized, OpenMP parallelized, multi-core CPU code. We consider high and low precision calculations by varying the number of Chebyshev nodes used in the bbFMM. The accuracy of the GPU codes is found to be satisfactory and achieved performance over 200 Gflop s on one NVIDIA Tesla C1060 GPU (Nvidia Corporation, Sta. Clara, CA, USA). This was compared against two quad-core Intel Xeon E5345 processors (Intel Corporation, Sta. Clara, CA, USA) running at 2.33 GHz, for a combined peak performance of 149 Gflop s for single precision. For the low FMM accuracy case, the observed performance of the CPU code was 37 Gflop s, whereas for the high FMM accuracy case, the performance was about 8.5 Gflop s, most likely because of a higher frequency of cache misses. We also present benchmarks on an NVIDIA C2050 GPU (a Fermi processor)(Nvidia Corporation, Sta. Clara, CA, USA) in single and double precision. Copyright © 2011 John Wiley & Sons, Ltd. We consider Multiclass and Multilabel classification with extremely large number of classes, of which only few are labeled to each instance. In such setting, standard methods that have training, prediction cost linear to the number of classes become intractable. State-of-the-art methods thus aim to reduce the complexity by exploiting correlation between labels under assumption that the similarity between labels can be captured by structures such as low-rank matrix or balanced tree. However, as the diversity of labels increases in the feature space, structural assumption can be easily violated, which leads to degrade in the testing performance. In this work, we show that a margin-maximizing loss with l1 penalty, in case of Extreme Classification, yields extremely sparse solution both in primal and in dual without sacrificing the expressive power of predictor. We thus propose a Fully-Corrective Block-Coordinate Frank-Wolfe (FC-BCFW) algorithm that exploits both primal and dual sparsity to achieve a complexity sublinear to the number of primal and dual variables. A bi-stochastic search method is proposed to further improve the efficiency. In our experiments on both Multiclass and Multilabel problems, the proposed method achieves significant higher accuracy than existing approaches of Extreme Classification with very competitive training and prediction time.