Kernel principal component analysis (KPCA) provides a concise set of basis vectors which capture non-linear structures within large data sets, and is a central tool in data analysis and learning. To allow for non-linear relations, typically a full @math kernel matrix is constructed over @math data points, but this requires too much space and time for large values of @math . Techniques such as the Nystr "om method and random feature maps can help towards this goal, but they do not explicitly maintain the basis vectors in a stream and take more space than desired. We propose a new approach for streaming KPCA which maintains a small set of basis elements in a stream, requiring space only logarithmic in @math , and also improves the dependence on the error parameter. Our technique combines together random feature maps with recent advances in matrix sketching, it has guaranteed spectral norm error bounds with respect to the original kernel matrix, and it compares favorably in practice to state-of-the-art approaches.
Techniques of this group update augment the eigenspace of kernel PCA without storing all training data. @cite_18 adapted incremental PCA @cite_24 to maintain a set of linearly independent training data points and compute top @math eigenvectors such that they preserve a @math -fraction (for a threshold @math ) of the total energy of the eigenspace. However this method suffers from two major drawbacks. First, the set of linearly independent data points can grow large and unpredictably, perhaps exceeding the capacity of the memory. Second, under adversarial (or structured sparse) data, intermediate approximations of the eigenspace can compound in error, giving bad performance @cite_9 . Some of these issues can be addressed using online regret analysis assuming incoming data is drawn iid (e.g., @cite_25 ). However, in the adversarial settings we consider, FD @cite_0 can be seen as the right way to formally address these issues.
@cite_18 @cite_9 @cite_24 @cite_0 @cite_25
2107275435 2950240291 2094900960 189214596
In this paper, a feature extraction method for online classification problems is presented by extending Kernel principal component analysis (KPCA). The proposed incremental KPCA (IKPCA) constructs a nonlinear high-dimensional feature space incrementally by not only updating eigen-axes but also adding new eigen-axes. The augmentation of a new eigen-axis is carried out when the accumulation ratio falls below a threshold value. We mathematically derive the incremental update equations of eigen-axes and the accumulation ratio without keeping all training samples. From the experimental results, we conclude that the proposed IKPCA works well as an incremental learning algorithm of a feature space in the sense that a minimum number of axes are augmented to maintain a designated accumulation ratio, and that the eigenvectors with major eigenvalues can converge closely to those of the batch type of KPCA. In addition, the recognition accuracy of IKPCA is similar to or slightly better than that of KPCA We introduce a kernel approximation strategy that enables computation of the Gaussian process log marginal likelihood and all hyperparameter derivatives in @math time. Our GRIEF kernel consists of @math eigenfunctions found using a Nystrom approximation from a dense Cartesian product grid of inducing points. By exploiting algebraic properties of Kronecker and Khatri-Rao tensor products, computational complexity of the training procedure can be practically independent of the number of inducing points. This allows us to use arbitrarily many inducing points to achieve a globally accurate kernel approximation, even in high-dimensional problems. The fast likelihood evaluation enables type-I or II Bayesian inference on large-scale datasets. We benchmark our algorithms on real-world problems with up to two-million training points and @math inducing points. We introduce an asymmetric sparse approximate embedding optimized for fast kernel comparison operations arising in large-scale visual search. In contrast to other methods that perform an explicit approximate embedding using kernel PCA followed by a distance compression technique in Rd, which loses information at both steps, our method utilizes the implicit kernel representation directly. In addition, we empirically demonstrate that our method needs no explicit training step and can operate with a dictionary of random exemplars from the dataset. We evaluate our method on three benchmark image retrieval datasets: SIFT1M, ImageNet, and 80M-TinyImages. With the growing availability of very large image databases, there has been a surge of interest in methods based on "semantic hashing", i.e. compact binary codes of data-points so that the Hamming distance between codewords correlates with similarity. In reviewing and comparing existing methods, we show that their relative performance can change drastically depending on the definition of ground-truth neighbors. Motivated by this finding, we propose a new formulation for learning binary codes which seeks to reconstruct the affinity between datapoints, rather than their distances. We show that this criterion is intractable to solve exactly, but a spectral relaxation gives an algorithm where the bits correspond to thresholded eigenvectors of the affinity matrix, and as the number of datapoints goes to infinity these eigenvectors converge to eigenfunctions of Laplace-Beltrami operators, similar to the recently proposed Spectral Hashing (SH) method. Unlike SH whose performance may degrade as the number of bits increases, the optimal code using our formulation is guaranteed to faithfully reproduce the affinities as the number of bits increases. We show that the number of eigenfunctions needed may increase exponentially with dimension, but introduce a "kernel trick" to allow us to compute with an exponentially large number of bits but using only memory and computation that grows linearly with dimension. Experiments shows that MDSH outperforms the state-of-the art, especially in the challenging regime of small distance thresholds.