Kernel principal component analysis (KPCA) provides a concise set of basis vectors which capture non-linear structures within large data sets, and is a central tool in data analysis and learning. To allow for non-linear relations, typically a full @math kernel matrix is constructed over @math data points, but this requires too much space and time for large values of @math . Techniques such as the Nystr "om method and random feature maps can help towards this goal, but they do not explicitly maintain the basis vectors in a stream and take more space than desired. We propose a new approach for streaming KPCA which maintains a small set of basis elements in a stream, requiring space only logarithmic in @math , and also improves the dependence on the error parameter. Our technique combines together random feature maps with recent advances in matrix sketching, it has guaranteed spectral norm error bounds with respect to the original kernel matrix, and it compares favorably in practice to state-of-the-art approaches.
Techniques of this group update augment the eigenspace of kernel PCA without storing all training data. @cite_18 adapted incremental PCA @cite_24 to maintain a set of linearly independent training data points and compute top @math eigenvectors such that they preserve a @math -fraction (for a threshold @math ) of the total energy of the eigenspace. However this method suffers from two major drawbacks. First, the set of linearly independent data points can grow large and unpredictably, perhaps exceeding the capacity of the memory. Second, under adversarial (or structured sparse) data, intermediate approximations of the eigenspace can compound in error, giving bad performance @cite_9 . Some of these issues can be addressed using online regret analysis assuming incoming data is drawn iid (e.g., @cite_25 ). However, in the adversarial settings we consider, FD @cite_0 can be seen as the right way to formally address these issues.
@cite_18 @cite_9 @cite_24 @cite_0 @cite_25
2107275435 2233419371 2951542269 2059774588
In this paper, a feature extraction method for online classification problems is presented by extending Kernel principal component analysis (KPCA). The proposed incremental KPCA (IKPCA) constructs a nonlinear high-dimensional feature space incrementally by not only updating eigen-axes but also adding new eigen-axes. The augmentation of a new eigen-axis is carried out when the accumulation ratio falls below a threshold value. We mathematically derive the incremental update equations of eigen-axes and the accumulation ratio without keeping all training samples. From the experimental results, we conclude that the proposed IKPCA works well as an incremental learning algorithm of a feature space in the sense that a minimum number of axes are augmented to maintain a designated accumulation ratio, and that the eigenvectors with major eigenvalues can converge closely to those of the batch type of KPCA. In addition, the recognition accuracy of IKPCA is similar to or slightly better than that of KPCA Matrices have become essential data representations for many large-scale problems in data analytics, and hence matrix sketching is a critical task. Although much research has focused on improving the error size tradeoff under various sketching paradigms, the many forms of error bounds make these approaches hard to compare in theory and in practice. This paper attempts to categorize and compare the most known methods under row-wise streaming updates with provable guarantees, and then to tweak some of these methods to gain practical improvements while retaining guarantees. For instance, we observe that a simple heuristic iSVD, with no guarantees, tends to outperform all known approaches in terms of size error trade-off. We modify the best performing method with guarantees, FrequentDirections , under the size error trade-off to match the performance of iSVD and retain its guarantees. We also demonstrate some adversarial datasets where iSVD performs quite poorly. In comparing techniques in the time error trade-off, techniques based on hashing or sampling tend to perform better. In this setting, we modify the most studied sampling regime to retain error guarantee but obtain dramatic improvements in the time error trade-off. Finally, we provide easy replication of our studies on APT, a new testbed which makes available not only code and datasets, but also a computing platform with fixed environmental settings. We adapt a well known streaming algorithm for approximating item frequencies to the matrix sketching setting. The algorithm receives the rows of a large matrix @math one after the other in a streaming fashion. It maintains a sketch matrix @math such that for any unit vector @math [ |Ax |^2 |Bx |^2 |Ax |^2 - |A |_ f ^2 .] Sketch updates per row in @math require @math operations in the worst case. A slight modification of the algorithm allows for an amortized update time of @math operations per row. The presented algorithm stands out in that it is: deterministic, simple to implement, and elementary to prove. It also experimentally produces more accurate sketches than widely used approaches while still being computationally competitive. A number of updates for density matrices have been developed recently that are motivated by relative entropy minimization problems. The updates involve a softmin calculation based on matrix logs and matrix exponentials. We show that these updates can be kernelized. This is important because the bounds provable for these algorithms are logarithmic in the feature dimension (provided that the 2-norm of feature vectors is bounded by a constant). The main problem we focus on is the kernelization of an online PCA algorithm which belongs to this family of updates.