Kaczmarz algorithm is an efficient iterative algorithm to solve overdetermined consistent system of linear equations. During each updating step, Kaczmarz chooses a hyperplane based on an individual equation and projects the current estimate for the exact solution onto that space to get a new estimate. Many vairants of Kaczmarz algorithms are proposed on how to choose better hyperplanes. Using the property of randomly sampled data in high-dimensional space, we propose an accelerated algorithm based on clustering information to improve block Kaczmarz and Kaczmarz via Johnson-Lindenstrauss lemma. Additionally, we theoretically demonstrate convergence improvement on block Kaczmarz algorithm.
The exact greedy idea has been highlighted in @cite_9 when they proposed RKA-JL algorithm. The idea is quite simple but reasonable. At the @math -th updating step, we can choose the hyperplane @math to project on where @math and update the estimate as Eq). But, after thinking of the practical issue, we may find that it is unaffordable to sweep through all rows of @math to pick the best one at each iteration. Thus, @cite_9 proposed a procedure to approximate the best hyperplane before performing each updating step, which is the key component of RKA-JL algorithm.
@cite_9
2062570725
Given a real matrix A∈Rm×n of rank r, and an integer k<r, the sum of the outer products of top k singular vectors scaled by the corresponding singular values provide the best rank-k approximation Ak to A. When the columns of A have specific meaning, it might be desirable to find good approximations to Ak which use a small number of columns of A. This paper provides a simple greedy algorithm for this problem in Frobenius norm, with guarantees on the performance and the number of columns chosen. The algorithm selects c columns from A with c=O(klogkϵ2η2(A)) such that ‖A−ΠCA‖F≤(1+ϵ)‖A−Ak‖F, where C is the matrix composed of the c columns, ΠC is the matrix projecting the columns of A onto the space spanned by C and η(A) is a measure related to the coherence in the normalized columns of A. The algorithm is quite intuitive and is obtained by combining a greedy solution to the generalization of the well known sparse approximation problem and an existence result on the possibility of sparse approximation. We provide empirical results on various specially constructed matrices comparing our algorithm with the previous deterministic approaches based on QR factorizations and a recently proposed randomized algorithm. The results indicate that in practice, the performance of the algorithm can be significantly better than the bounds suggest.