SpecNet2: Orthogonalization-free spectral embedding by neural networks
Spectral methods which represent data points by eigenvectors of kernel
matrices or graph Laplacian matrices have been a primary tool in unsupervised
data analysis. In many application scenarios, parametrizing the spectral
embedding by a neural network that can be trained over batches of data samples
gives a promising way to achieve automatic out-of-sample extension as well as
computational scalability. Such an approach was taken in the original paper of
SpectralNet (Shaham et al. 2018), which we call SpecNet1. The current paper
introduces a new neural network approach, named SpecNet2, to compute spectral
embedding which optimizes an equivalent objective of the eigen-problem and
removes the orthogonalization layer in SpecNet1. SpecNet2 also allows
separating the sampling of rows and columns of the graph affinity matrix by
tracking the neighbors of each data point through the gradient formula.
Theoretically, we show that any local minimizer of the new
orthogonalization-free objective reveals the leading eigenvectors. Furthermore,
global convergence for this new orthogonalization-free objective using a
batch-based gradient descent method is proved. Numerical experiments
demonstrate the improved performance and computational efficiency of SpecNet2
on simulated data and image datasets.