Pursuit of a Discriminative Representation for Multiple Subspaces via Sequential Games
We consider the problem of learning discriminative representations for data
in a high-dimensional space with distribution supported on or around multiple
low-dimensional linear subspaces. That is, we wish to compute a linear
injective map of the data such that the features lie on multiple orthogonal
subspaces. Instead of treating this learning problem using multiple PCAs, we
cast it as a sequential game using the closed-loop transcription (CTRL)
framework recently proposed for learning discriminative and generative
representations for general low-dimensional submanifolds. We prove that the
equilibrium solutions to the game indeed give correct representations. Our
approach unifies classical methods of learning subspaces with modern deep
learning practice, by showing that subspace learning problems may be provably
solved using the modern toolkit of representation learning. In addition, our
work provides the first theoretical justification for the CTRL framework, in
the important case of linear subspaces. We support our theoretical findings
with compelling empirical evidence. We also generalize the sequential game
formulation to more general representation learning problems. Our code,
including methods for easy reproduction of experimental results, is publically
available on GitHub.