Find $\mathbf{Q}$ by minimizing $\operatorname{trace}(\mathbf{Q}^\top \mathbf{AQ}$) such that $\mathbf{Q}$ is orthonormal Suppose $\mathbf{Q} \in \mathbb{R}^{m \times k}$ where $k < m$ and $\mathbf{Q}^\top\mathbf{Q} = \mathbf{I}$.
How can I find $\mathbf{Q}$ such that
$$
\operatorname{trace}(\mathbf{Q}^\top \mathbf{AQ})
$$
is minimized where $\mathbf{A}$ is positive semi-definite?
I tried the following simplification:
$$
\mathbf{Q}^\top \mathbf{AQ} = \mathbf{Q}^\top \mathbf{UD}\mathbf{U}^\top \mathbf{Q} = \mathbf{Q}^{*\top} \mathbf{DQ}^*
$$
where $\mathbf{Q}^* = \mathbf{U}^\top\mathbf{Q}$, $\mathbf{A} = \mathbf{UDU}^\top$ and $\mathbf{Q}^{*\top}\mathbf{Q}^* = \mathbf{I}$.
If I minimize this quantity, does that mean the columns of $\mathbf{Q}^*$ is given by the eigenvectors corresponding to the smallest eigenvalues of $\mathbf{A}$?

Your analysis is basically correct.  But more precisely, the answer should be like that the column vectors of the matrix $Q$ should be equal to the k unit eigenvectors corresponding to the k smallest eigenvalues of the matrix $A$.
