When is the Rayleigh quotient (spherically) convex? Suppose that $A$ is a real, $n \times n$ symmetric matrix. Define the map $F: \mathbf{R}^n \to \mathbf{R}$ by $x \mapsto x^T A x$. 

Question: For which matrices $A$ is the map $f = F|_{S^{n-1}}$ convex? 

Let us give an interpretation for convexity in the sense of the question above. 

Definition: Let $M$ be a Riemannian manifold. A map $f: M \to \mathbf{R}$ is convex, provided that for every geodesic $\gamma: [0, 1] \to M$, and every $t \in [0, 1]$, 
  $$ f(\gamma(t)) \leq t f(\gamma(0)) + (1-t) f(\gamma(1)).$$

Let us make three remarks now. 


*

*First, this definition implies that $f \circ \gamma$ is a convex map for all geodesics $\gamma$ (check this via restricting $\gamma$, yielding yet another geodesic). 

*Secondly, when $M = \mathbf{R}^n$ with the usual flat metric, this definition reduces to 
$$
f(x_t) \leq t f(x_0) + (1-t) f(x_1), \qquad \mbox{for every $x_0, x_1 \in \mathbf{R}^n$},
$$
where above $x_t:= tx_0 + (1-t)x_1$, $t\in [0, 1]$. In other words, the usual definition of convexity for maps $\mathbf{R}^n \to \mathbf{R}$.  

*Finally, if $S^{n-1}$ is replaced by $\mathbf{R}^n$ above, then the answer to the question is simply for $A$ nonnegative definite. 

I provide a partial answer below, but I would be interested to see what people think. 

Claim: If $A$ is not a multiple of the identity, then $f$ is not convex. 



*

*Let $\lambda_i$, $e_i$ denote the (real) eigenvalue-eigenvector pairs for $A$, for $i = 1, 2, \dots, n$. 

*Suppose that $\lambda_i < \lambda_j$ for some distinct $i, j \in \{1, 2, \dots, n\}$. Let us consider the geodesics with images lying in $E_{ij} := \mathrm{span}\{e_i, e_j\} \cap S^{n-1}$. 

*For $x \in E_{ij}$, write $x = c_i e_i + c_j e_j$, for $c_i^2 + c_j^2 = 1$, and note that $f|_{E_{ij}}(x) = \lambda_i c_i^2 + \lambda_j c_j^2$. We may reparameterize this as $f|_{E_{ij}}(\theta) = \lambda_i \cos^2(\theta) + \lambda_j \sin^2(\theta)$. 

*Geodesics $\gamma$ with images in $E_{ij}$ correspond to connected intervals $I \subset [0, 2\pi)$ (of length less than $\pi$). Hence such maps $f \circ \gamma$ are restrictions of $f_{E_{ij}}$ to intervals. 

*There exists a connected interval $I$ of length less than $\pi$ on which $\cos^2(\theta)$ is neither convex nor concave. 

*Hence, if $\lambda_i < \lambda_j$, $f \circ \gamma$ is not convex for some $\gamma$ whose image lies in $E_{ij}$. Thus $f$ is not convex.  

*If no such pair $i, j$ exists, then $A = cI$ for some $c \in \mathbf{R}$. 

