Q:

If $A$ and $B$ are linear transformations on a finite-dimensional inner product space, and if $\textbf{0} \leq A \leq B$, then det $A \leq$ det $B$.

Exercise 12 from SEC. 82 of Finite-Dimensional Vector Spaces - 2nd Edition by Paul R. Halmos.

If $A$ and $B$ are linear transformations on a finite-dimensional inner product space, and if $\textbf{0} \le A \leq B$, then $\det A \leq \det B$.

Notation: $0 \le A$ means that $A$ is positive.

If $\det B = 0$, choose $x \in \ker B$, then $0 \le ((B-A)x,x) = (Bx, x) - (Ax, x) = -(Ax, x) \le 0$. Therefore $\ker B \subseteq \ker A$ and $0 = det A \le det B = 0$.
How to prove for the case when $B$ is invertible? Intuitively, I feel that since both $A$ and $B$ are positive their proper values are also positive and $B$'s proper values should be greater the $A$'s and same for determinants.

A:

Hint: Reduce your inequality to a simpler case as follows:
$$
0 \preceq A \preceq B \iff 0 \preceq B^{-1/2}AB^{-1/2} \preceq B^{-1/2}BB^{-1/2}
$$

