Show that the set of extreme points of $S$ is its boundary Let $S:=\{x:x^tx\leq 1\}$.
Show that the set of extreme points of $S$ is its boundary.
An extreme point, in mathematics, is a point in a convex set which does not lie in any open line segment joining two points in the set

Suppose $a,b \in S$ and $a \neq b$, then $\langle a, b \rangle <1$.
Suppose $x \in \operatorname{ri} [a,b]$ with $a,b \in S$, then we can write $x=ta+(1-t)b$ with $t \in (0,1)$ and
$\|x\|^2 = t^2\|a\|^2 +2t(1-t) \langle a, b \rangle +  (1-t)^2\|b\|^2  < t^2 +2t(1-t)  +  (1-t)^2= 1$.
In particular, if $\|x\| =1$ then $x$ cannot line in the relative interior of a line segment whose end points are in $S$ and so must be an extreme point.
An extreme point cannot lie in the interior of $S$ and so must have unit norm.
