Q:

When the global optimal is outside of the constraint set, what will be the demand?

$u:\mathbb R^n\to\mathbb R$ is a quasi-concave utility function so the indifference curves are convex.
$a,b\in\mathbb R^n$ are two points. Our budget set is the (one-dimensional) segment $[a,b]$ that connects $a$ and $b$.
Given: $$x^*=\arg\max_{x\in[a,b]}u(x)$$
Let $b'$ be a point in the segment $[a,x^*]$. That is: $b'=\lambda a+(1-\lambda)x^*$ for any $\lambda\in[0,1]$.
Prove that:
$$b'=\arg\max_{x\in[a,b']}u(x)$$
Graphically this result is very straight forward but I don't know how to mathematically prove it.
I think we could start of proving that $u(\lambda a+(1-\lambda) x^*)$ is monotonically decreasing with $\lambda$.
Are there named theory related?

A:

Argue that, given your assumptions on the utility function, $x^*$ is the essentially unique (and hence global) maximum. (You need this because there may be local maxima when the assumptions on the utility function is relaxed - this will violate the proposition you're trying to prove).

Now simply use the definition of global optima: for any $x\leq x^*$, $u(x)\leq u(x^*)$. This should be enough to give you the result.

