Which polynomials fix the unit circle? Find all polynomials $P(x)$ with real coefficients such that for every $x,y\in \mathbb{R}$ satisfying $x^2+y^2=1$ we have

$$P(x)^2+P(y)^2=1$$
Edit: now a complete solution, though the D case is rather "hairy". It's conceptually very simple, as we basically only develop the squares and solve the equations they yield, but one has to be careful when doing this. This part is much longer than the preceding three.

We will use some basic facts about polynomials:


Two polynomials are equal iff they have the same coefficients.
The monomial of highest degree of $(\alpha_0+\alpha_1x+\cdots+\alpha_nx^n)^2$ is $\alpha_n^2x^{2n}$.
If a polynomial of degree $n$ has more than $n$ roots, it's the null polynomial. Especially true if there are infinitely many roots.


Now, let's hunt the Snark...

A. A solution $P$ is either an odd polynomial or an even polynomial

We suppose we have a polynomial $P$ satisfaying $P(x)^2+P(y)^2=1$ whenever $x^2+y^2=1$. Notice that $-P$ is also a solution.

The first thing to notice, is that for any $x\in[-1,1]$, you can find an $y$ such that $x^2+y^2=1$. There are two solutions: $y=\pm\sqrt{1-x^2}$. For such an $(x,y)$, you have an equally valid $(-x,y)$.

This means that for any $x\in[-1,1]$, there is an $y$ such that

$$P(x)^2=P(-x)^2=1-P(y)^2$$

Hence, for any $x\in [-1,1]$,

$$P(x)^2-P(-x)^2=(P(x)+P(-x))(P(x)-P(-x))=0$$

You have a product that vanishes for infinitely many $x$, so (pigeonhole principle), there is at least one factor that vanishes for infinitely many $x$.

But both factors are polynomials, so one of them is null, and

$$P(x)=P(-x)$$

or

$$P(x)=-P(-x)$$

So $P$ is either even or odd. It can't be both, since it would be null, and the null polynomial is not a solution. That is, either $P$ has only coefficients of even degree, either it has only coefficients of odd degree.

B In the even case, $P$ must be a constant.

Let's suppose the $\deg P=2n\gt 0$, and write, with $y=\sqrt{1-x^2}$,

$$P(x)^2+P(y)^2=\left(\sum_{k=0}^na_{2k} x^{2k}\right)^2+\left(\sum_{k=0}^na_{2k} (1-x^2)^k\right)^2=1$$

And with our hypothesis on degree, we must have $a_{2n}\neq0$.

Now, let's find the coefficient of highest degree of the left polynomial.

Inside the first square we have highest monomial $a_ {2n}x^{2n}$, and inside the second one, we have after expanding, $(-1)^na_{2n}x^{2n}$.

After squaring and summing, the monomial of highest degree on the left is thus $2a_{2n}^2x^{4n}$. But, it must be null, since the polynomial on the right of the equality is a constant, and we have supposed $n&gt;0$.

Contradiction! So if $P$ is even, its degree must be $0$.

Incidentally, a solution of degree $0$ satisfies $P(x)=a$, for all $x$, and

$$P(x)^2+P(y)^2=1=2a^2$$

Hence $P(x)=\frac{\sqrt2}{2}$ or $P(x)=-\frac{\sqrt2}{2}$

These are the only solutions if $P$ is even.

Notice that the same argument does not work for the odd case, because, when you write $P(x)^2+P(y)^2=1$ with $y=\sqrt{1-x^2}$, you get

$$P(x)^2+P(y)^2=\left(\sum_{k=0}^na_{2k+1} x^{2k+1}\right)^2+\left(\sum_{k=0}^na_{2k+1} \left(\sqrt{1-x^2}\right)^{2k+1}\right)^2$$
$$=\left(\sum_{k=0}^na_{2k+1} x^{2k+1}\right)^2+(1-x^2)\left(\sum_{k=0}^na_{2k+1} (1-x^2)^k\right)^2=1$$

And the coefficient of degree $4n+2$ vanishes.

C. There is an infinite sequence of solutions with odd $P$.

You have a general solution with $P(x)=T_{2n+1}(x)$, the Chebyshev polynomial of the first kind, for any $n$.

You have

$$T_{2n+1}(\cos \theta)=\cos (2n+1)\theta$$
$$T_{2n+1}(\sin \theta)=(-1)^n\sin (2n+1)\theta$$

For the cosine, it's a well known property (and sometimes the definition). For the sine it's not very difficult:

$$T_{2n+1}(\sin(\theta))=T_{2n+1}\left(\cos\left(\frac{\pi}2-\theta\right)\right)=\cos\left(\frac{(2n+1)\pi}2-(2n+1)\theta\right)$$
$$=\cos\left(\frac{(2n+1)\pi}2\right)\cos (2n+1)\theta+\sin\left(\frac{(2n+1)\pi}2\right)\sin(2n+1)\theta$$
$$=\sin\left(\frac{(2n+1)\pi}2\right)\sin(2n+1)\theta$$

And $\sin\left(\frac{(2n+1)\pi}2\right)=\sin\left(n\pi+\frac{\pi}2\right)=(-1)^n$, so

$$T_{2n+1}(\sin(\theta))=(-1)^n\sin(2n+1)\theta$$

With this $P$, you have obviously that for $x=\cos\theta$ and $y=\sin\theta$, which is always possible if $x^2+y^2=1$,

$$P(x)^2+P(y)^2=\cos^2 (2n+1)\theta+\sin^2 (2n+1)\theta=1$$

Again, the polynomials $-T_{2k+1}$ are also solutions.

D. There is no other solution for odd $P$

This one is a bit tricky.

We will need another fact about polynomials: any sequence $Q_n$ of polynomials of degree $\deg Q_n=n$, form a basis of the linear space of polynomials, $\Bbb R[X]$.

To see why, we just have to notice that then you can write $x^n$ as a linear combination of $(Q_0,Q_1,\cdots,Q_n)$, uniquely. To this end, annihilate first the monomial in $x^n$ with some $\alpha Q_n$, and continues with $x^n-\alpha Q_n$, and with subsequent differences.

It's true for Chebyshev polynomials of the first kind, and we have even a bit more: since $T_n$ is even or odd according to parity of $n$, an odd polynomial can be written uniquely as a linear combination of $\{T_{2n+1};n\in\Bbb N\}$, while an even polynomial can be written uniquely as a linear combination of $\{T_{2n};n\in\Bbb N\}$.

We will also need a similar statement about trigonometric polynomials.
If two polynomials of the form $\alpha_0+\sum_{k=1}^n \alpha_k\cos k\theta + \beta_k\sin k\theta$ are equal, then all corresponding coefficients are equal. This stems from the fact that the $\cos kx, \sin kx$ are orthogonal w.r.t the inner product $(u|v)=\frac1\pi\int_{-\pi}^\pi u(\theta)v(\theta) \,\mathrm{d}\theta$.

Thus, if you have a solution $P$ of your problem $P(x)^2+P(y)^2=1$, which is an odd polynomial, then we can write

$$P(x)=\sum_{k=0}^n a_{2k+1}x^{2k+1}$$

And, in the Chebyshev basis, there are unique $\{b_{2k+1}\}$ such that

$$P(x)=\sum_{k=0}^n b_{2k+1}T_{2k+1}(x)$$

And we can assume $a_{2n+1}\neq0$, or equivalently $b_{2n+1}\neq0$.

With $x=\cos\theta$ and $y=\sin\theta$, we have then (see the preceding part C for $T_{2k+1}(\sin\theta)$):

$$P(x)=P(\cos\theta)=\sum_{k=0}^n b_{2k+1}T_{2k+1}(\cos \theta)=\sum_{k=0}^n b_{2k+1}\cos (2k+1)\theta$$

$$P(y)=P(\sin\theta)=\sum_{k=0}^n b_{2k+1}T_{2k+1}(\sin \theta)=\sum_{k=0}^n (-1)^k b_{2k+1}\sin (2k+1)\theta$$

Write that $P$ is a solution:

$$\left(\sum_{k=0}^n b_{2k+1}\cos (2k+1)\theta\right)^2+\left(\sum_{k=0}^n (-1)^k b_{2k+1}\sin (2k+1)\theta\right)^2=1$$

Now, we are going to develop this beast carefully.

We develop the square like this:

$$(\alpha_0+\alpha_2+\cdots+\alpha_n)^2=\sum_{k=0}^n \alpha_k^2+2\sum_{i\lt j} \alpha_i\alpha_j$$

Where the second sums runs on all $i,j$ such that $0\leq i\lt j\leq n$.

For our sum of two squares, this yields

$$\sum_{k=0}^n b_{2k+1}^2\left(\cos^2 (2k+1)\theta + \sin^2 (2k+1)\theta\right)+\\
2\sum_{i\lt j} b_{2i+1}b_{2j+1} \cos (2i+1)\theta \cos (2j+1)\theta+\\
2\sum_{i\lt j} (-1)^{i+j} b_{2i+1}b_{2j+1} \sin (2i+1)\theta \sin (2j+1)\theta=1$$

We want to "linearize" this, that is use trigonometric identities to transform the expression so that there is no product of trigonometric functions. Then we will apply what we know on trigonometric polynomials, to annihilate all coefficients, in some interesting order.

The first sum, let's call it $U$, is simply a constant,

$$U=\sum_{k=0}^n b_{2k+1}^2\left(\cos^2 (2k+1)\theta + \sin^2 (2k+1)\theta\right)=\sum_{k=0}^n b_{2k+1}^2$$

The second and the third sum may be simplified by remembering that

$$\cos(a+b)=\cos a\cos b- \sin a\sin b$$
$$\cos(a-b)=\cos a\cos b+ \sin a\sin b$$

So, taking corresponding terms, according to whether $(-1)^{i+j}$ is $+1$ or $-1$, you will simplify to respectively $b_{2i+1}b_{2j+1}\cos (2j-2i)\theta$ or $b_{2i+1}b_{2j+1}\cos (2i+2j+2)\theta$.

Hence, the second and third sum in our preceding expansion, simplify to $V+W$ with

$$V=\sum_{i\lt j, \ i+j\ \mathrm{even}} b_{2i+1}b_{2j+1}\cos (2j-2i)\theta$$
$$W=\sum_{i\lt j, \ i+j\ \mathrm{odd}} b_{2i+1}b_{2j+1}\cos (2i+2j+2)\theta$$

We must be careful, because among the factors $2j-2i$ and $2i+2j+2$ in the cosines, many overlap. But we have, with $x=\cos\theta, y=\sin\theta$:

$$P(x)^2+P(y^2)=U+2V+2W=1$$

Let's have a closer look at those cosines. From now on, we will suppose $n$ is even. The proof will be similar for odd $n$. We have always $0\leq i\lt j\leq n$, and:


$U$ is a constant, so there is no cosine at all
In $V$, they appear with factor $2j-2i$, and $i+j$ is even. So it can run from $4=2\cdot2-2\cdot0$ to $2n=2\cdot n-2\cdot 0$.
In $W$, they appear with factor $2i+2j+2$, and $i+j$ is odd. So it can range from $4=2\cdot1+2\cdot0+2$ to $4n=2\cdot n+2\cdot(n-1)+2$.


Since there is no constant in $V$ or $W$, it means $U=1$ and $V$ and $W$ must be zero. Moreover, we can see that the "highest cosine", that is $\cos 4n\theta$, appears only when $i=n-1$ and $j=n$, thus with only coefficient $b_{2n+1}b_{2n-1}$.

It's here we use our hypothesis that $b_{2n+1}\neq0$, so $b_{2n-1}=0$.

The latter developments are devoted to proving that all coefficients must vanish except $b_{2n+1}$. And we will have at each step a "highest remaining cosine" (that is, cosine with the highest integer in factor of $\theta$, we will call this factor the $\theta-factor$ for short), with in its coefficients a single remaining term, even though they originally overlap. That is, by annihilating the $b_{2k+1}$ in the right order, the problem is much simplified.

To give an idea, the order for $k$ will be (for even $n$):

$$n-1,n-3,n-5,\cdots,3,1,0,2,4,6,\cdots,n-2$$

We have already proved $b_{2n-1}=0$, that's for $k=n-1$.
Notice also that we can consider only $W$, as long as our $\theta$-factor is greater than $2n$, since there is no factor beyond this one in $V$.

Let's represent our $\theta$-factors in a table, as function of indices $i$ and $j$, for $n=8$, in order to show how (and why) the following proof works.

$$\begin{array}{|c|c|c|c|c|c|c|c|c|c|}
\hline
j\backslash i  &amp; 0 &amp; 1 &amp; 2 &amp; 3 &amp; 4 &amp; 5 &amp; 6 &amp; 7 &amp; 8\\
\hline
0 &amp;  * \\
\hline
1 &amp;  4 &amp;  * \\
\hline
2 &amp;  * &amp;  8 &amp;  * \\
\hline
3 &amp;  8 &amp;  * &amp; 12 &amp;  * \\
\hline
4 &amp;  * &amp; 12 &amp;  * &amp; 16 &amp;  * \\
\hline
5 &amp; 12 &amp;  * &amp; 16 &amp;  * &amp; 20 &amp;  * \\
\hline
6 &amp;  * &amp; 16 &amp;  * &amp; 20 &amp;  * &amp; 24 &amp;  * \\
\hline
7 &amp; 16 &amp;  * &amp; 20 &amp;  * &amp; 24 &amp;  * &amp; 28 &amp;  * \\
\hline
8 &amp;  * &amp; 20 &amp;  * &amp; 24 &amp;  * &amp; 28 &amp;  * &amp;  32 &amp;  * \\
\hline
\end{array}$$

The table reads like this: for $i=2$, $j=3$, there is a $b_{2i+1}b_{2j+1}\cos k\theta$ term in $W$, with $k=12$ (the $\theta$-factor, from table). Also, elements in each growing diagonal belong to the same $k$. Hence, given that the overall coefficient before each cosine must vanish, a growing diagonal gives rise to an equation between the $b_{2i+1}$.
For example, for $i+j=9$, you get $(b_{17}b_{3}+b_{15}b_{5}+b_{13}b_{7}+b_{11}b_{9})\cos 20\theta =0$.
Since $i+j$ must be odd, all stars in the table correspond to zero coefficients, and they don't appear in $W$.

For a given $n$ like here, we can continue by hand: we have supposed $b_{2n+1}=b_{17} \neq 0$. Also we already saw that $b_{2n-1}=b_{15}=0$. It's really because $32$ is alone on its diagonal. On the next diagonal, there are two terms, $b_{17}b_{11}$ and $b_{15}b_{13}$, and the sum must be $0$. But $b_{15}=0$, so $b_{11}=0$.
And we go on, proving successively that $b_{15}$, $b_{11}$, $b_{7}$, $b_{3}$, that is the $b_{2k+1}$ for $k=7,5,3,1$, are null. Then, we are left with only even $k$, and we will attack them with $V$, a bit later.

For now, we prove this approach actually always works, for any even $n$, by induction.
We suppose $b_{2n+1}\neq 0$, as before, and also $b_{2(n-1)+1}=b_{2(n-3)+1}=\cdots=b_{2(n-2p+1)+1}=0$,
and we want to prove $b_{2(n-2p-1)+1}=0$, working on the diagonal
that contains $b_{2(n-2p-1)+1}b_{2n+1}=b_{2i_0+1}b_{2j_0+1}$, with $i_0=n-2p-1$ and $j_0=n$ (on the bottom line).

Consider all coefficients before $\cos (2i+2j+2)\theta=\cos 4(n-p)\theta$ (that is, the whole diagonal), they yield the equation:

$$\sum_{i+j=2n-2p-1} b_{2i+1}b_{2j+1}=\sum_{i=n-2p-1}^n b_{2i+1}b_{2(2n-2p-1-i)+1}=0$$

Where the sum is done on valid $i,j$ that satisfy also $0\leq i\lt j\leq n$.

We notice that $i$ and $2n-2p-1-i$ don't have the same parity (by construction, since $i+j$ must be odd), so one of them is odd, for any $i$.
If it's $i$, appart from the first that corresponds to $b_{2(n-2p-1)}$, all other are greater, so $b_{2i+1}=0$ by hypothesis.
And if it's $2n-2p-1-i$, it's also greater than $n-2p-1$, since $i\lt j\leq n$ implies $i\lt n$, so again $b_{2(2n-2p-1-i)+1}=0$, by hypothesis.
Hence, appart for the term $b_{2(n-2p-1)+1}b_{2n+1}$, all other are zero. But since the sum must vanish, and $b_{2n+1}\neq0$, we must have $b_{2(n-2p-1)+1}=0$, and the induction step is proved.

We can continue the induction as long as $4(n-p)\gt2n$, otherwise there would be terms from $V$ in the equation, and we would not be able to conclude.
That is, $p&lt;\frac{n}2$, or until $p=\frac{n}2-1$, included.

Hence it is proved that $b_{2(n-2p-1)+1}=0$ for all $p$ such that $0\leq p\leq \frac{n}2-1$.
And $n-2(\frac{n}2-1)-1=1$. Thus, simplifying a bit, we have

$$b_{2i+1}=0$$

For all odd $i$ such that $1 \leq i \leq n-1$, that is, all odd $i$.

Now, we will do the same on $V$.

Again, let's have a look at $\theta$-factors for $i,j$: the corresponding term in $V$ is 
$b_{2i+1}b_{2j+1} \cos 2(j-i)\theta$, and now $i+j$ must be even, and we have still $0\leq i \lt j \leq n$.

$$\begin{array}{|c|c|c|c|c|c|c|c|c|c|}
\hline
j\backslash i  &amp; 0 &amp; 1 &amp; 2 &amp; 3 &amp; 4 &amp; 5 &amp; 6 &amp; 7 &amp; 8\\
\hline
0 &amp;  \\
\hline
1 &amp;  * &amp;  \\
\hline
2 &amp;  4 &amp;  * &amp;  \\
\hline
3 &amp;  * &amp;  4 &amp;  * &amp;  \\
\hline
4 &amp;  8 &amp;  * &amp;  4 &amp;  * &amp;  \\
\hline
5 &amp;  * &amp;  8 &amp;  * &amp;  4 &amp;  * &amp;  \\
\hline
6 &amp; 12 &amp;  * &amp;  8 &amp;  * &amp;  4 &amp;  * &amp;  \\
\hline
7 &amp;  * &amp; 12 &amp;  * &amp;  8 &amp;  * &amp;  4 &amp;  * &amp;  \\
\hline
8 &amp; 16 &amp;  * &amp; 12 &amp;  * &amp;  8 &amp;  * &amp;  4 &amp;  * &amp;  \\
\hline
\end{array}$$

This time, $\theta$-factors are constant along decreasing diagonals, and we will start with the bottom-left corner, where we have $b_{2n+1}b_{2\cdot0+1}=0$ thus $b_1=0$.
This works for any even $n$, because $2(n-0)$ is the maximum value that can be taken by $2(j-i)$, and it appears only for $j=n$ and $i=0$, thus there is always only one term $b_{2n+1}b_{2\cdot0+1}$ before $\cos 2n\theta$.

We continue with induction, supposing that $b_{2\cdot0+1}=\cdots=b_{2(2p-2)+1}=0$, and we want to prove that $b_{2(2p)+1}=0$.
We work on the diagonal starting on the bottom line at $j_0=n$ and $i_0=2p$. The whole diagonal yields the equation

$$\sum_{i=0}^{2p} b_{2i+1}b_{2(n-2p-i)+1}=0$$

Considering $b_{2i+1}b_{2(n-2p-i)+1}$, and appart from $b_{2i_0+1}b_{2j_0+1}$, either both $i$ and $n-2p-i$ are odd,
and the term is null for we have proved earlier that $b_{2i+1}=0$ in this case (when working with $W$).
Either they are both even, and one of them is smaller than $i_0$, thus $b_{2i+1}=0$ or $b_{2(n-2p-i)+1}=0$.
Again, there is only one term remaining in the sum, and it's $b_{2i_0+1}b_{2j_0+1}=b_{2(2p)+1)}b_{2n+1}=0$.
Since we suppose $b_{2n+1}\neq0$, we must have $b_{2(2p+1)}=0$, and the induction step is proved.

We can follow the induction as long as $i_0 \lt j_0 \leq n$, that is $2p \lt n$. Hence we have proved that $b_{2i+1}=0$ for all even $i$ below $n$.

To sum up, we have proved that all $b_{2i+1}$ are null, except $b_{2n+1}$. We can now rewrite the equation $P(x)^2+P(y)^2=1$ with $x=\cos \theta$, $y=\sin\theta$ and

$$P(x)=\sum_{k=0}^n b_{2k+1}T_{2k+1}(x)=b_{2n+1}T_{2n+1}(x)$$

Or

$$P(x)=b_{2n+1}\cos (2n+1)\theta$$
$$P(y)=(-1)^nb_{2n+1}\sin(2n+1)\theta$$

Then

$$P(x)^2+P(y)^2=b_{2n+1}^2=1$$

And finally $b_{2n+1}=\pm1$.

So, we have proved that if $n$ is even and $\deg P=2n+1$, then $P=\pm T_{2n+1}$.

I'll leave the case when $n$ is odd as an "exercise", as the proof would be exactly similar, with a little modification in the order of elimination of coefficients $b_{2i+1}$.
Above it was $n-1,n-3,\cdots,3,1,0,2,\cdots,n-2$, and if $n$ is odd, it will be in the order $n-1,n-3,\cdots,2,0,1,3,\cdots,n-2$.

So, the conclusion of the whole answer is:

If $P(x)^2+P(y)^2=1$ whenever $x^2+y^2=1$, then either $P=\pm \frac{\sqrt{2}}2$, either $P=\pm T_{2n+1}$ for some $n$.
The following solution is for $1=P_n(x)+P_n(y)$, not for $1=(P_n(x))^2+(P_n(y))^2$.

Let $x=\cos z$ and $y=\sin z$ so that $x^2+y^2=\cos^2 z+\sin^2 z=1$ can be trivially satisfied.

Let $P_n(z)=\sum_{k=0}^{n} a_{2k} z^{2k}$.

$$1=P_2(\sin z)+P_2(\cos z)=2a_0+a_2 \implies a_2=1-2a_0\tag{1}$$

$$1=P_6(\sin z)+P_6(\cos z)=(1/8)(-8 + 16a_0 + 8a_2 + 6a_4 + 5a_6 + (2a_4 + 3a_6)\cos(4x))$$
$$\implies a_4 = -3 (-1 + 2 a_0 + a_2), a_6 = 2 (-1 + 2 a_0 + a_2)\tag{2}$$

$$1=P_{10}(\sin z)+P_{10}(\cos z)=(1/128)(-128 + 256a_0 + 
      63a_{10} + 128a_2 + 96a_4 + 80a_6 + 70a_8 + 4(15a_10 + 8a_4 + 12a_6 + 
           14a_8)\cos(4x) + (5a_10 + 2a_8)\cos(8x))$$
$$\implies a_6 = -2 (-5 + 10 a_0 + 5 a_2 + 2 a_4), a_8 = 
 5 (-3 + 6 a_0 + 3 a_2 + a_4), a_{10} = -2 (-3 + 6 a_0 + 3 a_2 + a_4)\tag{3}$$

Thus there seem to exist infinite many such polynomials for $P_2(z)$, $P_6(z)$, $P_{10}(z)$, or $P_{2+4k}(z),k=0,1,2,...$.