System of two quadratic equations in two variables with two parameters leads to quintic polynomial Actually, it's two closely related systems. Let $a,b \in \mathbb{Q}$ be the parameters.

The first system has the form:
$$(1+a y)x^2-2(a+y)x+(1+a y)=0 \\ (1-b x)y^2-2(b-x)y+(1-b x)=0$$
One of the solutions can be written as:
$$x=\frac{\sqrt[5]{(a+1)^2(b+1)}-\sqrt[5]{(a-1)^2(b-1)}}{\sqrt[5]{(a+1)^2(b+1)}+\sqrt[5]{(a-1)^2(b-1)}}$$
$$y=\frac{\sqrt[5]{(a-1)(b+1)^2}-\sqrt[5]{(a+1)(b-1)^2}}{\sqrt[5]{(a-1)(b+1)^2}+\sqrt[5]{(a+1)(b-1)^2}}$$

The second system has the form:
$$(1-a y)x^2+2(a+y)x-(1-a y)=0 \\ (1-b x)y^2-2(b+x)y-(1-b x)=0$$
One of the solutions can be written as:
$$x=\tan \left(\frac{1}{5} \arctan \frac{1-a^2-2ab}{(1-a^2)b+2a} \right)$$
$$y=\tan \left(\frac{1}{5} \arctan \frac{1-b^2+2ab}{(1-b^2)a-2b} \right)$$

In both cases $x,y$ are in general algebraic integers of order $5$, meaning their minimal polynomial with integer coefficients is quintic (putting aside the degenerate cases).

Is there an intuitive explanation about the reason why a system of two coupled quadratic equations leads to two separate quintic equations for each variable? Is $5$ the maximal degree of a polynomial such a system can lead to?


Just a note - I derived both solutions myself, so there is no need to show how these systems can be solved. Unless you really want to, but that's not what I'm asking.
By changing the signs we can make a lot of related systems, but most of them have rational solutions, so are not very interesting.

Edit
Actually, as @egred pointed out, these are not quadratic equations, but cubic equations (or at least cubic forms, because we have terms like $ayx^2$ and $bxy^2$ which are of course, cubic).

You're wrong in considering the two systems as consisting of quadratic equations.
The curve represented by
$$
(1+a y)x^2-2(a+y)x+(1+a y)=0
$$
is, in general, cubic (unless $a=0$). For $a\ne0$ and $b\ne0$, the first system has, properly counting multiplicities, nine solutions and it's very possible that one of the solutions consists of numbers having degree $5$ over the rationals.
To make a simpler example, Menaechmus found the duplication of the cube by intersecting two conic sections; in modern terms, he intersected
$$
\begin{cases}
xy=2
\\[4px]
y=x^2
\end{cases}
$$
finding $x^3=2$.
Similarly, the two cubics
\begin{cases}
x^2y=2
\\[4px]
y=x^3
\end{cases}
intersect in a point where $x^5=2$. Not the same form as your curves, but that's the idea.
