Quadratic iterative system A general linear iteative system can be represented as a matrix:
$$(x,y)\mapsto(ax+by,cx+dy)$$
is essentially the same as
$$\left[\begin{array}{cc}
a&b\\
c&d\\
\end{array}\right]
\left[\begin{array}{c}
x\\
y\\
\end{array}
\right]$$
which is useful because it can be iterated quickly (matrix exponentiation) and enables various matrix techniques for determining asymptotic behavior and the like. (Of course the number of variables can be increased as needed.)
Is there a similar tool for quadratic iterative systems like
$$(x,y)\mapsto(ax^2+bxy+cy^2,dx^2+exy+fy^2)$$
? I'm interested in computing the $n$th iterate ($n$ not too small), finding asymptotic behavior, and any other interesting things that can be determined for a given collection of constants $a,b,\ldots$.
My immediate interest (genetics, oddly enough) does not use any of the diagonal terms $x^2,y^2$ so a treatment that ignores them would be fine (though I suspect including is more natural).

The short answer is no. The dynamics of linear maps is very easy to understand, as you mention, but the dynamics of nonlinear maps usually is very complicated, and there is no easy way to describe the iteration or "asymptotics".
Recall that the dynamics of the logistic map
$$ \lambda\mapsto \lambda x(1-x)$$
can be very complicated ("chaotic"), and can depend sensitively both on the starting value and on the parameter $\lambda$. 
In your setting, we can simulate this map by studying
$$(x,y)\mapsto (-\lambda x^2 + \lambda xy , y^2),$$
and using a starting value with $y=1$.
However, in the two-variable case, it may be interesting to note that we can project $\mathbb{R}^2$ to projective space (since your polynomial is homogeneous), and the iteration is semiconjugate to a one-dimensional map. More precisely, if we set $p := x/y$, then your map is semiconjugate to the quadratic rational map
$$ R(p) = \frac{ap^2+bp+c}{dp^2+ep+f}.$$
I mention this because dynamics in one variable is much, much better understood than dynamics in several variables. For example, the Hénon family, in two variables, still poses many mysteries, while the real quadratic family (in one variable) is by now rather well-understood. (Although it takes a lot of deep mathematics!)
