Find the area of overlap of two triangles Suppose we are given two triangles $ABC$ and $DEF$. We can assume nothing about them other than that they are in the same plane. The triangles may or may not overlap. I want to algorithmically determine the area (possibly $0$) of their overlap; call it $T_{common}$. 
We have a multitude of ways of determining the areas of $ABC$ and $DEF$; among the "nicest" are the Heronian formula, which is in terms of the side lengths alone, and 
$T = \frac{1}{2} \left| \det\begin{pmatrix}x_A & x_B & x_C \\ y_A & y_B & y_C \\ 1 & 1 & 1\end{pmatrix} \right| = \frac{1}{2} \big| x_A y_B - x_A y_C + x_B y_C - x_B y_A + x_C y_A - x_C y_B \big|$
which is in terms of the coordinates alone.
Obviously, there does exist a function from $A,B,C,D,E,F$ to $T_{common}$, but my question is: is there a "nice" (or even not-"nice") expression for $T_{common}$ in terms of the $x$ and $y$ coordinates of $A,B,C,D,E,F$?
I've drawn out on paper what I think are the various cases, but my issues with this approach are: identifying the case is a job in itself, which I can't easily see how to algorithmise ("just look at a picture" doesn't work for a computer); even within each case the algebra is fiddly and error-prone; and I have little confidence that I've enumerated all possible cases and got the computations right!
In my imagination there is a neat approach using ideas from analysis (treating the triangles as functions from $\mathbb{R}^2$ to $\{0,1\}$ and... multiplying them??) but I have no idea whether that's just a flight of fancy or something workable.

Here's a way to think about the problem that might help.
Each side of a triangle defines an infinite line, and this line divides the plane into two halves (called "half-spaces"). So, for example, the side $AB$ defines a line $L_{AB}$ with equation $L_{AB}(x,y) = 0$. The two corresponding half-spaces are $L_{AB}(x,y) \le 0$ and $L_{AB}(x,y) \ge 0$. You can multiply the line equation by $-1$, if necessary, and thereby arrange that the triangle actually lies in the half-space $L_{AB}(x,y) \ge 0$.
Then, the triangle $ABC$ is the intersection of three half-spaces: $L_{AB}(x,y) \ge 0$,  $L_{BC}(x,y) \ge 0$, $L_{CA}(x,y) \ge 0$.
The intersection of the two triangles is then the intersection of 6 half-spaces.
Or, saying at another way, the intersection is the set of points $(x,y)$ for which all six of $L_{AB}(x,y)$ through $L_{DF}(x,y)$ are positive. So, if we define 
$f(x,y) = \min\{L_{AB}(x,y), \ldots, L_{DF}(x,y)\}$, 
then the triangle is the set $\{(x,y) : f(x,y) \ge 0\}$.
I don't think any of this is very useful computationally. But, it provides a way to enumerate cases, and it converts the problem into a "function-based" one, as suggested by your flight of fancy.
