Find maximum and minimum of $f(x,y) = x^2 y^2 - 2x - 2y$ in $0 \leq x \leq y \leq 5$. 
Find maximum and minimum of $f(x,y) = x^2 y^2 - 2x - 2y$ in $0 \leq x \leq y \leq 5$.

So first we need to check inside the domain, I got only one point $A(1,1)$ where $f(1,1) = -3$. and after further checking it is a saddle point.
Now we want to check on the edges, we have 3 edges: $(1) x=y, 0\leq x\leq y \leq 5$ and $(2) y=5, 0 \leq x \leq 5$ and $(3) x=0, 0\leq y \leq 5$.
So I started with each of them, using Lagrange. I started with the first and got:
$l(x,y,\lambda) = x^2y^2 - 2x - 2y + \lambda (x-y)$.
$l_x(x,y) = 2xy^2 - 2 + \lambda = 0$
$l_y(x,y) = 2x^2y - 2 - \lambda = 0 $
$x-y = 0 \rightarrow x=y$.
But then what to do ? I always get $x,y = \sqrt[3]{1+ 0.5\lambda}$, which gets me nowhere. Any help would be appreciated

I just ploted your function and wonder whether there exist max or min at all. It does.

