How do I find the maximum volume for a box when the corners are cut out? The question reads :
A box (with no top) is to be constructed from a piece of cardboard of sides $A$ and $B$ by cutting out squares of length $h$ from the corners and folding up the sides as in the figure below: 


Suppose that the box height is $h = 3 in.$ and that it is constructed using $134 in.^2$ of cardboard (i.e., $AB = 134$). Which values $A$ and $B$ maximize the volume? 

How do I approach this question when there are the corners cut out? I understand I need to label important things with variables and find an appropriate formula, however what do I do when it comes to the corners?
How do I approach this question when there are the corners cut out? I understand I need to label important things with variables and find an appropriate formula, however what do I do when it comes to the corners?


When you remove the four corners of the cardboard, you obtain exactly the unfolded box.  The base of the box is the rectangle defined by the four inner vertices. The rest of the carboard are the folded front, back, left and right sides. If you fold up the sides the box is the open parallelepiped sketched on the right.



Given that the length of the four squares is $h=3\,\text{in}$, the base of the box is a rectangle whose lenght is $A-2h=A-6\,\text{in}$ and width is $B-2h=B-6\,\text{in}$. Therefore the base has area  $A_{\text{base}}=(A-6)(B-6)$ $\text{in}^2$.

Since $AB = 134$ $\text{in}^2$, we conclude that $B=134/A$ $\text{in}$, and 

$$A_{\text{base}}=\left(A-6\right)\left(\frac{134}{A}-6\right)=170-6A-\frac{804}{A}\text{ in}^2.$$

The hight of the folded up box is $h$ (see sketch); hence its volume is $V(A)=A_{\text{base}}\times 3\text{ in}^3$. Then

$$V(A)=3\left(170-6A-\frac{804}{A}\right)\text{in}^3.$$


  Which values $A$ and $B$ maximize the volume? 


We just need to find $V'(A)=\frac{dV}{dA}$ and solve for $A$ the equation $V'(A)=0$.
edit: $x=2h$
hint: 

$S=(A-x)(B-x)=A*B-(A+B)*x+x^2,2V=S*x$

$A*B,x$ are fixed, $V_{max} \implies S_{max} \implies (A+B)_{min}$

now the problem become when $AB$ is fixed,what is min of $A+B$,can go from here?

edit:

if the question is $A*B$ is fixed, $x$ is unknown, you want $V$ get max, then the problem is :

$f(x)=x^3-(A+B)x^2+A*B*x \le x^3-x^2*2\sqrt{A*B}+A*B*x=g(x) $

let $\sqrt{A*B}=t \to g(x)=x^3-2tx^2+t^2x ,x\le t$

you need to find $g_{max}$ 

Is it the real question?

you may just discuss $f(x)$ and keep $A+B$ and suppose $A\ge t \ge B \ge x$