Problem while using divergence theorem Evaluate $$\iint_S \mathbf A\cdot n \ \mathrm d S$$
where $\mathbf A=y\mathbf i+2x\mathbf j-z\mathbf k$ and $S$ is the surface of the plane $2x+y=6$ in the first octant cut off by the plane $z=4$
I was doing this problem then $\operatorname{div} A = -1$ . So I got eventually integration of $\mathrm dz\mathrm dy\mathrm dx$ where $z$ varies from $4$ to $0$, $y=  6- 2x$ to $0$ and $x$ from $3$ to $0$. By this I got $-36$ whereas the answer is $108$. Can someone spot where is my mistake?

The divergence theorem applies to the flux through a closed surface. You have only one piece of a plane (not a closed surface). You would need to create a volume instead. Add surfaces where calculating the integral might be easier (such as the $xy, xz, yz$ sides), to create a closed surface.
There are two ways to solve this problem, and both involve calculating surface integrals.
Direct way: $n$ for the given plane is proportional to $2\mathbf i+\mathbf j$. To make it a unit vector, just divide by the norm, to get $$n=\frac{1}{\sqrt 5}(2\mathbf i+\mathbf j)$$
The surface element in the plane is $ds=dl\cdot dz$, where $dl$ varies along the intersection of the $2x+y=6$ plane and $z=0$ plane. You can write this in terms of $dx$ as $dl=\sqrt 5 dx$ (just draw a picture). Now, using $y=6-2x$, $$\iint_S \mathbf A\cdot n \ \mathrm d S=\iint_S(y\mathbf i+2x\mathbf j-z\mathbf k)\frac{1}{\sqrt 5}(2\mathbf i+\mathbf j)dl\cdot dz=\iint_S(2(6-2x)+2x)dx\cdot dz\\=\int_0^4dz\int_0^3(12-2x)dx=4\cdot12\cdot3-4\cdot2\cdot\frac{3^2}{2}=108$$
Using divergence theorem:
We need to create a closed surface. You can choose $S$, together with $x=0$, $y=0$, $z=0$, and $z=4$ planes, obviously only in the intersection regions. That means for $x=0$ plane we choose $0\le z\le4$ and $0\le y\le 6$; for the $y=0$ plane $0\le x\le 3$ and $0\le z\le 4$; and for the $z=0$ and $z=4$ planes, we have $0\le x\le 3$ and $0\le y\le6-2x$. The divergence theorem states that the sum of the surface integrals through all these surfaces equal to the volume integral of the divergence.$$\iint_S \mathbf A\cdot n \ \mathrm d S+\iint_{x=0} \mathbf A\cdot n \ \mathrm d S+\iint_{y=0} \mathbf A\cdot n \ \mathrm d S+\iint_{z=0} \mathbf A\cdot n \ \mathrm d S+\iint_{z=4} \mathbf A\cdot n \ \mathrm d S=\iiint_V\nabla A dV$$
So we would need to calculate all these integrals to get the first one.
At $x=0$ surface, the normal points in the $-\mathbf i$ direction, so $\mathbf A\cdot n=-y$.$$\iint_{x=0} \mathbf A\cdot n \ \mathrm d S=\int_0^4 dz\int_0^6 dy(-y)=-72$$
At $y=0$, $n=-\mathbf j$, $\mathbf A\cdot n=-2x$, so $$\iint_{y=0} \mathbf A\cdot n \ \mathrm d S=\int_0^4 dz\int_0^3 dx(-2x)=-36$$
At $z=0$, $n=-\mathbf k$, $\mathbf A\cdot n=z=0$ so $$\iint_{z=0} \mathbf A\cdot n \ \mathrm d S=0$$
At $z=4$,  $n=+\mathbf k$, $\mathbf A\cdot n=-z=-4$ and $$\iint_{z=4} \mathbf A\cdot n \ \mathrm d S=-4\int_0^3dx\int_0^{6-2x}dy=-4\int_0^3dx(6-2x)=-36$$
We now use the result for the volume integral $$\iiint_V\nabla A dV=-36$$ to get $$\iint_S \mathbf A\cdot n \ \mathrm d S-72-36+0-36=-36$$ which yields $$\iint_S \mathbf A\cdot n \ \mathrm d S=108$$
