Rotation of general conic equation - angle $\theta$ quadrant I need to identify the conic represented by the equation
  $$9x^2 -6xy +y^2 - 40x -20y + 75=0$$


The book provides the solution as below: $\tan 2\theta = -3/4$

Then says take $\tan \theta = 3$. I reached this point as well.
However it then calculates

$\sin \theta = -3/{\sqrt {10}}$

$\cos \theta = -1/{\sqrt {10}}$

I was solving it assuming $\theta$ is in first quadrant, however solution shows it lies in third quadrant but not sure how it arrived at that conclusion. 

Please help if I am missing something.
The tangent function has period $\pi$, and the tangent of the double angle has period $\dfrac\pi2$. So you are free to take $\theta$ in any quadrant.

Whatever the choice, you will reduce to a parabola, and

$$9x^2-6xy+y^2=(3x-y)^2,$$ is the same as $$(-3x+y)^2.$$
Since the discriminant vanishes it is a parabola. The presence of $xy$ term  indicates that its axis of symmetry is parallel to neither axis, but inclined at $\theta$ to $x-$ axis.

Evaluation of $\theta $ from $ 2 \theta$ gives two  values of arctan function value separated by $ \pi$ from square root of quadratic equation solution, two inclinations one each for the tangent and symmetry axis corresponding to  your calculated values. 

EDIT1:

I try to elucidate fully; the given parabola equation
$$ 9x^2-6xy + y^2 -40x-20y +75=0    \tag1$$
To get rid of $xy$ term the slope (= rotation) of the axis of symmetry is $3$ ( given by OP ) and slope of tangent at vertex is its negative reciprocal $\dfrac{-1}{3}$

So let the equation of tangent be 

$$ y= \frac{-x}{3} +p \tag 2$$

where $p$ is the $y-$ axis intercept yet to be calculated.

Eliminate $y$ from above to get 

$$ x^2(9+2+1/9) + x(-6p -2p/3-40+20/3) + (p^2-20p+75) = 0 \tag3$$
simplifies to

$$ x^2(100/9) + x(-20p/3 -100/3) + (p^2-20p+75) = 0 \tag4$$

Condition for double root of tangent is that its discriminant should vanish.

$$ \left(\frac{10p+50}{3} \right)^{2}= \frac{100}{9} (p^2-20p+75)=0 \rightarrow p=\frac{5}{3} \tag5 $$

Plug 5) into 4) and simplify to get equation of tangent line as

$$x+3y=5 \tag6$$ 

Again plug 5) into 1)  and simplify

$$ [(10p +50))^2 = 100/9(p^2-2p+75) \tag7$$

or

$$p= \frac{ 5}{3} \tag8 $$

plugging it into 6) gives 

$ x_t=(2,2)$ and corresponding $y_t= (1,1) $ as it is a double (tangent/ coincident) point.

Thus the vertex point is 

$$(2,1) \tag 9 $$

Finally axis of symmetry is given by a straight line of given slope through known point $V$as

$$ y=3x - 5 \tag {10} $$

The above are all graphically depicted for the parabola with its tangent at vertex and normal which is also the axis of symmetry.