Determining angle for rotation of conics I am working on rotation of conic sections and I'm having trouble determining the angle of rotation from the coefficients of the general conic equation. I'm given $$11x^2-24xy+4y^2+20=0$$ From this equation I know that $$cot(2\theta)= \frac{A-C}B=\frac{11-4}{-24}=\frac{7}{-24}$$ But when I draw a triangle with angle $2\theta$ to find $cos(2\theta)$, I end up with $$cos(2\theta)=\frac{7}{25}$$ Using this value for $cos(2\theta)$ with the half-angle formulas results in an incorrect solution, which does not eliminate the original $xy$-term. However, if I rewrite $$cot(2\theta)=\frac{-7}{24}$$ and draw a new triangle with angle $2\theta$, I end up with $$cos(2\theta)=\frac{-7}{25}$$ which yields the solution given in the back of the book after substitution and simplification. I keep encountering problems of this sort and am having trouble understanding why changing the sign of $$cot(2\theta)=\frac{7}{-24}$$ is allowed. I'm sure I am missing something with respect to the trig functions. I would be grateful for any clarification.

It may be more helpful to resolve the orientation issue using the double-angle formula for tangent, $ \ \tan(2\theta) \ = \ \frac{2 \ \tan \theta}{1 \ - \ \tan^2 \theta} \ $ .  Working with the formula for the rotation angle of the conics, you have  $ \ \tan (2 \theta) \ = \ - \frac{24}{7} \ $ (here, we don't worry about whether the numerator or the denominator should be negative), we obtain
$$  \frac{2 \ \tan \theta}{1 \ - \ \tan^2 \theta} \ \ = \ \ - \frac{24}{7} $$
$$ \Rightarrow \ \ 2 \ \tan \theta \ \ = \ \ - \frac{24}{7} \ + \ \frac{24}{7}  \ \tan^2 \theta \ \ \Rightarrow \ \ 12  \ \tan^2 \theta \ - \ 7 \tan \theta \ - \ 12 \ \ = \ \ 0 $$
$$ \ \tan \theta \ = \ \frac{4}{3} \  , \ - \frac{3}{4} \ \ . $$
The ambiguity is unavoidable, but note that this gives the slopes of two perpendicular lines, one of which is the line for the "focal axis" of the conic section, the other being the perpendicular axis to that.  This gives just two choices for eliminating the $ \ "xy" \ $ term in the conic equation.  For this conic, which proves to be a hyperbola, the focal axis is along $ \ y \ = \ \frac{4}{3} \ x \ $ . 

EDIT: It actually doesn't matter which choice you make for the slope of the line and the associated cosine and sine values, if all you want to do is eliminate the "mixed" term.  The "correct" choice will rotate the axes so that the $ \ x'- $ axis runs through the conic's focal axis [ $ \ x'^2 \ - \ 4y'^2 \ = \ 4 \ $ ], while the "other" choice puts the $ \ y'-$ axis there [ $ \ y'^2 \ - \ 4x'^2 \ = \ 4 \ $ ].  (This is a point that I can't recall seeing mentioned in at least introductory texts.  And most instructors don't have the patience to show the calculation worked out both ways...) If one desires to extract the properties of the conic, the differing alignments are not important.
EDIT 2: (fixed an algebra error in the alternate conic equation)  In view of DanielV's added comment, one can consider the four angles as rotating the conic so that the "positive" direction on its focal axis is pointed along each ray of the lines with slope $ \ \frac{4}{3} \ $ or $ \ -\frac{3}{4} \ $.
