What is the minimum value of $8 \cos^2 x + 18 \sec^2 x$? As per me the answer should be $26$.
But when we apply AM-GM inequality it gives $24$ as the least value but as per the graph 24 can never come.
What I think is that in AM-GM, it gives $8 \cos^2 x = 18 \sec^2 x$ which gives $\cos x > 1$ which is not possible and because of this, AM-GM is giving a wrong minimum value.
If we had $18 \cos^2 x + 8 \sec^2 x$, then AM-GM would have worked and $24$ would be a right answer since $18 \cos^2 x = 8 \sec^2 x$, which gives $\cos x < 1$ which is true.
Is this reason correct?

No need for AM-GM.
Differentiate wrt $x$ and set $f'(x)=0$ as follows:
$$\implies -16\sin x\cdot \cos x + 36\sec^2 x\cdot \tan x=0$$
$$4\cos x\sin x=\frac{9\sin x}{\cos^3 x}$$
If $\sin x$ is non-zero, then:
$$\cos^4 x=\frac{9}{4}\implies \cos x=±\sqrt{\frac{3}{2}}>1\implies \text{no solution}$$
Hence $\sin x=0\implies \cos x=±1$ and hence $\cos^2 x=1$.
Substitute this into your original equation and you get $8+18=26.$
NOTE: You can confirm that this is a minima by evaluating $f''(x).$
