What does it mean when two functions are "orthogonal", why is it important? I have often come across the concept of orthogonality and orthogonal functions e.g in fourier series the basis functions are cos and sine, and they are orthogonal. For vectors being orthogonal means that they are actually perpendicular such that their dot product is zero. However, I am not sure how sine and cosine are actually orthogonal. They are 90 out of phase, but there must be a different reason why they are considered orthogonal. What is that reason? Does being orthognal really have something to do with geometry i.e 90 degree angels?
Why do we want to have orthogonal things so often in maths? especially with transforms like fourier transform, we want to have orthogonal basis. What does that even mean? Is there something magical about things being orthogonal?

When a function basis is orthogonal, the representation is much smaller (say, $N$ coefficients instead of $N^N$) and unique.
Suppose you have a function $f(x)$ and you want to approximate it using 3 basis functions - $g_1(x)$, $g_2(x)$, $g_3(x)$.
You try to approximate $f(x)$ using those functions: 
$$f(x) = c_1*g_1(x) + c_2*g_2(x) + c_3*g_3(x) + c_{1,2}*g_1(x)*g_2(x) + c_{2,3}*g_2(x)*g_3(x) + c_{3,1}*g_3(x)*g_1(x) + c_{1,2,3}*g_1(x)*g_2(x)*g_3(x)$$
As you see, there are quite a lot of coefficients (growing exponentially as the number of function grows). Also the expansion is not unique.
If the functions were orthogonal, then all non-single-function coefficients would be zero:
$$f(x) = c_1*g_1(x) + c_2*g_2(x) + c_3*g_3(x)$$
The set of the coefficients would also be unique for each function.
