Intuition for orthogonal vectors in $\Bbb R^n$ Two vectors in $\Bbb R^n$ are orthogonal iff their dot product is $0$. I'm aware that the dot product can be defined in other spaces, but to keep things simple let's restrict ourselves to $\Bbb R^n$.
Given that the idea of orthogonality is roughly to identify when two vectors have no "overlap", then apart from the fact that in $\Bbb R^2$ and $\Bbb R^3$ this corresponds to the geometrical notion of orthogonality, why is this chosen as the definition of orthogonality? Ideally give examples of concrete mathematical problems where this definition arises naturally.

Firstly, let'd consider the geometry, to expand on Raskolnikov's comments:
Consider two independent vectors in $\mathbb R^n$. Then there's a unique plane through them (and the origin). Pick a linear transformation from $\mathbb R^n\to\mathbb R^2$ that sends that plane onto all of $\mathbb R^2$ and preserves lengths. Then the vectors are dot-product-orthogonal in $\mathbb R^n$ if and only if their images are geometrically-orthogonal in $\mathbb R^2$ (use the converse of Pythagorean theorem or whichever other criterion you like). This says that the geometry in $\mathbb R^n$ really acts the same way as in $\mathbb R^2$.

However, you seemed to be saying that you were not interested in the geometry side of things, and wanted an independent reason why orthogonality of vectors is important. Pretty much everything in your linear algebra book about orthogonality is an example, with one of the biggest ones being the spectral theorem, which says that every nice linear map (given symmetric matrices in the $\mathbb R^n$ case) is just a linear combination of orthogonal projections. 
Another important use of orthogonality is in the singular value decomposition, which has many applications, including image compression and topography.
