Are all conjugacy classes in $\text{GL}_n(\mathbb R)$ path-connected? Suppose $A$ and $B$ are conjugate invertible real $n \times n$-matrices. Does there always exist a path from $A$ to $B$ inside their conjugacy class?

I thought I had an easy proof for odd $n$ which goes as follows, but it was incorrect as pointed out in this answer. To show where my misunderstanding arised, here is the wrong argument.
Suppose there exists a real matrix $P$ such that $B = PAP^{-1}$. By replacing $P$ with $-P$ if necessary, we can assume that $\det P > 0$ (this is what goes wrong in even dimensions, see this question).
Then we have that $P = e^Q$ for some real matrix $Q$ (since the image of the exponential map is the path-component of the identity in $\text{GL}_n(\mathbb R)$). But now the path $$t \mapsto e^{tQ}Ae^{-tQ}$$
is a path connecting $A$ to $PAP^{-1} = B$.

No.
Here is a counterexample: Let $n=2$, $A=
\begin{pmatrix}
0 & -1\\
1 & 0
\end{pmatrix}$ and $B=-A=\begin{pmatrix}
0 & 1\\
-1 & 0
\end{pmatrix}$. Then, $A$ and $B$ are conjugate, since $B=C^{-1}AC$ for the
invertible matrix $C=\begin{pmatrix}
1 & 0\\
0 & -1
\end{pmatrix}$. Thus, there exists a conjugacy class $\mathcal{C}$ in
$\operatorname{GL}_{2}\left(  \mathbb{R}\right)  $ that contains
both $A$ and $B$. However, there exists no path from $A$ to $B$ in
$\mathcal{C}$. Why not?
Probably there is a nice conceptual reason [EDIT: yes, and @Wojowu explains it in his answer], but you can just as well
brute-force it: I claim that every matrix in $\mathcal{C}$ has a nonzero
$\left(  1,2\right)  $-th entry. To see this, just notice that any arbitrary
element of $\mathcal{C}$ has the form
\begin{align}
\begin{pmatrix}
a & b\\
c & d
\end{pmatrix} ^{-1}A
\begin{pmatrix}
a & b\\
c & d
\end{pmatrix}
=
\begin{pmatrix}
-\dfrac{ab+cd}{ad-bc} & -\dfrac{b^{2}+d^{2}}{ad-bc}\\
\dfrac{a^{2}+c^{2}}{ad-bc} & \dfrac{ab+cd}{ad-bc}
\end{pmatrix}
\end{align}
for some $\begin{pmatrix}
a & b\\
c & d
\end{pmatrix}
\in\operatorname{GL}_{2}\left(  \mathbb{R}\right)  $, and
thus its $\left(  1,2\right)  $-th entry $-\dfrac{b^{2}+d^{2}}{ad-bc}$ is
nonzero (because $b^{2}+d^{2}$ can only be $0$ if both $b$ and $d$ are $0$,
but then $\begin{pmatrix}
a & b\\
c & d
\end{pmatrix}$ cannot belong to $\operatorname{GL}_{2}\left(
\mathbb{R}\right)  $).
Thus, every matrix in $\mathcal{C}$ has a nonzero $\left(  1,2\right)  $-th
entry. But if there was a path from $A$ to $B$ in $\mathcal{C}$, then some
point on this path would be a matrix in $\mathcal{C}$ with zero $\left(
1,2\right)  $-th entry (since the $\left(  1,2\right)  $-th entries of $A$ and
$B$ have opposite signs). Thus, there cannot be a path from $A$ to $B$ in
$\mathcal{C}$.
