What does the symbol $\operatorname{Tr}$ in the Yang-Mills action mean? I find that many authors write the Yang-Mills action as follows:
$$\mathcal{J}= \int \operatorname{Tr}(F \wedge  \star F).$$
I have yet to find a formal description of the symbol $\operatorname{Tr}$ and am especially interested in its relation to differential geometry. How should I understand it?
We choose the physics convention
$F_{\mu\nu} = \partial_\mu A_\nu - \partial_\nu A_\mu - i g [A_\mu,A_\nu]$, where the generators of the Lie algebra are Hermitian.
In this case the integral has a minus sign,
$\mathcal{J} = -\int \operatorname{Tr}(F \wedge  * F)$.

The first thing to note is that $F$ has vector and Lie algebra indices.
The trace is over the Lie algebra, not over the vector indices.
The vector indices are just like those of the field strength in QED.
In Yang-Mills the field strength (also called the curvature form) is Lie algebra valued.
In this case $F_{\mu\nu} = F_{\mu\nu}^a T^a$, where the summation convention is used, and where the $T^a$ are the generators of $\mathfrak{su}(n)$.
To be explicit, $F$ has not only tensor components but matrix components
$$(F_{\mu\nu})_{ij} = F_{\mu\nu}^a T^a_{ij}.$$

The inner product of $F$ with itself is by definition $(F,F) = \int F\wedge * F$ where $*$ is the Hodge $*$ operator.
Thus, we are calculating $\mathcal{J} = -\operatorname{Tr}(F,F)$.
It is a standard exercise to find the exterior product of two $r$-forms.
We find
$$F\wedge * F  = \frac{1}{2!} F_{\mu\nu} F^{\mu\nu} dx^1\wedge\cdots\wedge dx^4.$$
Note that the differential forms don't "know" the Lie algebra.
The algebra hasn't come into the calculation yet.

From here we must calculate the trace of $F_{\mu\nu}F^{\mu\nu}$,
$$\begin{eqnarray}
\operatorname{Tr} \left(F_{\mu\nu}F^{\mu\nu}\right)
&amp;=&amp; \operatorname{Tr} \left( F_{\mu\nu}^a T^a F^{\mu\nu b}T^b\right) \\
&amp;=&amp; \operatorname{Tr}\left(T^a T^b\right)
 F_{\mu\nu}^a F^{\mu\nu b} \\
&amp;=&amp; \frac{1}{2} \delta^{a b}
 F_{\mu\nu}^a F^{\mu\nu b} \\
&amp;=&amp; \frac{1}{2} F_{\mu\nu}^a F^{\mu\nu a},
\end{eqnarray}$$
where we have used the standard normalization convention for the $T^a$, 
$\operatorname{Tr} T^a T^b = \frac{1}{2}\delta^{ab}$. 
(This comes from the fact that we want $\mathfrak{su}(2)$ to live naturally in $\mathfrak{su}(n)$, and the generators of $\mathfrak{su}(2)$ are taken to be $T^a = \sigma^a/2$, where $\sigma^a$ are the Pauli matrices.) 
Thus, we find
$$\begin{eqnarray}
\mathcal{J} &amp;=&amp; -\int \operatorname{Tr}(F \wedge  * F) \\
&amp;=&amp; -\frac{1}{4} \int d^4x \, F_{\mu\nu}^a F^{\mu\nu a}.
\end{eqnarray}$$
Tr is the trace. Usually defined as
$$ \operatorname{Tr}\left(F_{\mu\nu}\right) = \sum_\eta F_{\eta\eta} $$
With corresponding generalizations for continuous function etc. In your case this definition should be sufficient though, as $F$ is something as the following?
$$ F_{\mu\nu} = \frac{1}{ig}\left[D_\mu,D_\nu\right] $$