What does $ ( \nabla u) \circ \tau \cdot D \tau $ and $ \nabla u \cdot (D \tau_\gamma)^{-1} $ mean? To understand the question here
$\def\abs#1{\left|#1\right|}$
\begin{align*}
  F(u_\gamma) &= F(u \circ \tau_\gamma^{-1})\\
        &= \int_\Omega \abs{\nabla(u \circ \tau_\gamma^{-1})}^2\\
        &= \int_\Omega \abs{(\nabla u) \circ \tau_\gamma^{-1} \cdot D\tau_\gamma^{-1}}^2\\
        &= \int_{\tau_\gamma^{-1}\Omega} \abs{(\nabla u) \circ \tau_\gamma^{-1}\circ \tau_\gamma\cdot D\tau_\gamma^{-1}\circ \tau_\gamma}^2\abs{\det(D\tau_\gamma)}\\
        &= \int_\Omega \abs{\nabla u\cdot (D\tau_\gamma)^{-1}}^2\abs{\det(D\tau_\gamma)}
\end{align*}
I know that by chain rule $ \cdots $ componentwise we have
$$ \partial_i ( u \circ \tau) =   \sum_{j}  (\partial_j u)  \circ  \tau \cdot \partial_i \tau_j. $$
Thus, $ \nabla ( u \circ\tau )= ( \nabla u) \circ \tau \cdot D \tau $. I'd like to understand this equality or this notaition. I know that
\begin{equation}
\nabla u = (\partial_1 u, \partial_2 u, \cdots , \partial_n u) 
\end{equation}
and  I guess that
$$ D \tau = \left[
\begin{array}{cccc}
\partial_1 \tau_1 & \partial_2 \tau_1 & \cdots & \partial_n \tau_1\\
\partial_1 \tau_2 & \partial_2 \tau_2 & \cdots & \partial_n \tau_2\\
\vdots & \vdots & \ddots & \vdots\\
\partial_1 \tau_n & \partial_2 \tau_n & \cdots & \partial_n \tau_n\\
\end{array}
\right] $$
Then, what does $ ( \nabla u) \circ \tau \cdot D \tau$ mean?  And what does $ \nabla u \cdot (D \tau_\gamma)^{-1} $ mean?


$\nabla ( u \circ\tau )= ( \nabla u) \circ \tau \cdot D \tau$. I'd like to understand this equality or this notaition. 

Think of the chain rule: derivative of composition is the product of derivatives. On the left, $u\circ \tau $ is composition (not Hadamard product, as suggested in the other answer). On the right, we have a product of $( \nabla u) \circ \tau$ (which is a vector) and $D \tau$ (which is a matrix); this is the usual application of matrix to a vector, except that the vector, being written as a row, appears to the left of the matrix. It is not necessary to use the dot here: $ (( \nabla u) \circ \tau ) D \tau$ would be better. 
In the chain of computations in your question, the chain rule is applied to the composition of $u$ with $\tau_\gamma^{-1}$, which is why $\tau_\gamma^{-1}$ appears instead of $\tau$. 
