Q:

The integral expression for stretch/similarity theorem of Fourier transform

The integral expression for stretch/similarity theorem of Fourier transform, which is
$\displaystyle \mathcal{F}_x[f(ax)](y)=\int^\infty_{-\infty}f(ax)\exp\left(-2\pi iyx\right) \,dx$
But the definition of Fourier transform is 
$\displaystyle \mathcal{F}_x[f(x)](y)=\int^\infty_{-\infty}f(x)\exp\left(-2\pi iyx\right) \,dx$
My question is why the expression is not in the form
$\displaystyle \mathcal{F}_x[f(ax)](y)=\int^\infty_{-\infty}f(ax)\exp\left(-2\pi iy(ax)\right) \,d(ax)$

A:

It's somewhat ambiguous notation. Think of it like this: define the stretch operator $S_a$ by $$(S_af)(x) = f(ax).$$
The stretch similarity theorem is a statement about the Fourier transform of $S_af$:
$$
\mathcal{F}_x[(S_af)(x)](y) = \int_{-\infty}^\infty (S_af)(x)e^{-2\pi iyx}~dx,
$$ 
which your author has chosen to express in the equivalent form
$$
\mathcal{F}_x[f(ax)](y) = \int_{-\infty}^\infty f(ax)e^{-2\pi iyx}~dx.
$$

