Need help with integration by substitution for $\int_x^{x+1}\left(\sin\ t^2\right)dt$ I am doing exer 13 of baby Rudin Ch 6. I learned integration by substitution and by parts only recently, hence having trouble doing it.

Exercise: Define $$f(x)=\int_x^{x+1}\left(\sin\ t^2\right)dt\ .$$ Prove that  $|f(x)|<\frac{1}{x}$ if $x>0.$

Hint says to substitute $t^2=u$, which, I think means that substitute $t=\sqrt u$ or $t=-\sqrt u$. Which one should I substitute, and why?
Also, the solution I have says that $|f(x)|<\frac1x$ is obvious for $0<x\le 1$. I do not see how it is obvious, or why we have to consider this as separate case. 


Which one should I substitute, and why?

Whichever is convenient. It'll work out either way; it just changes a few signs around in a way that makes no net difference.
I wouldn't follow the hint myself; I'd go with an integration by parts instead - or, if I followed the hint, I'd probably follow up with an integration by parts. The key to this integral is that the rapid oscillation will lead to things mostly canceling out. We need to get a handle on that, either with something like an alternating series estimate or by converting it into an absolutely convergent form.

... that $|f(x)|<\frac1x$ is obvious for $0<x\le 1$. I do not see how it is obvious, or why we have to consider this as separate case.

Just estimate that $\sin$ by $1$:
$$|f(x)|\le \int_x^{x+1} 1\,dt = 1 \le \frac1x$$
As for why we'd have to treat that case separately? The suggested substitution introduces a factor of $\frac1{\sqrt{u}}$ which blows up if $x$ is too small. Similarly, my integration by parts introduces a factor of $\frac1t$. The inequalities we want to use to estimate $f$ elsewhere fail for small $x$, so we use a different method to estimate it there.
