Find order of group given by generators and relations Let $G$ be the group defined by these relations on the generators $a$ and $b$: $\langle a, b; a^5, b^4, ab=ba^{-1}\rangle$. I need hints how to find order of $G$.

The only good way to do problems like this is to play around with words in the letters $a,b,a^{-1},b^{-1}$, seeing if you can use those given relations to get any word in some standard form. Then, you can say every element in $G$ is equivalent to one in this form, and there are so many words in that form, so $G$ has this order.
In your case, the $ab=ba^{-1}$ relation is powerful: it allows you to essentially move $a$'s though $b$'s (though it inverts the $a$'s in the process), so you can group all the $a$'s and $b$'s in any word together. Then you can use the $a^5$ and $b^4$ relation to reduce the size of the $a$ and $b$ clumps.
