Smallest example of a group that is not isomorphic to a cyclic group, a direct product of cyclic groups or a semi direct product of cyclic groups. 
What is the smallest example of a group that is not isomorphic to a cyclic group, a direct product of cyclic groups or a semi-direct product of cyclic groups?

So finite abelian groups are ruled out, as are the groups up to order $7$ I believe, since $1,2,3$ are all isomorphic to cyclic groups, $4$ has two non-isomorphic groups one cyclic and one isomorphic to the direct product $\Bbb{Z/2Z\times Z/2Z}$. $5,7$ are prime and so must be cyclic, and there's only one nonabelian group of order $6$ which is isomorphic to $\Bbb{Z/3Z\rtimes Z/2Z}$. Is it the quaternion group?

Yes, quaternion group is non abelian and not isomorphic to the direct product of $\Bbb{Z/2Z\times Z/2Z\times Z/2Z}$ or $\Bbb{Z/2Z\times Z/4Z}$, and is the one you are looking for. 
