Why can't this be a coset? Let $H$ be a subgroup of $G$ and H is not normal, there are left  cosets $aH$ and $bH$ whose product isn't a coset.
My attempt:
$ab H\subset aHbH$ and if H is not normal, if $ah_1bh_2=abh_3$ for all $h_1$, $h_2$, and $b$ then $h_1b=bh'$ for all $b$ and $H$ is normal a contradiction. So,$ab H\neq aHbH$ for some  $a,b$ and as cosets don't overlap $aHbH$ is not a coset of $H$. However, I don't know how to show that this isn't a coset of some other subgroup.

As noted by Jyrki Lahtonen, it might well be that the product $a H b H$, for a particular choice of $a, b$, is a coset of another subgroup $K$. Here is a general argument, and another particular example, also based on the dihedral group of order $8$.
Suppose $a H b H = c K$. Multiplying by $a^{-1}$ on the left, we have $H b H = a^{-1} c K$. Now $b \in H b H = a^{-1} c K$, so we have $a^{-1} c K = b K$ and multiplying $H b H = b K$ on the left by $b^{-1}$ we have
$$
(b^{-1} H b) H = K.\tag{prod}
$$
So the point is to construct a group $G$, subgroups $H \le K \le G$, and an element $b \in G$ such that (prod) holds.
Consider the dihedral group $K$ of order $8$, whose elements are
$$
1, r, r^2, r^3, s, s r, s r^2, s r^3.
$$
(Here $r$ is a rotation of $\pi/ 2$, and $s$ is a reflection. A presentation is given by $\langle r, s : r^4 = 1, s^2 = 1, s^{-1} r s = r^{-1} \rangle$.)
Consider the subgroup $H = \{ 1, r^2, s, s r^2 \}$ of $K$.
Now $K$ has an automorphism $b$ of order $2$ that maps $r \mapsto r$ and $s \mapsto s r$. Consider the semidirect product of $G$ by $\langle b \rangle$.
In $G$ we have $b^{-1} H b = \{ 1, r^2, s r , s r^3 \}$, and $b^{-1} H b H = K$.
Jyrki Lahtonen's example is simpler than mine, as it relies on $H = \{ 1, s \}$, $b = r$, so that $b^{-1} H b = \{ 1, s r^{2} \}$, and $K = \{ 1, s, r^{2}, s r^{2} \}$. 
