Why can't this be a coset? Let $H$ be a subgroup of $G$ and H is not normal, there are left  cosets $aH$ and $bH$ whose product isn't a coset.
My attempt:
$ab H\subset aHbH$ and if H is not normal, if $ah_1bh_2=abh_3$ for all $h_1$, $h_2$, and $b$ then $h_1b=bh'$ for all $b$ and $H$ is normal a contradiction. So,$ab H\neq aHbH$ for some  $a,b$ and as cosets don't overlap $aHbH$ is not a coset of $H$. However, I don't know how to show that this isn't a coset of some other subgroup.

Others have given you hints as to why normality of $H$ follows, if we want $(aH)(bH)$ to be a coset of $H$ for all $a,b\in G$. I am fairly sure that this is the intended meaning of whoever gave you this exercise. This is because otherwise the claim is false. I proffer the following counterexample.
Let $G$ be the dihedral group of 8 elements, so it is generated by an element $r$ (=rotation by 90 degrees) and an element $s$ (= a reflection). These are subject to the relations $r^4=s^2=1, sr^i=r^{-i}s$ (for all integers $i$). We observe that the element $r^2$ (=rotation by 180 degrees) is in the center. Let $H=\langle s\rangle=\{1,s\}$. This is clearly not normal, for $rH=\{r,rs\}\neq \{r,sr=r^3s\}=Hr$. Yet I claim that $(aH)(bH)$ is always a coset of some subgroup of $G$.
All the cosets of $H$ are of the form $r^iH$ for $i=0,1,2$ or $3$. We have
$$
(r^iH)(r^jH)=\{r^{i+j},r^{i-j}s,r^{i+j}s,r^{i-j}\}.
$$
If $j$ is even, then $r^{i+j}=r^{i-j}$, and this is the coset $r^{i+j}H$ of $H$. If $j$ is odd, then $r^{i+j}=r^{i-j}r^{2j}=r^{i-j}r^2$. As $r^2$ is central this means that
$$
(r^iH)(r^jH)=r^iK,
$$
where $K$ is the subgroup generated by $s$ and $r^2$.
So it stands to reason that the (surrounding) context implies that the question is only about cosets of the fixed subgroup $H$.
