Is Set "prime" with respect to the cartesian product? (Motivated by Stefan Perko's question here)

Suppose $C, D$ are two categories such that $\text{Set} \cong C \times D$. Is either $C$ or $D$ necessarily equivalent to the terminal category $1$?

I haven't thought about it much, but my guess is that the answer is yes. Consider, for example, the special case that $C$ and $D$ are categories of sheaves on spaces. We have $\text{Sh}(X) \times \text{Sh}(Y) \cong \text{Sh}(X + Y)$ (where $+$ denotes coproduct), and $\text{Sh}(1) \cong \text{Set}$, so (after restricting to sober spaces) being able to write $\text{Set}$ as a product of sheaf topoi corresponds to decomposing the one-point space $1$ as a disjoint union, but of course no nontrivial such decomposition is possible. 
I can show that either $C$ or $D$ must have a zero object, which settles the question if $C$ and $D$ are required to be topoi. 

Suppose you have an equivalence $Set\cong C\times D$ where $C$ and $D$ are both nontrivial.  Let $(0_C,0_D)$ be a pair of objects corresponding to the empty set.  Then $0_C$ and $0_D$ must be initial and we must have that if there exist maps $c\to 0_C$ and $d\to 0_D$, then $c\cong 0_C$ and $d\cong 0_D$ (since no nonempty set can map to the empty set).  In particular, taking $d=0_D$, we find that every object of $C$ that maps to $0_C$ is initial, and similarly for $0_D$.  Since $C$ and $D$ are both nontrivial, we can choose objects $c\in C$ and $d\in D$ which are not initial.  Note that there are then no maps $(c,0_D)\to(0_C,d)$, since there are no maps $c\to 0_C$.  But this is impossible, because $(0_C,d)\not\cong (0_C,0_D)$ so $(0_C,d)$ must correspond to a nonempty set, and every set can map to every nonempty set.

Here's another argument, which is more similar in spirit to the arguments for other categories I sketched in comments below.  Let $(0_C,0_D)$ be as above and let $(1_C,1_D)$ be the terminal object.  Note that $(1_C,1_D)$ is the coproduct of $(1_C,0_D)$ and $(0_C,1_D)$.  Since the terminal object of $Set$ cannot be written as a coproduct unless one of the summands is initial, this implies either $1_C\cong 0_C$ or $1_D\cong 0_D$; suppose $1_C\cong 0_C$.  Then for any $c\in C$, note that there is exactly one map from $(0_C,1_D)\cong (1_C,1_D)$ to $(c,1_D)$.  But this means that as a set, $(c,1_D)$ has one point, so it is terminal, so $c\cong 1_C$.  Thus every object in $C$ is terminal and $C$ is trivial.
