Why does this series have a different sum when its terms are rearranged? The problem is:

Give an example of a convergent series such that, when the terms are rearranged, the series sums to a different value.

A solution is:



Although everything makes sense in this solution, I don't get how what it claims is possible.  How can a series have two different sums?  

Is this caused by the fact that subtraction is not commutative or associative?  Hence, in fact, we really have two different series here?
No, that's not the reason.

When we sum a series, the order by which we sum is essential. That is, we first take $a_1$, then $a_1+a_2$, then $a_1+a_2+a_3$ and so on. Then we see which limit we get.

If, instead, we sum it as$$a_1,a_1+a_3,a_1+a_3+a_2,a_1+a_3+a_2+a_5,a_1+a_3+a_2+a_5+a_7,a_1+a_3+a_2+a_5+a_7+a_4,\ldots$$then we may well get a different limit. However, if the series $|a_1|+|a_2|+|a_3|+\cdots$ converges, this cannot happen. That is, the limit will always be the same.

In order to know more, read about Riemann's rearrangement theorem.
I think that main (and only) reasons why sum of series that is not absolutely convergent is not invariant with respect to permutations are that the operation of subtraction is neither commutative nor associative in general.

Because, if the terms are all positive we obtain always the same sum, no matter how the series is rearranged, because of commutativity and associativity of addition over the positive numbers.

So, I tend to believe that you are on the right track when viewing different sums as a consequunces of non-commutativity (and non-associativity) of subtraction.

Because of non-commutativity and non-associativity of subraction it is natural to have different sums when we permute the series, but it can come as a surprise that we can obtain any value we want with different permutations.