Is the number $-1$ prime? From my understanding it's not prime because it's not greater than $0$. So my followup question is why did mathematicians exclude $-1$?
The definition of prime is having only two factors. 
$-1 \cdot 1 = -1$

The simplest answer to the question is that "numbers divisible only by 1 and themselves" isn't the strict mathematical definition of prime numbers, but rather the simple description that is used to teach the concept of prime numbers.
A more exact definition (in the context of integers) is "numbers, $n$, for which any factorisation must contain exactly one value divisible by $n$".
So no matter how you factorise $3$, such as $1\times3$, $(-1)\times(-1)\times3$, or $(-1)\times(-3)$, there will be exactly one factor that is divisible by $3$.
In the case of $-1$, we have $(-1)\times(-1)\times(-1)$, all of which are divisible by $-1$, and thus it does not satisfy the condition (indeed, $(-1)\times1$ demonstrates the same issue, as $-1$ divides $1$). That factorisations can contain multiple values divisible by $-1$ makes $-1$ a "unit".
And of course, in the case of $4$, we can write $2\times2$, with neither of the $2$s being divisible by $4$. That there are factorisations of $4$ for which no factors are divisible by $4$ makes $4$ a "composite".
I do not know for certain if this definition works in general for the more general concepts of prime elements and units, but my instinct says that it should work generally for any domain in which the concept of divisibility is well-defined (note that this isn't strictly consistent with the definition of "unit" in all domains, as the normal definition of "unit" is "number with a multiplicative inverse", which means that $2$ (and all other reals except zero) is a unit in $(\mathbb{R},+,\times)$, whereas my definition makes $1$ (and all other reals except zero) "composite" in that domain).
