Prime factorization rules When you have a number like 81.
Is it safe to assume that if the number can't be divided by 2 or 3 that it's prime if it ends with a 1?

Definitely not, the smallest two examples are  $91=7\times 13$ and $121=11\times 11$. (I am not counting $1$ as an example, though technically it qualifies!)
In fact, there are infinitely many non-primes of the required type, that is, ending in a $1$ and not divisible by $3$.  There are also infinitely many primes of that type.
With smallish numbers, a substantial proportion  of the numbers of the required type are prime.  But after a while, the primes of that type "thin out".  If you pick a huge number of that type "at random", then with high probability it will be a non-prime.  
Comment: You can "roll your own" counterexamples.  Suppose that one of the following holds:
(i) $a$ and $b$ end in $1$ and are not divisible by $3$, or
(ii) $a$ and $b$ end in $9$ and are not divisible by $3$, or
(iii) one of $a$ and $b$ ends in a $3$, the other in a $7$, and neither is divisible by $3$.
Then $a \times b$ ends in a $1$, is not divisible by $3$, and is not prime.
So for example pick $a=11$, $b=31$.  Their product $341$ ends in a $1$, is not divisible by $3$, and is not prime. 
