Why this happens
The documentation states
Integers are assumed to be represented in two's complement form, with an unlimited number of digits, so that
BitNot[n] is simply equivalent to $-1-n$.
So Mathematica does not assume a fixed number of binary digits (you can't assume your number to be 64 bit long or 32 bit long). It always takes the exact number of bits necessary to represent the number in two's complement form.
If you start with $n=5$, the binary representation is $0101_2$ because 4 bits are the minimal number needed to represent 5 in this form. The leading 0 is needed for the sign. The negation of this is $1010_2 = -2^4 + 10 = -6$.
How to achieve what you want
One way is to convert the number to a list of digits and work with that:
not[lst_] := 1 - lst
The third argument of
IntegerDigits controls the number of digits you need.
FromDigits converts back to a number.