Am I allowed to define $e$ through algebraic means using this limit? By using the formal definition of a derivative
$$f'(x) = \lim \limits_{h \to 0} \frac{f(x+h)-f(x)}{h}$$
one can get
$$\frac{d}{dx}[e^x] = e^x \times \lim \limits_{h \to 0} \frac{e^h-1}{h}$$
which implies
$$ \lim \limits_{h \to 0} \frac{e^h-1}{h} = 1$$
If we remove the limit but remember that $h$ is a really small number one can then do the following
$$ 
\begin{align*}
\frac{e^h-1}{h} = 1 \\ \\
e^h-1=h\\ \\
e^h=h+1\\ \\
e=(h+1&)^\frac{1}{h}
\end{align*}$$
Apply the limit to get
$$e=\lim \limits_{h \to 0} (h+1)^\frac{1}{h}$$
Which is correctly one of the representations/definitions of $e$. Then one can set $n = \frac{1}{h}$ and get the common
$$e=\lim \limits_{n \to \infty} (\frac{1}{n}+1)^n$$
But it doesn't feel right however. It feels like I cheated. So am I allowed to actually derive $e$ like this? Is it "formal" in other words.

No.
In one of your first steps, you at best have
$$ \frac{e^h-1}h\approx 1.$$
And applying $(\cdot)^{\frac1h}$ might principally make the small-ish error much larger.
E.g., we know that $\lim_{n\to\infty}\sqrt[n]n= 1$.  Arguing like you tried, ...

If we remove the limit but remember that $n$ is a really big integer, one can then do the following
$$\sqrt[n]n=1,$$
$$n=1^n,$$
$$n=1.$$

This is obviously wrong.
