Q:

Matrix exponential is differentiable at $0 \in \mathbb{R}^{n,n}$

Given$$\exp : \mathbb{R}^{n,n} \mapsto \mathbb{R}^{n,n} \qquad A \mapsto \sum_{k=0}^{\infty} \frac{A^k}{k!}$$ where $ \mathbb{R}^{n,n}$ is equipped with Operator Norm.
I am trying to show that $\exp$ is differentiable at $0 \in \mathbb{R}^{n,n}$ with differential $L = D_{0}(\exp) = \text{Id} \in \mathbb{R}^{n,n}$
First we let $R : \mathbb{R}^{n,n} \mapsto \mathbb{R}^{n,n}$ be the error function such that $R(A) = \exp(A) - \exp(0) - L(A)$
We can show that $$\frac{R(A)}{\ \|A\|_{op}} =  \frac{\exp(A) -\text{Id}- A}{\|A\|_{op}} = \frac{\sum_{k=2}^{\infty} \frac{A^k}{k!}}{\ \|A\|_{op}} \longrightarrow 0 \quad \text{as } \ A \rightarrow 0$$
or equivalently that $R(A) \in o(\|A - 0\|_{op})$ which means for $\epsilon > 0$ there exists $\delta := \ ?$ $$\|A - 0\|_{op} = \|A\|_{op} < \delta \implies \frac{\ \|R(A)\|_{op}}{\ \|A\|_{op}} < \epsilon$$
I am having trouble working out this estimation. Any help would be great!

A:

As has been noted,
$\exp(A) = \displaystyle \sum_0^\infty \dfrac{A^n}{n!}; \tag 1$
to validate $D_A\exp$ at any point $A$, where $D_A\exp$ is an $\Bbb R$-linear mapping, we need to show that for any $H \in \Bbb R^{n \times n}$, 
$\dfrac{\Vert \exp(A + H) - \exp(A) - D_A\exp(H) \Vert}{\Vert H \Vert} = o(\Vert H \Vert), \tag 2$
where
$o(H) \to 0 \; \text{as} \; H \to 0; \tag 3$
for general $A \in \Bbb R^{n \times n}$, $\exp(A + H)$ can be difficult to evaluate when $AH \ne HA$; but when $A = 0$, (2) takes on the much simpler form
$\dfrac{\Vert \exp(H) - \exp(0) - D_0\exp(H) \Vert}{\Vert H \Vert} = o(\Vert H \Vert), \tag 4$
that is,
$\dfrac{\Vert \exp(H) - I - D_0\exp(H) \Vert}{\Vert H \Vert} = o(\Vert H \Vert); \tag 5$
now by virtue of (1) we have
$\exp(H) - I = \displaystyle \sum_1^\infty \dfrac{H^n}{n!} = H + \sum_2^\infty \dfrac{H^n}{n!}, \tag 6$
whence
$\exp(H) - I - H = \displaystyle \sum_2^\infty \dfrac{H^n}{n!}; \tag 6$
thus,
$\Vert \exp(H) - I - H \Vert = \left \Vert \displaystyle \sum_2^\infty \dfrac{H^n}{n!} \right \Vert = \left \Vert H^2 \displaystyle \sum_2^\infty \dfrac{H^{n - 2}}{n!} \right \Vert \le \Vert H \Vert^2  \left \Vert \displaystyle \sum_2^\infty \dfrac{H^{n - 2}}{n!} \right \Vert, \tag 7$
and so
$\dfrac{\Vert \exp(H) - I - H \Vert}{\Vert H \Vert} \le \Vert H \Vert  \left \Vert \displaystyle \sum_2^\infty \dfrac{H^{n - 2}}{n!} \right \Vert = o(\Vert H \Vert), \tag 8$
which shows that in fact
$D_0 \exp(H) = H, \tag 9$
i.e., 
$D_0 \exp = I \in \Bbb R^{n \times n}, \tag{10}$
as desired. $OE\Delta$.
Note Added in Edit, Tuesday 22 January 2019 8:47 PM PST:  our OP mike expressed some interest in the reasons why (in his terms)
$\dfrac{R(A)}{\ \|A\|_{op}} =  \dfrac{\exp(A) -\text{Id}- A}{\|A\|_{op}} \longrightarrow 0 \quad \text{as } \ A \rightarrow 0; \tag{11}$
but this is easily seen to be addressed in equations (6)-(8) of this answer; the lynchpin here is the factoring out of $\Vert H \Vert^2$ from 
$\Vert \sum_2^\infty \frac{H^n}{n!} \Vert$ in (7); once this is done, the consequences of dividing by $\Vert H \Vert$ become clear, and since only one factor of $\Vert H \Vert$ cancels out, we are left with an expression of order $o(\Vert H \Vert)$ as indicated; the same argument applies to (11). Of course, to complete the demonstration we need to insure that the power series $\sum_2^\infty \frac{\ H^{n - 2} }{n!}$ is convergent; but an easy application of the ratio test shows that it in fact $\sum_2^\infty \frac{\Vert H \Vert^{n - 2} }{n!}$converges for all $\Vert H \Vert$, so  $\sum_2^\infty \frac{\ H^{n - 2} }{n!}$ converges absolutely and uniformly as long as $\Vert H \Vert$ is bounded.  End of Note.

