Japanese Temple Geometry Problem: Radii of inner circles inside quarter arcs 
I was able to get the equation for the radius of larger circle but couldn't think for the smaller one.

Source: wu riddles

Since you've already shown how to determine the radius of the larger circle, here's a method for calculating the smaller circle's radius. First, below is your diagram with a few lines, points and line lengths added to it:

Note the tangent line at $G$ to the smaller circle, and the one for the circular arc with center at $A$, are the same. Thus, the perpendicular line to that tangent from $G$ to $C$ (the center of the smaller circle), and from $G$ to $A$, are colinear (i.e., the two circle's tangent point's normal line goes through their centers), so $\lvert AC\rvert = a + r$. Similarly, the line joining $B$ to $F$ goes through $C$, thus $\lvert CF\rvert = a - r$.
With $CE$ being perpendicular to $AF$, then $\triangle AEC$ and $\triangle FEC$ are both right-angled triangles. Thus, using the Pythagorean Theorem to relate the squares of the various triangle side lengths to the square of the length of their common side of $CE$ gives
$$\begin{equation}\begin{aligned}
\lvert AC\rvert^2 - \lvert AE\rvert^2 & = \lvert CF\rvert^2 - \lvert EF\rvert^2 \\
(a+r)^2 - (a-r)^2 & = (a-r)^2 - r^2 \\
(a^2 + 2ar + r^2) - (a^2 - 2ar + r^2) & = (a^2 - 2ar + r^2) - r^2 \\
4ar & = a^2 - 2ar \\
6r & = a \\
r & = \frac{a}{6}
\end{aligned}\end{equation}$$
