Further examination of the gradient data is undertaken by training a 0.2B model, meticulously recording every step gradient information, and evaluating the differences between consecutive steps, thereby providing an approximation of second-order gradient information. We treat the gradient at step \(t\) as a flattened vector \(\mathbf{g}^{(t)}\), and the parameter (also flattened as a vector \(\mathbf{x}^{(t)}\) ) update between step \(t\) and \(t+1\) is \(\mathbf{v}^{(t)}=\mathbf{x}^{(t+1)}-\mathbf{x}^{(t)}\). The gradient norm take the \(L2\) norm of the gradient \(\|\mathbf{g}^{(t)}\|\), gradient inner product is \(\mathbf{g}^{(t+1)}\cdot\mathbf{g}^{(t)}\)the cosine of the gradient's angle is given by \(\frac{\mathbf{g}^{(t+1)},\mathbf{g}^{(t)}}{\|\mathbf{g}^{(t+1)}\|\|\mathbf{g}^{(t )}\|}\). Imaging the optimization process as a trajectory over a high-dimension manifold, first order directional derivative along the trajectory is computed as \(D_{1}=\frac{\mathbf{g}^{(t+1)},\mathbf{v}^{(t)}}{\|\mathbf{v}^{(t)}\|}\), and the second order directional derivative is \(D_{2}=\frac{(\mathbf{g}^{(t+1)}-\mathbf{g}^{(t)}),\mathbf{v}^{(t)}}{\|\mathbf{ v}^{(t)}\|^{2}}\). \(D_{1},D_{2}\) enables an approximate estimation of the loss curvature on the trajectory, \(K=\frac{|D_{2}|}{(1+D_{1}^{2})^{\frac{1}{2}}}\). The results of these statistics over time are shown in Figure 8. We can see that the gradient norm diminishes during the decay phase, and upon commencement of this stage, the cosine between gradients predominantly assumes positive values, suggesting that in the decay phase, model parameters undergo consistent changes across steps. Concerning directional derivatives, it is remarkable that the first-order directional derivative diminishes exponentially with each step, aligning closely with the learning rate, while the second-order directional derivative exhibits a slight increase in magnitude. The curvature of the loss function also increases by a magnitude, indicating the proximity to a local optimum. These findings potentially offer a deeper insight into the shape optimization space, a subject reserved for future exploration.