A comparison between the performance of the three algorithms is shown in Figure REF . Symbolic regression and polynomial regression were the only algorithms that predict the peaks and valleys of the \(\sin (x)\)  function within the range of [-\(\pi \) ,\(\pi \) ]. This success can be attributed to the fact that both algorithms represent the \(\sin (x)\)  function as a polynomial. For the case of the symbolic regression, it discovered the polynomial representation by itself. On the contrary, the ANN failed to generalize beyond the training data. We hoped that by making the network deeper, we could help the network extract sufficient features from the training data, and generalize. However, what we observe is that the ANN memorizes the training data instead of generalizing it. For example for \(x\)  \(>\)  \(\dfrac{\pi }{2}\)  the output of the ANN is always a constant value of one, which is the value of \(\sin (\dfrac{\pi }{2})\) , and for
\(x<0\)  the output of the ANN is always a constant value of zero, which is the value of \(\sin (0)\) . This type of memorization by an ANN is also observed in several other studies such as [1]}. Also, the work of Kim et al. [2]} showed that if feed-forward ANN is integrated with symbolic regression, one obtains a better generalization behaviour compared to pure ANN. Another interesting observation is that despite the fact that both symbolic regression and ANN optimize the mean square difference, they come up with totally different generalization behaviour.
