I am using a basic singular value decomposition (via LAPACK) routine in FORTRAN to solve an overdetermined system in the form of $A\cdot X = B$ where $\mathrm{size}(A) = [m,n]$ with $m > n$.My sample data points come from a noisy sine function and I am trying to use linear regression with $x^i$ as my basis functions. I find that (with the noisy sine function) I get very good approximations when I keep my polynomial low. That is, when I fit a function of the form $$ a_0 + a_1 x + a_2 x^2 + \ldots + a_Nx^N $$with $N \lesssim 10$, I get great results. When I allow $N$ to get higher (I have 1000 data points), say, up to 250, my fit goes to hell and my sum of squares is $\approx 5.6\times 10^{41}$ (as opposed to about 200 with $N \lesssim 10$).Why does this happen? If a degree 6 polynomial, for example, provides the best fit, then shouldn't a degree $N$ polynomial produce that same fit with $0$ for $a_7 \ldots a_N$?
Linear regression via SVD not producing best fit with escalating polynomial degree
linear algebra;regression