<jupyter_start><jupyter_text>Lets assume that $\lambda_k$ monotonically increase and obey the following law:$$\lambda_{max} - \lambda_k = bc^k$$or$$ \lambda_k = \lambda_{max} - bc^k.$$This could be seen as a function (for discrete values)$$ y = a - bc^x. $$ We would like to run a nonlinear regression on it, but it's hard in this form. Lets do some transformations.First, lets get rid of $a$ by doing differences:$$ y(x^{k+1}) - y(x^k) = bc^{x^k}(1-c) $$and if we define $Y(x) = y(x+1) - y(x)$, we could write$$ Y(x) = b(1-c)c^x. $$Lets now say $B = b(1-c)$, then$$ Y = Bc^x $$which is amenable to logarithmic transformation$$ ln(Y) = ln(B) + xln(c), $$which we can run a linear regression on.$ \lambda_{max} \approx \lambda_k + bc^k $, right? :)<jupyter_code>first = 2

from scipy import polyfit
import math
import math

print('lambda_max estimates')
lambda_max_full = 1.0
lambda_max_last = 1.0
for n in range(first,N):
    # Do LS using *all* values
    Y = [lambdas[k+1] - lambdas[k] for k in range(0, n)]
    X = [k for k in range(0, n)]
    lnY = [math.log(y) for y in Y]
    (ar, br) = polyfit(X, lnY, 1)
    # Validation check
    # for i in range(len(X)):
    #     print(lnY[i] - (ar*X[i] + br))
    B = math.exp(br)
    c_all = math.exp(ar)
    b_all = B/(1-c_all)
    new_lambda_max_full = lambdas[n] + b_all*math.pow(c_all,n)


    # Do LS using last few values
    last = 2
    Y = [lambdas[k+1] - lambdas[k] for k in range(n-last, n)]
    X = [k for k in range(n-last, n)]
    lnY = [math.log(y) for y in Y]
    (ar, br) = polyfit(X, lnY, 1)
    B = math.exp(br)
    c_last = math.exp(ar)
    b_last = B/(1-c_last)
    new_lambda_max_last = lambdas[n] + b_last*math.pow(c_last,n)

    if n == first:
        print('%2d its: %.3f         | %.3f' % 
          (n+1, new_lambda_max_full, new_lambda_max_last))
    else:
        print('%2d its: %.3f [%4.2f%%] | %.3f [%4.2f%%]' % 
              (n+1, new_lambda_max_full, 100*(new_lambda_max_full/lambda_max_full - 1.0),
                    new_lambda_max_last, 100*(new_lambda_max_last/lambda_max_last - 1.0)))
    lambda_max_full = new_lambda_max_full
    lambda_max_last = new_lambda_max_last<jupyter_output>lambda_max estimates
 3 its: 1.708         | 1.708
 4 its: 1.776 [3.99%] | 1.829 [7.11%]
 5 its: 1.815 [2.18%] | 1.860 [1.67%]
 6 its: 1.838 [1.29%] | 1.872 [0.64%]
 7 its: 1.854 [0.85%] | 1.881 [0.49%]
 8 its: 1.865 [0.63%] | 1.890 [0.50%]
 9 its: 1.875 [0.51%] | 1.900 [0.51%]
10 its: 1.883 [0.44%] | 1.910 [0.51%]
11 its: 1.891 [0.40%] | 1.918 [0.45%]
12 its: 1.897 [0.36%] | 1.926 [0.40%]
13 its: 1.904 [0.32%] | 1.933 [0.34%]
14 its: 1.909 [0.29%] | 1.939 [0.32%]
15 its: 1.914 [0.26%] | 1.943 [0.22%]
16 its: 1.919 [0.24%] | 1.947 [0.21%]
17 its: 1.923 [0.22%] | 1.952 [0.23%]
18 its: 1.927 [0.20%] | 1.954 [0.14%]
19 its: 1.930 [0.18%] | 1.957 [0.12%]
20 its: 1.933 [0.17%] | 1.959 [0.11%]
{'fraction_non_alphanumeric': 0.19525801952580196, 'fraction_numerical': 0.1101813110181311, 'mean_word_length': 2.8510067114093958, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '234508', 'license': 'MIT', 'n_tokens_mistral': 1489, 'n_tokens_neox': 1297, 'n_words': 348, 'path': 'chebyshev_eig_estimate.ipynb', 'repo_name': 'aprokop/jupyter', 'size': '2868'}