<jupyter_start><jupyter_text>solar demostration  this ".ipynb" file produces the graphical and numerical results for solar demonstration  this demostration is not included in the paper  we set it up for the step-by-step illustration of solar computation  ---  Check the following before running the code (a) To avoid confusion, reset your kernel before you running the notebook (to clear memory): Menu "Kernel" $\rightarrow$ "Restart Kernel and clear all outputs". (b) To evaluate the code for simulation replication,*  click : Menu "Kernel" $\rightarrow$ "Restart Kernel and Run All Cells" *  or, select a cell of code, press "shift" and "enter". Run all cells to avoid errors  (c) Check "joblib", "scikit-learn", "numpy", "matplotlib" and "tqdm" are installed. If not,*  run "pip install joblib scikit-learn numpy matplotlib tqdm" in terminal (Mac OS or Linux) or command prompt as administrator (Windows) if you use Python3 without any IDE. *  we highly recommend installing Anaconda3 directly to avoid unecessary package management (all packages mentioned above are installed by default). ---  1: import all modules*  For simplicity and elegancy, all relevant functions and classes are coded in "solar_simul_demo.py".<jupyter_code>import numpy             as np
import matplotlib.pyplot as plt

from solar_simul_demo import one_shot_simul<jupyter_output><empty_output><jupyter_text>--- 2(a): define inputs values|  variable name  |   meaning  ||-|-||   sample_size   |   the sample size $n$ in the paper; | |   n_dim         |   the number of variables(informative + redundant) in $X$, $p$ in the paper; | |   n_info        |   the number of informative variables in $X$; | |   n_repeat      |   the number of subsamples generated by solar; | |   step_size     |   the step size for tuning $c$; | |   rnd_seed      |   the random seed value; | |   plot_on       |   the binary value that controls whether the simulation plots will be saved as PDF; |  2(b): define DGP*  the population regression equation is $$Y = 2\cdot \mathbf{x}_0 + 3\cdot \mathbf{x}_1 + 4\cdot \mathbf{x}_2 + 5\cdot \mathbf{x}_3 + 6\cdot \mathbf{x}_4  + y,$$ *  To change the simulation settings, simply change the input values. If you change *n_info* you will adjust the DGP as follows:     *  If $i > \mbox{n_info} - 1$ and $i \in \left[ 0, 1, 2, \ldots, p-1 \right]$, $\beta_i = 0$ in population;    *  If $i \leqslant \mbox{n_info} - 1$ and $i \in \left[ 0, 1, 2, \ldots, p-1 \right]$, $\beta_i = i + 2$ in population<jupyter_code>sample_size = 200
n_dim       = 100
n_info      = 5
n_repeat    = 10
step_size   = -0.02
rnd_seed    = 0
plot_on     = False<jupyter_output><empty_output><jupyter_text>--- 4: compute solar, CV-lars-lasso (lasso solved by CV-lars) and CV-cd (lasso solved by warm-start cylindrical coordinate descent) Numpy, sklearn and python are actively updated. If you use different version, replication results may be slightly different from the paper (see Read_me_first.docx for detail).<jupyter_code>#control the random seed for reproduction
np.random.seed(rnd_seed)<jupyter_output><empty_output><jupyter_text>first, we call the class for simulation from "solar_simul_demo.py"<jupyter_code>trial = one_shot_simul(sample_size, n_dim, n_info, n_repeat,  step_size, rnd_seed, plot_on)<jupyter_output><empty_output><jupyter_text>then we compute cv-lars-lasso, cv-cd and solar on the simulated data and return the result<jupyter_code>solar_coef, opt_c, test_error, Qc_list, la_list, la_vari_list, cd_list, cd_vari_list = trial.simul_func()<jupyter_output><empty_output><jupyter_text>--- 4: plots and results of the simulation  first, we plot the result of $c$ tuning (via validation error minimization) in algorithm 2<jupyter_code>trial.q_plot(test_error, opt_c)<jupyter_output><empty_output><jupyter_text>then we report the $Q(c) = \left\{ \mathbf{x}_j \; \vert \; \widehat{q}_j \geqslant c \right\}$ under different value of $c$.<jupyter_code>trial.q_list(Qc_list)<jupyter_output>q_hat value >=  1.0
[['X4']]
q_hat value >=  0.9777777777777777
[['X4'], ['X3', 'X2']]
q_hat value >=  0.9555555555555556
[['X4'], ['X3', 'X2'], ['X1', 'X0']]
q_hat value >=  0.9111111111111111
[['X4'], ['X3', 'X2'], ['X1', 'X0'], ['X19', 'X18']]
q_hat value >=  0.8888888888888888
[['X4'], ['X3', 'X2'], ['X1', 'X0'], ['X19', 'X18'], ['X68']]
q_hat value >=  0.8666666666666667
[['X4'], ['X3', 'X2'], ['X1', 'X0'], ['X19', 'X18'], ['X68'], ['X71', 'X40']]
q_hat value >=  0.8222222222222222
[['X4'], ['X3', 'X2'], ['X1', 'X0'], ['X19', 'X18'], ['X68'], ['X71', 'X40'], ['X37']]
q_hat value >=  0.7777777777777778
[['X4'], ['X3', 'X2'], ['X1', 'X0'], ['X19', 'X18'], ['X68'], ['X71', 'X40'], ['X37'], ['X91']]
q_hat value >=  0.7111111111111111
[['X4'], ['X3', 'X2'], ['X1', 'X0'], ['X19', 'X18'], ['X68'], ['X71', 'X40'], ['X37'], ['X91'], ['X72', 'X35']]
q_hat value >=  0.6888888888888889
[['X4'], ['X3', 'X2'], ['X1', 'X0'], ['X19', 'X18'], ['X68'], ['X71', 'X40'], ['X37'], ['X91'], ['X72', 'X35[...]<jupyter_text>then we report solar regression coefficents<jupyter_code>print(solar_coef)<jupyter_output>[[1.90590232]
 [2.92660102]
 [3.94706396]
 [5.06522281]
 [6.12764936]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.        ]
 [0.  [...]<jupyter_text>report $c^*$ of solar the reason we print max(opt_c) is because sometimes multiple values of $c$ is assigned with the same value of validation error<jupyter_code>print(max(opt_c))<jupyter_output>0.96<jupyter_text>and the number of variables selected by cv-lars-lasso<jupyter_code>print(la_list)<jupyter_output>14<jupyter_text>and the number of variables selected by cv-cd<jupyter_code>print(cd_list)<jupyter_output>17<jupyter_text>and variables selected by cv-lars-lasso<jupyter_code>print('variables selected by cv-lars-lasso: ', [ 'X' + str(i) for i in la_vari_list])<jupyter_output>variables selected by cv-lars-lasso:  ['X4', 'X3', 'X2', 'X1', 'X0', 'X71', 'X91', 'X90', 'X17', 'X40', 'X70', 'X65', 'X94', 'X28']<jupyter_text>and variables selected by cv-cd<jupyter_code>print('variables selected by cv-cd: ', [ 'X' + str(i) for i in cd_vari_list])<jupyter_output>variables selected by cv-cd:  ['X0', 'X1', 'X2', 'X3', 'X4', 'X17', 'X19', 'X28', 'X31', 'X40', 'X41', 'X65', 'X70', 'X71', 'X90', 'X91', 'X94']<jupyter_text>Finally we produce the HTML file<jupyter_code>!rm -rf solar_demo.html
!jupyter nbconvert --to html solar_demo.ipynb<jupyter_output>[NbConvertApp] Converting notebook solar_demo.ipynb to html
[NbConvertApp] Writing 638650 bytes to solar_demo.html
{'fraction_non_alphanumeric': 0.16691525423728815, 'fraction_numerical': 0.06793220338983051, 'mean_word_length': 3.1000555864369095, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 60, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 9, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '324479', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3029, 'n_tokens_neox': 2770, 'n_words': 785, 'path': 'demo/solar_demo.ipynb', 'repo_name': 'isaac2math/solarpy', 'size': '7375'}