<jupyter_start><jupyter_text>Regularization for Sparsity:  L1 Regularization (lasso)This exercise is based on the tensorflow [playground](https://playground.tensorflow.org) program (developed by google to teach machine learning principles).You'll experiment L1 regularization for a small, noisy training data set to perform 'supervised) binary classification. In this kind of setting, overfitting is a real concern. Fortunately, regularization might help.The input data are bivariate, this yields the two features $X_1$ and $X_2$. Feature crosses such that their product $X_1X_2$, their squared values $X_1^2$ and $X_2^2$, or their sinus are also included as input for the linear model.  Task IConsider now this [tensorflow playground model](https://playground.tensorflow.org/activation=linear&regularization=L2&batchSize=7&dataset=xor&regDataset=reg-plane&learningRate=0.01&regularizationRate=0.1&noise=30&networkShape=&seed=0.10080&showTestData=false&discretize=false&percTrainData=20&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false&hideText=false&playButton_hide=false&regularizationRate_hide=false&percTrainData_hide=false&numHiddenLayers_hide=true&noise_hide=false&problem_hide=true&regularization_hide=false&dataset_hide=false&activation_hide=true), the exercice  consists of five related demos. To simplify comparisons, run each demo in a separate tab.  Notice that the thicknesses of the lines connecting FEATURES and OUTPUT represent the relative weights (coefficients) of each feature.| Demo | Regularization Type | Regularization Rate (lambda) ||------|---------------------|------------------------------|| 1    |   $L_2$             |       0.1                    || 2    |   $L_2$             |       0.3                    || 3    |   $L_1$             |       0.1                    || 4    |   $L_1$             |       0.3                    || 5    |   $L_1$             |      experiment              | **Questions:** - How does switching from $L_2$ to $L_1$ regularization influence the delta between test loss and training loss? - How does switching from $L_2$ to $L_1$ regularization influence the learned weights? - How does increasing the $L_1$ regularization rate (lambda) influence the learned weights? - Why the $L_1$ penalty seems most appropriate than $L_2$ one for this problem?   Task IIWe consider now this [neural net model](https://playground.tensorflow.org/activation=relu&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.01&regularizationRate=0.01&noise=50&networkShape=8,8,8,8,8,8&seed=0.91875&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false&hideText=false&playButton_hide=false&regularizationRate_hide=false&percTrainData_hide=false&numHiddenLayers_hide=false&noise_hide=false&problem_hide=true&regularization_hide=false&dataset_hide=false&activation_hide=false) with six hidden layers with eight units (neurons) each. We choose a non-linear activation function, namely the `ReLU` $g(x) = \max (0,x)$ which is a standard choice in deep learning ([why?](Some-words-on-activation-functions-for-deep-neural-nets)).Run the model as given, without regularization, for at least 1000 epochs. We can adjust if necessary the learning rate (increase it to speed up the convergence, or decrease it to gain in stability).Note the following :- delta between Test and Training loss.- The learned weights for features and hidden units (neurons)Redo the same operation using now a L1 regularization, with the regularization rate as given, in a separate tab**Questions:**- How does introducing $L_1$ regularization  - influence the test loss and also the  delta between test loss and training loss?  - influence the learned weights?- What are the only features that are selected with $L_1$ regularization? Is it in agreement with the 'optimal' decision boundary for this data set? Are the hidden layers useful here?<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1018985820716174, 'fraction_numerical': 0.020187454938716654, 'mean_word_length': 4.77253814147018, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '142529', 'license': 'Apache-2.0', 'n_tokens_mistral': 1228, 'n_tokens_neox': 1141, 'n_words': 366, 'path': 'Notebooks/6_Lasso_regularization/N1_L1_regularization.ipynb', 'repo_name': 'shouhaiel1/Machine_Learning_Labworks', 'size': '4161'}