Exploring fastai with Excel and Python - #8 by Daniel - Part 1 2022 - Deep Learning Course Forums Deep Learning Course Forums Exploring fastai with Excel and Python Part 1 2022 Daniel (深度碎片) July 14, 2022, 1:55pm #1 About the topic This topic is a test on how far I can explore theories and techniques taught in fastai using Excel with great help and guidance from @Moody and many others in fastai community. Why Doing it? Jeremy said it multiple times Excel is greatly underappreciated by many in deep learning On how to do fastai, Jeremy said numerous times: just finish one thing and if you can, finish it nicely! I am deeply inspired by @Moody’s story with excel and fastai I want exploring fastai with detailed note and excel experiments to be my thing. I want to truly test my tenacity this time. (I was ashamed when Jeremy and Radek said you can only fail fastai by giving up. Just imagine it, I joined the forum at the end of 2016. Pray for no accidents this time) What is the plan Experiments and contemplation: so far just the first step and possible future steps (e.g., on learning rate) Building and consolidate notes on fastai lectures, fastbook and docs Reading docs and experimenting src of fastai Learning excel along the way Python basics Fastai spreadsheets Jeremy’s spreadsheets from fastai/fastai1 and spreadsheets from fastai/course22. Sarada’s advanced spreadsheets on fastai. 5 Likes Daniel (深度碎片) July 14, 2022, 1:56pm #2 Daniel: Learning excel along the way Excel tips How to draw graph using columns of data? How to reuse Jeremy’s macros for all my workbooks? get developer tool ready open project explorer open Jeremy’s excel file and your workbook copy and paste the macro to the worksheet you want it to have How to debug or inspect data flow in your formula? How to set primary and secondary axis? How to write a custom VBA function with docs? How to construct a complex Lambda + Let function step by step in Excel? Daniel (深度碎片) July 14, 2022, 2:01pm #3 Daniel: What is the plan so far just the first step and possible future steps (e.g., on learning rate) Experiment and contemplation on graddesc.xlsx Questions experimented Jeremy’s original graddesc.xlsm, and my experiment workbook graddesc-daniel.xlsm How does ReLU (an activation function) make a linear layer/neuron non-linear? (demo by Sarada) Does converting a linear neuron into a non-linear one make training much easier? (experiments) How to calculate derivatives of weights with respect to error without analystical derivative formula? What happens to a 2-neuron model when you give it a ReLU? What happens when you train a 2-neuron model to find a simple linear function y = 2x + 30? How important is your learning rate to ensure training to get started and going? Derivatives of changing weights seem unpredictable, how SGD using learning rate and derivatives to manage weights toward optimal in most cases? How SGD loses control and derivatives and errors go exploding? 2 linear layer model without non-linear function in between is just another linear layer model. But why in experiment the 2 linear layer model is much worse than a 1-linear layer model? What happens when you add a ReLU to the 1st neuro of 2-neuron model? (train freely and 3 weights fixed and derivatives stay zero, does 1-neuron do the same? ) What does momentum look like and what is the intuition behind? Implement average_grad in Excel Daniel (深度碎片) July 14, 2022, 11:11pm #4 Daniel: Building and consolidate notes on fastai resources to assist the excel experiments Notes from fastai teaching You can search all fastai videos with keywords! Thank you very much @fmussari for this wonderful tool! What Jeremy said about Affine function and non-linear activation function and what does ReLU do for neuralnet in general? How should we think of deep learning jargons? Why is non linear activation the power source of neuralnet mathematically? What does Jeremy say about epochs? Summary What is an epoch? What could happen if you run 10 epochs with lots of weights and high learning rate? What would happen related to overfitting if you do just 1 epoch? Are these Jeremy’s experimental knowledge or there is some more blogs or papers can read upon? Radek on weight decay Do weight_decay vs l2_reg behave differently under the same but more complex optimization stepping functions? kurianbenoy (Kurian Benoy) July 15, 2022, 12:32am #5 Thanks for creating this topics and writing about this @Daniel and @Moody. Really appreciate this. 2 Likes Moody (Sarada Lee) July 15, 2022, 12:43am #6 Daniel: Excel tips Cautions: The tips I provided here are just to get you started. They are not the best practice to use Excel in terms of optimizing performance and memories. But, it seems you have so much fun already. 2 Likes Daniel (深度碎片) July 15, 2022, 1:02am #7 Moody: Cautions: The tips I provided here are just to get you started. They are not the best practice to use Excel in terms of optimizing performance and memories. But, it seems you have so much fun already. Yes, thank you Sarada! Daniel (深度碎片) July 15, 2022, 5:23am #8 Daniel: Notes from fastai teaching How should we think of deep learning jargon according to Jeremy? Screen Shot 2022-07-15 at 1.19.26 PM942×1140 91 KB 1 Like Daniel (深度碎片) July 15, 2022, 10:04am #9 Why non-linear function is the power source of neuralnet from a mathematical perspective? Quote from the fastbook Amazingly enough, it can be mathematically proven that this little function can solve any computable problem to an arbitrarily high level of accuracy, if you can find the right parameters for w1 and w2 and if you make these matrices big enough. For any arbitrarily wiggly function, we can approximate it as a bunch of lines joined together; to make it closer to the wiggly function, we just have to use shorter lines. This is known as the universal approximation theorem . The three lines of code that we have here are known as layers . The first and third are known as linear layers , and the second line of code is known variously as a nonlinearity , or activation function . 1 Like fmussari (Francisco Mussari) July 16, 2022, 11:31pm #10 I was playing with the Titanic spreadsheet from the course repository , watching again Lesson 3 and was wondering how to replicate the linear regression in Google Sheets. I didn’t knew of Google Sheets having a Solver (it seems that there are some implementations), but I explored other alternatives. The first one I found was LINEST. LINEST is a function (also available in Excel) that “calculates various parameters about the ideal linear trend using the least-squares method”. When applying it, a loss of 0.396 was obtained. So, according to the Google documentation it should do the job, but the Excel solver was giving us a much better result with loss of 0.1443. In order to achieve a better result, I was wondering on how to apply Gradient Descent but wasn’t sure how to do it. I thought about Apps Script but was a bit reluctant to do it that way (I didn’t want slow for loops). I knew about this article of how to use Tensorflow.js inside Google Sheets: Machine Learning in Google Sheet with Tensorflow.js and Apps Script. I thought that maybe using ts.js loss functions or matrix operation would help me accomplish the objective, but not being proficient neither in Javascript or in Tensorflow would play against the objective. But in the process I finally found all the Google Sheets formulas that allows to apply the Gradient Descent (in a relative simple way) and to get a result as good as Excel Solver. Formulas for Gradient Descent Explore the file here: Titanic - Gradient Descent. – LINEST: The first parameters were obtained with the LINEST formula. As I mentioned, its results produced a loss of 0.396 The other functions used to calculate the gradients are the following: – SUMXMY2: Calculates the sum of the squares of differences of values between two arrays. – MMULT: Calculates the matrix product of two matrices specified as arrays or ranges. – TRANSPOSE: Transposes the rows and columns of an array or range of cells. For calculating the partial derivatives a delta of 1E-6 was established. So, the partials were calculated as follow: ∂v1 = (LOSS(v1, v2, ..., v10) - LOSS(v1+delta, v2, ..., v10)) / delta ∂v2 = (LOSS(v1, v2, ..., v10) - LOSS(v1, v2+delta, ..., v10)) / delta (...) ∂v10 = (LOSS(v1, v2, ..., v10) - LOSS(v1, v2, ..., v10+delta)) / delta