Modeling with spatial and spatiotemporal data in R
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Hengl, Tomislav
Software requirements: opengeohub/r-geo docker image (R, rgdal, terra, mlr3), QGIS, Google Earth Pro This tutorial is an introduction to RStudio, and shows how to start packages and load data. It also serves as an introduction to spatiotemporal datasets, eumap spatiotemporal datasets, as for example: landcover 2000-2020 training dataset. It also focuses on visualizing spatiotemporal data and data summaries.
 So welcome back. We did the first session on a short training course on machine learning for spatial temporal data. The first session was about basically about spatial temporal data, what's specific about it. And I also show you some examples how you visualize space time data, what are the strategies to visualize. And then we also spoke about complexity of time dimension, how you do time series analysis. This is the second block here. We will also do a bit of introduction to modeling with special temporal data. And specifically, I will focus on doing spatial interpolation. I call it spatial interpolation, but in general also later on we will be doing some regression classification. So that's the focus of the second block and then the third block will be after lunch, we'll be doing basically ensemble machine learning. So we go another layer of complexity. And for those of you who are new ensemble machine learning, I will explain you that. I will show you how it's done efficiently, the framework and how you can basically combine multiple learners and also look at the effects of that on the accuracy and go step by step. So that's the plan for the second block. So I will in this session, I will also show you how we can model seasonality effects. And we have a little formula that allows you to model seasonality at every point in the on the planet surface. Then I'm going to talk a bit about machine learning, what are the standard steps. Then I have a really nice example with the synthetic data about the extrapolation over fatigue of machine learning. Many people start using machine learning without being aware of all the, let's say, dangers of using machine learning. And then they sometimes because it's very complex and it looks like a black box. Some I noticed some people make a prediction or some results and even send a paper to publish. And then they discovered that there was overfitting or there was some artifacts. And that can be quite a problem, especially when the paper is already accepted for publication. So, so that's why we're going to talk about it, how to prevent that. And then I will show you this patient interpolation, I will show you the thing that I used to do in geostatistics using a creating fitting viral grams, etc. And I will show you how does it look how you do that in with machine learning. So something I specialize with and the last paper we published went really viral. So it receives now over 100 citations a year. So they people got really interested in that. And we kind of make a bit creating redundant because we have a more flexible method to do special interpolation. So that's the problem for the second block. So that this is the data set I was showing you. And this is a this cook farm, it's a four dimensional data set. And as you see, when you look at many space time data, especially ecological data, you will see the seasonality effect. And so that's very interesting. So it's very regular, let's say, but differs a bit. Seasonality also, when you look at the seasonality, if you have a winter in Europe, in Brazil, it's a summer, some southern Argentina, let's say it can be summer, so it's completely opposite. So the climate is also even seasonality of climate is very complex, you know, you have tropics, you have moderate zones, you have a southern northern hemisphere. So also, even this is anality, it's complex. And so I'm super happy we published this paper in, let me see 2014. And Milan actually is a first author, it was part of his PhD. So he tried to tackle that problem of seasonality of daily temperatures. At that stage, it was just we were just focusing on daily temperatures. And he came to a formula, I don't know if you ever saw this, but imagine this temperature, daily temperature anywhere on the planet, actually, it's, you can kind of model it based on the angles. So it's a kind of sphere geometry. And this is the formula. So the two formulas. And if you put the coefficients of these formulas, then below here, you can calculate minimum daily temperature and maximum daily temperature. So you as an input, you only need the latitude, you need the latitude and you need the day of the year. If you know just the latitude and day of the year, you can model global temperature and that way you can also model the difference between southern northern hemisphere tropics, etc. So I don't know if you ever saw this formula. I'm still not sure if he derived the formula. First time, I think he saw pieces of the formula and then he combined it together. And then he made it as a single thing. And you can put it in, you can put it in R. And this is what it looks like in R. So this is a mathematical notation. And this one is in R. And you can, you can also estimate your own A and B, A and B parameters. So you can also estimate them. So I think that's kind of some type of a maximum minimum temperature, a zero latitude or something. But you could put that to estimate your own parameters. And then if I go to the R studio, so that's this modeling seasonal components, you can see that we put that formula here. Here's the formula. And as you see, I can go in, let me see. I can go and run that formula to some arbitrary day. So let's say if it's a 52 degree, which is we are now 52 degrees, then I get for day 120, I get 8.7 degrees, the minimum daily temperature. But I could also put it somewhere under 22 degrees. I don't know somewhere in Morocco or something. Let's see. I deleted something wrongly. Looks at this visual editor. It's not working. So let's say 22. And if I compute it, I get 23. You see? So it will change. And I can also play with the day of the year. So I can just keep on changing these numbers. And you can play a bit with it. And what happens, you get constantly, you have a different number. So that's basically when you put it in this formula like this, that I take five years, right? I take five years and I say, I want to look at the 52 degree in Oords and I want to look at all the days. And then I will get this plot. And so that's the seasonality. And so this formula, by the way, I can plug in and compute for pixel. So I can also make images because you see it changes with the latitude and day of the year. And so I could do it for every pixel. And then I create, I could basically map the temperature or geometric temperature for the Netherlands, for every moment. So a very useful formula. And we discovered that you can actually use it universally anywhere you have a day, a day or monthly values. So let's say for the land cover for these predictions here, we also use this geometric temperature as the covariate. So we also use it to model the land cover and the forest free species, etc. So we can use it for forest free space. No, we didn't use it, but because it's annual annual doesn't make sense to use, but we use it for the land cover because land cover we model, we can model per season, we know the seasonal changes. So then you can use it as you basically add it as a covariate. And it helps you, it helps machine learning because if you miss any other parameter that especially for larger areas that basically represent the latitude correctly, because if you use the latitude, it's not the same, right? Because latitude, it's like a X dimension. And then if you go, for example, from a latitude, the 42 to 52, you move for 10 units. But actually that's not one to one to change us in the temperature. You know, if you move from the 42 degrees to 52 degrees and from 52 to 62 degrees, then from the 52 to 62 degrees, the temperature drops much, much more rapidly. Okay, so you don't model the latitude and correctly, but if you apply this formula, this trigonometric formulas, then you can model really seasonality effects much more, I think realistically. And that's why this formula is super important. And as you see, this is with Milan took the actual data on the methodological station, this is super interesting. So what you see is, let me increase this. So what you see here is the gray line is the actual measurements of daily temperature at the station. So you can see the temperatures daily, they can differ from day to day. I mean, you can have some rainy periods, cooler periods, warm periods. So they differ like within a one week, you have a bit cooler, a bit warmer. And then this one is the modest, the broken one. Modest is a bit overestimating temperature looks like. So it's a bit above for these two points at least. And then the geometric temperature is the very smooth line. That's the geometric temperature. And you see the geometric temperature, the actual daily temperature reflects very well. And he was surprised that when he made this model for geometric temperature, he took the global data set. And he said this geometric temperature explains already 80% in variation in values. So you can already with the geometric temperature, you already, you know, 80% of ellipse you can predict. So that's just a nice proof of concept of that formula. But, but of course, there's a remaining variation that you would like to improve. And then the final prediction is to do this linear model and this linear model. So it uses the geometric temperature and modest, and it makes the prediction in between. You see that's in between the modest and geometric. And that's kind of already on sampling, you know, you basically have two predictions of the same thing. And you're doing on sampling, and then you see that you come closer and closer to daily temperatures. And that's what he did in this paper, basically, that's the that's the work. And so we really like that. And it's a really, it's a very good response. And, and we started later on, we started using that geometric formula, we can use it universally as a covariate for any modeling where you have seasons or days. Yes, so you can use it universally, basically anywhere, as long as I mean, if the error is very small, then the effect of latitude will be very small. Then there's probably no need to use. But as soon as you move to an area, let's let's say size of Netherlands, then it makes every sense to plug in that as one of the components, and you just add it to machine learning. Okay, so I started with the geostatistics, then about seven, eight years ago, maybe with less, I, I kind of started switching to using machine learning for predictive modeling. And this is the general framework that we still use basically for, for the machine learning for predictive mapping. It has a, we put about eight steps and you can read about this in this article. But let's say the first steps will be to define the area of interest and variables and consider all the inputs that you will need. And then you base on that you make an implementation plan. And then the second step is to prepare the covariates. That can take a lot of time. And also to prepare all the training points. And so that could take a long time because you have to prepare earth observation data, etc. Then you could, if you are starting, if you don't have the points, then you could do a new sampling, then we use a Latin hypercube sampling. That's the usual we do because the, let's say a more suited sampling for a multivariate regression problems, a classification problems. And then we will do the fieldwork, you collect data. And then you do overlay. In the case of space time, we do space time overlay. And sometimes you need to also make covariates, which have this, let's say some road network or accessibility for the Latin hypercube sampling. And then you create this regression classification matrix. And then from there on, it's just machine learning. So if you do everything properly, then you have the machine learning. And then you can, you can do generate also maps, eventually. And then you look at the accuracy and you look at the patterns and you say, well, we have a problem, this still doesn't work still. Then you could go and collect maybe new points, extra points. And then you remodel everything. If you call everything, if you do reproducible research, every step is in code. Then eventually when you finish the coding, if you add more points, then it's just a rerunning and repredicting. And that's a general framework we use for predictive mapping and modeling. So more or less all our examples, everything also you see in this training material, more or less all follows the same principle, except we don't have, often we don't have an opportunity to go do sampling. So that's a pity. Sometimes the samples, they are basically, they're defined how they find. So and sometimes they are purposeive sampling, and there's a bit of clustering and there's a bias over representation of areas. And that's a problem. And there's some remediation techniques for that in machine learning. But you cannot do sometimes you just cannot do magic. Sometimes we just do to have to extrapolate and predict outside of the, let's say, a feature space that we have for training. So that's the general approach. And then this is also just in text, inside the machine learning. So just if you zoom in on this part, the, so level seven. So if you zoom in on that part, so what you have here is the, you do the fine tuning. So you can do fine tuning, you can do, you can do feature selection. So you try to reduce model complexity. You can also do a learner selection. So you can just test the learners that are most significant. So you can do many steps. And then it's also very good to do kind of cross evaluation, cross validation. And then you estimate what is really your uncertainty and what is the most, most important variables, etc. And then you generate predictions and then off you go, you make a web GS and you get this thing. And then people can access it and use it. Notice that most of the GS we make, it has also URL bookmarking system at the moment is without authentication. So it's really purely open. But we might put some session ID. So at least we know if you have a bug, you said, I had a session last week and I had these problems. So we'll put some session ID. But if I take, if I just copy this URL, and I send it to some, yeah, if I open it in a new browser, if I send it to colleagues, when they click on it, they will get exactly the same view. And that way you can communicate about the same problems without even understanding GIS and space time. I don't know. So that makes it very accessible to also to non experts. So that's something I think we find increasingly important. Okay. When I started using our, I noticed one of my favorite books at that time was the carer book, car package book. I don't know if you ever use it, but it's very systematic. And I think it's the best way to understand also as introduction to machine learning, all the many of the techniques here, they come really from statistics, right? And this book, the beginning was about classification regression trees, you know, modeling different type of regressions, which are like can be even just standard GLMs. And I don't know. So this book, I really enjoyed that book. And I really, it was my, let's say a mind changer for me. And I started really thinking about, you know, it's, it's very important to do the cross validation, repeated cross validation, refitting. So, you know, many times try to do from different angles, you know, and then learn from there. And that's how machine learning works. Machine learning is a hybrid between computer science and statistics. And so some techniques, you know, machine learning, actually, they're really just statistical techniques, just combine into computer science techniques. And like three common computer science technique in machine learning is, for example, sorting, you know, sorting, then, let's say, finding a pattern, finding a combination that has a best match, best matching and things. And then learning is, you know, making decisions based on repeated cases, making some decision based on some, let's say threshold values, these are like computer science things. But then RMSE, ordinary, really squares, fitting technique, you know, this is often statistics. Right. So then, when you combine the two, you get machine learning. So please take a look at the title book, I highly recommend it. So when you look at this framework, you know, we do what this framework, this is purely spatial framework, as you see it. So how do you apply that framework to space time? And there's an extra layer of complexity, how do you do Latin hypercube space time, you know, it is possible to extend Latin hyper to space time, also how do you do machine learning on space time data? So it looks like I showed the space time data is no complex than spatial data. But actually extension, you will see many extensions to space time. They're not so, they're not so complicated. So as long as you, you pick up the key things that, that are kind of represented difference. So one of the, one of the things I showed you is the seasonality. Yes. And I showed you that seasonality, you can represent it with the, for example, geometric formula, and you can plug it in into a spatial data. So we can also estimate it for every grid cell. Yes. And then in that way, we represent seasonality in our raster model, raster stacks. Okay. So that's a very efficient way to then they represent that in the special temporal data. Then the other thing is, when you do, for example, overlays spatial overlay, which is relatively simple, let's say, when you do space time overlay, we made this extra functionality that allows us it's called extract, extract ST. So we made, I made a little function, which basically, it's a bit of complexity, but allows allows us to do a space time overlay. So I still use basically, I use a spatial overlay, but I made this little function that extract ST. So here's the function. So I made this little function that I subset the points in space time. So if I have a raster, imagine I have a raster and this raster has a begin and date. So it applies to some begin and date. Let me put it here. So if I have a time series of rasters, yeah, if I want to do a space time machine learning, obviously, if I have points, remember points like in a space time cube. So let's say this is the space time cube. So the points can be different, different places in the space time cube. Yeah, so you have the image and it's like multiple rasters. And then we have the points. And I want to overlay. So I need to know for this rasters, I need to know to which time period they apply. So let's say more these images, temperature images, there's a day. So they apply to the eighth day period. Eight day. Yes, but my points were daily. Okay, so my points are daily and these ones are eighth day. So I have to subset first all the points that fall into the time period. Then I get the subset of points if I have a one million points or 1000 points, then I take the points that fall into the eighth day period. Okay, and I have X, X, Y, T points from the original X, Y, T that fell. So that's like a subset. So let's put it something like S. So this is a subset. And these points, I can now overlay with this raster. And then it becomes just a spatial overlay, you understand? So actually it's not so complicated, space time overlay, you just have to match, you have to match points first in the time, then you match them in the space. So first the time, and then you match in space. And then you do overlay. Yes, and then because it's the large data, what this function doesn't do any parallelization, this function is just simply extract first subset. Subset, subset here is the select, select is the subset. So I said, it has to be less than the upper date and more than the lower date. Then it's subset the points. And then I do extract, which is very fast. Yes, there are, there are package extract. So the just the points over the raster. And then because it's a very computational to do it sequentially, then I make a function that goes and does that. So let me see where was I showing that here. Sorry. Then I make a, then I make it in parallel with the parallel package. There's a MCL apply. I make it in parallel. So if I have a time series of 1000 images, then I will just run it as fast as possible. And then my overlay will be as fast as possible. And you see, it takes literally to overlay 155 points going through one year data. So lots of space time points. I can overlay within seconds. And that's the, that's the, that's what we do. That's, that's the stuff I kind of created over the years. And now you can just go and use it. It will allow you that in all you can do as fast as possible. I don't know if it can get faster. Trust me, I try many things. This is the fastest with using the default packages for spatial analysis. This is as fast as you can get. And you can do the space time overlay. And the only as an input, you only have to say, you have to say what is the, for every thief, you have to say, what is the begin time of a thief and the end time of a thief. That's the only thing you have to say is this. And then you put it here, you say, this is the list of images. And then for each image you do overlay versus the meta values. And this is the begin time of a thief and time of a thief. And then you do the overlay. But we're going to show you that we're going to run that example in the last block. And then you can, you can look at that. So, so this is the way it's another way to move from space to space time. So we do a bit. So we are the seasonality. And then we do the space time overlay. And then once you get the space time regression matrix, actually then machine learning, it becomes the, actually it's the same or more or less the same, except you have to be aware it's a space time, always, but more or less the same principle supplies. So you solve couple of methodological problems, and you get to the machine learning regression classification matrix. And from there it becomes more as a central mellow. Can you please close the door? Then then the issue becomes that the data becomes very, you know, it becomes very large. That is one of the issues. And that's what I told you. And you will see it in this, when we do here do training. I mean, look, I have 155 stations, but it's a space time problem. Suddenly I have almost 45,000 values. And you say, okay, 45,000 nothing, but imagine when we had seven million space time points, and we have seven million rows with 300 covariates. And that's a problem. I cannot open that in laptop. You know, I cannot do machine learning on my laptop. Then you slowly have to move from using just PCs, whatever to using high performance computing infrastructure. And we don't, of course, any of our project, we don't compute anything on computers. We use all the servers, right? But we will test, we will make the core development. We subset this space time regression matrix, we subset 2%. And we do core development, you can do it on laptop. That's fine. And once you're done with that, then you put the 100%, you put on some computing infrastructure. And computing infrastructure, everything we use also answer, Buntu, actually, you have a GUI, you can, you log into this computing infrastructure, it also looks like a GUI and except it's the servers that cost, I don't know, 50,000 US. So, so you have to be careful what you do. But this is the space time going from machine learning to space time. So I got really interested, of course, in machine learning. One of the reasons automation, why automation? Well, you want to retire earlier and have more free time, obviously. That's why we like automation. So we have to do, well, you definitely will automation, you want to do less repetitive work yourself. Okay, everything basically in science and in life, anything you can automate, you should automate doesn't, I think repetitive processes, they are good, sometimes for meditation, they're good for relaxation, I don't know. But they're not good for creative growth. So they are not good. So any work, anything we go, like if you imagine today, people sit in the car and they drive to work and drive back, you know, it's really bad. It's really bad for brain. I mean, unless you feel meditate in the car. But let's say there's nothing to meditate, and there's no interesting radio station, you know, it's a waste. So it's best that we use our brain only for the creative things. And so anything you can automate, you should automate. So I really like that about automation. Then machine learning, it's also, I think, sometimes we have been statistics and in the classical methods, they have, they're very limited. So they're not so flexible. So sometimes I heard in sciences that even people look for data sets that match their methods, right? There are people who say, well, I have this method, I know how to use it, but it has limits. So they will only look for the data sets that match their methods. And of course, that's a wrong way to go in science. And science, the right way to do, of course, is to look at the complexity of world, and then see, okay, how do I represent that complexity? So like if you look at T-R-F-K-R-O-S, if you look at, I don't know, chaotic behavior of stock exchange or climate, and I don't know. So you say, how do we model that? That's a much better way. And if you look at the many things in the world, you need models, which, for example, can represent normal linear relationship, different distributions, different types of covariates, many covariates, everything influence everything. And that's where this models come, the neural nets, random forests. These models have a capacity to have many variables overlap, the relationship can be complex. And actually, when you look at just the random forests, actually, so generic, you can apply it to regression problems, classification problems, survival, modeling, whatever. So it's also very generic in sets of how you can apply to data. And that's what I got really interesting. The problem with machine learning is computational complexity. So the computational complexity I had when I did some linear statistics and geostatistics, as I said, it was an order of magnitude less, even sometimes 1,000 times more computing. But computers, if you look at them, you know, we are now somewhere 2021. And, you know, when they started, I don't know, this was the first computer, IBM tabulator, I don't see the year, wait. So this is something 1930 or something. And then the first IBM computers 1950, and you know, the operations they could do. So they were very, very basic. And now they say, yeah, you know, this Apollo, the space mission, you know, that also computers that spend billions of dollars. And now you have the same computing capacity on one iPhone, you know, the same computing capacity from 1960s. So when you look at further down the road, 2045, we are now getting closer to the brain power computing. I don't know how they estimate this computing. I don't know what is here on the white line. But apparently by 2000, let's say, 30 or something, we will surpass the capacity of brain computing in 2045. We will surpass the because it follows this more slow exponential growth. By 2045, we have computers that can take it all people ever born at once. So and that's a bit scary. And there's even this Ray Kurzweil wrote this, have you ever seen this? It's like a six epoch. And so we are now somewhere in the between epoch four and five, where technology is basically combined with biological life. And then when it gets to the high level, so you will see the biological life will disappear slowly, and you will have just the technology. And then you will have this highest level of intelligence, and it will have a capacity to spread around universe and also terraform the universe, not by traveling, not people traveling in sci fi movies, but just by electromagnetic light, it will perform the whole universe and turn it into a supercomputer. So that's the six epoch and that's called universe lake wakes up. Anyway, read about it. But that's a real extrapolation of more slow and many people don't buy it. Not every exponential growth is good. The CO2 emissions, very sad to see, you know, it's very exponential plot also. The CO2 emissions and population growth, population growth a bit slowing down, at least they think it's going to slow down. But CO2, it's been now from, I don't know, this real meeting and Kyoto protocol as all these meetings and we still admit CO2 exponentially more. Now the exponential growth started slowing down, but we still meet every year more and more. So that's a sad thing to see. But me back in 2019, I was very excited about geostatistics. Okay, so when I started out creating and I was learning I followed two courses. And then I published some work on maybe combining creating and regression and applying it also to space time problem. So I slowly published. And then we started doing bigger and bigger data sets. And then we switch I switched 2013. I first time I switch I wanted to do geostatistics of Africa. And then I saw, I don't know, I'm going to do this and points are clustered. So we just did the random forest and I saw random forest where I get the better accuracy. So it looks like it's more suited for these type of problems. Then I did some global data sets. And from about 2018, I stopped, I don't use so much geostatistics only use it for teaching. The problem is in science is that my most cited work is in geostatistics. So when people see me now, they say, I'm the regression cleaning guy. So that's what people tell me. You know, in science, when you reach a level, people will publish people site you then you become the term, you know, you're not your name or something. You know, you are the, you are this guy or that guy, you become a term. So I was there, the regression king guy, and I said, look, I stopped using in 2018. So officially, I'm not, you know, it's like with the other and our people, they made this SP package and that, but they would drop it a bit. So now you're the SP guy, no, I'm the SF guy, right. And also Robert Heimers, he made the roster package. So you're the roster package guy. No, I'm the terror package guy. So, so these things also change. But unfortunately, there's a bit of echo delay. And so I'm definitely not using a random regression cleaning and stuff. So I'm much more interested in, in machine learning. This is just to show you, like at that stage, you know, I looked at the ordinary cleaning. And, and I made this paper called mapping efficiency information content. And so what we saw that if you plug in this regression part, and you combine it with cleaning, you get the map on the right. And then the uncertainty drops here, we kind of visualize the answer to the very simple, we mask out the pixels that have the error larger than the variance. So we kind of mask out this pixel because it means like you don't create a nice mythic and prediction. And when you switch from this model to this, you can see that you can map more pixels. So very simple. And you can see the patterns, you know, they're a bit changing, it becomes like more and more detailed. So that was this paper. So that was that time I was really excited about that. And then we started looking at this using a random for us to do interpolation. So basically, not as a combination of geostatistics, but really just as a replacement. So I just want to show it. So this is like this creating technique. Do you use creating? Have you screenings? Yes. One, two, no, no. So so in creating, so you have the points and you fit a variogram, a model of the spatial dependence, you need to decide about transformation to understand your statistics, you cannot just like I'm just going to do geostatistics. So you have to know what you're doing, but the many choices are new. So you have to pick up the transformation of data, you have to pick how you're going to fit the variogram. You want to do an isotropy, yes, no, tata, tata. So you have to pick that all the top and then you fit a model. And then you get the predictions. And let me show you. And then the regression creating is to add to that also regression model. But let me now show you how it looks like. So when you do model based statistics, you know, you say it's a GLM, and you have the transformation, you have the, you have to specify the co-variates you're going to use, and you have to specify this link function. So you have to put all this up. This is this model based geostatistics or geostatistics. And then machine learning, very different. You just say, I want a quantum regression random forest. And off you go. Okay, so very, very different. And there's a famous article by Leo Breiman called two cultures. And you should read that it's a really classic. If you want to get into machine learning without leaving that article. And also, the other articles that follow discussion articles, you will, you will, you will never get to machine learning. So please read that. Leo Breiman two cultures. And basically, so what he says that these two cultures just to summarize it. In both cultures, they have different obsessions, let's say, and they see the aesthetics of things differently. So when you do model based statistics, let's say, so like just when you think about Gaussian distribution, Gaussian distribution, you represent it with how many coefficients. You have the mathematical formula for Gaussian distribution, and then how many coefficients you have? Do you know? So it has two coefficients, the mean and the standard deviation. So you have mathematical formula, and you have two coefficients. And this is beautiful. You have something which is using statistics all across, right? And universally many fields. And it's a represent you can represent it with two coefficients. And the beauty of that is the simplicity to coefficients in your model, different distributions, then you can have maybe a skew distribution, I don't know, you can add a coefficient. But otherwise, the most simple case, we have two coefficients, and you model the distribution of errors, the distribution of anything you measure. So it's really beautiful when you think about this. So the beauty in the model based statistics in the figure out what is the simplest possible way to present something universally. And that's in the case of for Gaussian distribution, that formula that Friedrich Gauss made, and he says it's two coefficients. I mean, maybe it could be done in one coefficient, I don't know, but two coefficients already, it's very simple. When you do machine learning, it's a different direction. You don't mind that this random forest, it can have 1000 parameters, you don't mind, it can have many parameters. Because the beauty is the how good you learn, how good you match the data. That's why it's called also data science. It's not the essence is to understand the models. You have very generic algorithms that come from mixture of computer science statistics, you have very generic algorithms, and then it's just how efficient are you in finding the structures and data. Yes, and that's the second culture. You don't worry about the beauty of the models, you don't worry if there's like, if it's Friedrich Gauss or if it's 1000, if it's two parameters, no problem. Yeah, but you do way more computing. So these are these two cultures and I kind of switch to that. Yes, question. Yes. I don't think you can easily say this technique is data science and this technique is model based statistics. Sometimes there's a transition. So there's some things that have elements of learning and repetition and sorting and then some has elements of just, no, this is the model and we just define the minimal complexity model and that's fine. So I don't know if there's a straight line that I could say, I don't think so. So it's a bit of fuzzy term, you know, but in right, yeah, super interesting. And in right, I think you can combine it in an ensemble system, you can put it in a sample system, and then becomes one of the learners. And then you apply this data science machine learning on top of it. And that's fine. So that's how I would do it. Right. So if it does work, but I think this by using methods as far as understand, you need to know for every observation, you need to also know some uncertainty or some prior, I don't know, and that makes it complex because you don't have it from any data. So it is, it's a specific method that you can assume that it's all the same or something, I don't know. But the real effect if I understand that by easy and correctly, the real effect is if you know these priors, if you have them like first estimate or if you have the worst uncertainty data, you know, then you have the real effect because that's the thing you don't get so much good in machine learning. Right. That's how I see it. But I haven't used in law. So I cannot, you know, give you a more detailed answer. I haven't personally used in law. This is the geo package, just statistics. So the same thing as I was showing here more or less, but this was just a regression. Now in geo statistics, you specify the lambda, which is if it's a lambda zero, it means a low normal variable. And then you have to say this range of you want to use to fit the diagram. And you have to say which a variable model use. So you have to specify these things. And then you fit the diagram and then you do the cringing. And then you have all the steps that goes fit the model, do the box, cross transformation, then on the cringing, et cetera. And then why are you sometimes you have to do on isotropy. So you consider also extra parameters of higher ground on isotropy. And you can see that it is sometimes in different directions, the diagrams can be different, like the red and the blue, they're quite different. So you sometimes you have to do all these things. And so what I started looking at, okay, let me now look, what if I just, I just do like this, it's a, I just take a buffer distances to points. I take the points and I take buffer distance to all the points. So there's this formula. And then I run random forest on this. So no, I don't have to know any geo statistics. I don't have to pass any parameters. There's nothing is just fully totally automated. Nothing. I don't have to know model, I don't know modeling assumptions, no distribution, nothing. Okay, I just say, here the points buffer distances. And this is where they go to the beginning. This, the left one is the state of the art cutting edge geo statistics. The one on the right is just random forest on distances. And I showed this to a few geo statisticians and they're, wow, this is, wow, this looks quite similar. Right. And first, what is similar as smoothness. So machine learning finds the smoothness in the values, right? The values, they could be changing like, or the shorter distance or longer distances. So it finds like, it's almost like recognize the, the, the, the, you know, and it matches it. And then it's, it's in between the values, it's kind of like the values like interpolating. And, and I repeated that for many datasets. And so again, you're also creating random forest. Here you have even an, an isotropy. It's a one famous dataset. It's a very anisotropic. So you have to model an isotropy in Varder. And then the machine learning find also this anisotropy. You see, it's refined that the, the values change more in this direction than in this direction. Okay. So, and, and you can estimate also the, the prediction error, which can be quite different from a key. So get more detail. So I got very interested in that. So I said, wow, this, with machine learning you, with the key, you have this stationarity requirements, you, you, the distribution should be normal. Then you have to fit the Varder. Then you might have some hot sports and they can be a problem and you have to do some, something in geostatistics. And with machine learning, none of this actually don't even have to fit any variety of nothing. And actually you can also extend it. You can add the more complex distances. You can add also distances in the watershed. You can add distances to some sources of pollution or something. So you can put everything together, which you cannot do in the geostatistics. So you just, you can just throw it on a pile. It's unbelievable. And even when you look at this, even 155 points, I calculate 155 buffers, this one co-radiates for one point. And for machine learning, it's no problem. So technically, I can run it here in the code. Let me see. So here's the buffer distances, right? Let me run this code. This takes a bit of a time. You see, it takes, and it pulls up in a RAM, I think. It will pull up in RAM because I'm not running it in parallel. No, actually, it is running in parallel. But it takes five seconds because I have to, I create 155 maps. If I have 1000 points, I need to create 1000 maps. And let's plot one map. So let's plot this one. So this is a buffer distance from a point somewhere, I don't know, here, this point. So I have 155 buffer distances. I mean, I can go to some other point, I don't know, 28. And it's all different. So you're moving around, right? So these are all just buffer distances. Yeah. Yes, here, no, nothing. That's random forest. As long as if it's non-linear or skewed, it will try to get to the data. No, nothing. You just try it, and you get, often, you get a higher accuracy. So here's the model. And it estimates that it explains 50% of variation based on out-of-bag, which is already parallel with the estimate. And then you make a map. And so it looks like this. So here's the, so it looks like basically like a crevice. And I can plot it already in the, I did plot it in the, let me see. I didn't check with the land or the, what's up with the, but I will just copy here. So let me just copy all this stuff. And we can open it here in Google Earth. Let me drop it from the, so here's the prediction. I just plot it now in the Google Earth. And we can see the points and the interpolated values. These are the, basically observe the values. And this is the interpolated. And so it looks like it's creating interpolation. Even this point, you see, it's a higher point. It tries to come close to that point. So it's a really magical that without any geostatistics, you can go and make predictions. I can play with this. It was very nice in Google Earth, of course, to get this super high resolution. So if I set up a transparency, you could actually, let's say if this was heavy metals, and you can see whether there's a town or something. So you say, oh, wow, we have a really, people have to be careful here. Right. So you can see where they're to last the town. Also, that's very nice. You see here, predict water. Then you say, what's going on here? Why did you predict water? Well, we can look at the time scale. Let's say here. So we can go back to the time and we can see that actually this used to be very dynamic. So sometimes it floods and the water goes up and down. And sometimes the since it's very dynamic, this area. And if I go maybe way back, when they did the sampling, maybe then maybe it wasn't maybe it wasn't water body. So you can also scroll through. You see how the water changes. So that's also good to important to be able to interpret the results and see how do they match with the actual space. And you can you can play with looking at it 3D or space time. So so that's that's this data set. So you never you said you never experience anything like that. And yeah, we made a paper as I said, and this is this paper. Where was it that paper? Or is it? We made the paper and then we also made this tutorial. And then we kind of worked out, okay, that was just a quantity variable. Now let's look at the categorical variable. Let's look at the extreme values. Like look at the space time, multi-variate problems. And then I saw, wow, this random forest, you can use it as a really generic case. So you can use it generic case for a spatial interpolation. And it's horribly direct, you know, it's kind of demotivates you to learn geostatistics, which I don't I don't recommend. I think I still I like the idea of fitting variogram, you know, variography. It's very good because you can you fit something and you understand, and you have a proof in the model that there's some spatial dependence or people what trains etc. So I do still fit variograms. But I don't use it to make the predictions. I don't use it so much this paper here. So that that's the paper we publish and it has a lot of attention and many people now test similar approaches. And the other approach is even more brilliant, calling from Belgrade. He made this approach with using distances to the neighbors. So you calculate the distance to the second. So the first second, third up to 10th neighbor, and then also the values of the neighbors. So and that even closer to geostatistics, but because you actually, you use the values of the neighbors and the distance to neighbors, I just use no distances, right? It's a bit primitive, let's say, but he uses also the values. He also got good results. And it's also super simple and he applies it to space time also. Yes. Yes, yes, very similar to Kriging, but it you fit a random forest model. I'll show you also how this is done and you comply to your data. So at the same time, when I got interested in machine learning, this world champion in Go, he was, have you watched that documentary? It's excellent, one of the best documentaries I've ever saw. It's very tense, you know, there's the Google team and this person, I don't know his name, but he was the world master and like Lee Seedle, Lee Seedle. So and he said, I'm going to be a bit, beat you guys five zero. Yes. And you know what was the score in the end? It was a four one for Google computer, the deep mind. And he was surprised, he couldn't believe it. He couldn't believe he's losing you, but he won one game and he made some moves. They call it the move for God. He had to make some moves that had never been made. That's the only way you can beat computer. So you go really beyond and he made some moves and the computer couldn't never had that because computer is, I mean, it's not, there's no intelligence in computer yet. We don't have any intelligent computer, right? It's just if he has enough data that those data mining and fine stuff, then it can, you know, implement that knowledge. But this guy had to do completely new move and that's how he won one game. Then after that they played some much more complex starcraft. And this one, they said no way computer can beat and also they lost because this is way more complex, like the chess, you know, the like strategies and rules that you could explain them on a book on 20 pages. When you play something like this, I mean, the strategy is like 10,000 pages. Yeah. And even more, there's no end, I don't know. And they're also, they also won. And then even these people made the software after this game that they won, they published a nature of paper where they trained the machine to play with itself. It learns by teaching itself. It doesn't even human knowledge anymore. It will generate, it will simulate things and it teaches itself. So it plays the games that human, it didn't even play ever. And it learns from that. It's really scary when you think about it. And so the things are really taking, so we are somewhere here now. We are somewhere here. And so let's say when the things cross, I don't know what's going to happen. But again, in that way, when you look at it, it's most likely going to be some merger, I don't know. So this is all like really great, right? It's fantastic when you think about this. Very simple. You plug in the data and off you go. You don't have to think about anisotropy, modeling, parameters, distribution, nothing. But it doesn't work always. So there's also cases when the machine learning doesn't work. And I also, I burned my fingers. I did some maps where I did extrapolation, I created artifacts. And we did some overfitting. And so I learned the hard way about also drawbacks of machine learning. So number one drawback is the extrapolation problems. Then you have also bias in training data. Data science means data is super important. As you see the metals are secondary. The data is important. So if you have a blunders in data, you have a typo, you know, somebody puts instead of zero comma, and you get like instead of five, you get 5000. And when you think it's a small error, right? But on machine learning, it has huge impact. So you can really, just from few blunders, like a 0.1% of data has blunders, they can mess up 80% of predictions. It's very sensitive to blunders. You have to spend more time with the point data. So you spend instead of like in data science, you spend in, sorry, in model based statistics, you spend a lot of time conceptualizing, picking up the model, do I use this by easy and do I use this? That's what the learner, the analyst spend time. With the machine learning, you have to spend a lot of time cleaning the data. So you do extra, actually, data mining, and you look, is this an artifact or not? Is it, you know, checking the data? So you have to spend much more time in the beginning and on the after the machine learning. So in a model base, you know, you kind of in the center, you spend the most time the modeling. But as I showed you here, you don't have to do any more than just show the data, you get the values, but you spend more time in the beginning on the end. So checking the input data, and then after doing expectation and seeing whether it matches the everything matches the input data and relationships, etc. And so also I did the overfitting, overfitting also a big problem. And then you get the bias. If you overfit to get the bias and, and also then in AI now, they say, yeah, there's a problem in any machine learning you train with the bias data. Yeah, this bias is going to propagate. And who is going to be to blame when you run the machine learning and you publish the paper, but the biases in the training data, you are responsible. Trust me, everybody call me responsible. They never said, there, it's fine. No worry, it's the data. No, they say, what did you publish? It was your responsibility to check the bias. Yes. And now also Facebook and all these companies that use this automated phase recognition, whatever, you know, for job, you know, they had the model where they say they will predict where the candidate will get the job or not. Did you hear about that? And the model just went, yeah, okay, the first thing most important variable is the sex. If it's a female cut it down, just take out the males. So you'll get arrested for discrimination, right? But the models was fine. I mean, it was because that was the data, but the data comes from the past. And you apply that model to the future. And that was a problem. When you do a special prediction, these are very memorize this figure. When you have a clustering of the points, and you fit a model with the cluster points, and then you slowly start removing for cross validation, you start moving the using only the neighbors which are further and further away to check the models. Then they got the plot like this. So if you take the cluster points, you fit a model, you get a 0.96 accuracy. And then you slowly start testing a model. How does it work when I separate the training points from validation points in space? Let's call a spatial cross validation. And they saw this that it actually drops to 0.5. So it's a really big difference. Imagine here, 97, 52. It's a 50% overshooting. But later on they saw, okay, this thing when you move points further and further, it's also eventually you create a two pessimistic estimate of your model. It can be two pessimistic. You force a model to be too strict. But let's say the truth is somewhere in between. So probably their truth is that they have maybe 60% accuracy. If they would just randomly throw the points in an area, an estimate accuracy, they will get maybe 60%. But they got 97. So that's why it's also important to apply that. Now, this is a famous example. It's called, it's a little board that person wrote, Peter Ellis. And it says a random for extrapolation stuff for the trees. Have you ever seen this? So what you see here, this is a similar to data set point data set. And if you fit a linear regression, you can predict out of the space, and you will predict the values are here. If you do random forest, random forest, it has no structure basically assumes no structure. That's why it's very nice. It's very flexible. But the door back is because it assumes no structure. When you come to the edges of the training space, if it goes beyond the edges, it can only use the last element. So it says, well, this is the last element. So it must be this. And basically, it predicts all the weather from the last element. And this, these things people don't like. Of course, because you see, look, this data has a structure. That's very clear structure. And the random forest predicts this. So it loses that structure. And we feel like, Oh, this is over shooting. It means anybody applying this model to decisions with extrapolation, it's making a mistake. Well, most likely we don't know in this case, you know, it could be some models, they had this break, you know, but let's say if you have 100 people and you won't tell them what which model would you use 100 people, 100 people will say I use the model on the left. Okay. So we can do that. We can play with that this thing. So this is the extrapolation overfitting. So we can play with that. And I wrote this little post in the medium. So let's see. So here's this post you can read about this. I basically I don't make it as the code, but the more like in a human language, explain all the things. And but we can also play with this in the the studio. And so we basically this, this is the simulation of this model. And so what do I say, it's equal to x, x is y is equal to x plus the noise. Very simple. That's basically I simulate a linear, perfect linear model with just the noise component. Yes, and then when I go and I fit a linear model, I will get the R square of, let's say 0.85. And you see the RMSE 11.8. You see this RMSE should match this one here. So it's almost close to the 15. So it's approved that the linear model for linear structure, it will do exactly what you expect. So it will estimate the correct RMSE. And it will say that yeah, 85% of the signal is the noise. I could have increased this noise if I had a higher standard deviation, then the noise gets noise component gets bigger and bigger. So that's a very simple way to reconstruct that. And then we fit a random forest. Yeah, so imagine so now I'm teaching you random forest, great, you can automate it, you can retire earlier, just show your data, and you come back and oh, here, projects delivered. But look at this now, we get about the same variance explained. And there is a package called random forest allows you to also estimate the prediction errors for anywhere in the plot. So we do this random forest to fit it. We actually get the RMSE of the random forest is a bit higher, 14. But it's about the same, it corresponds to the 15 simulated. And then when we plot it, he's a nice plot. So you see the uncertainty of the linear model. So not necessarily you see the linear model fit the red line. And now what happened and this is where the problem starts happen. So you see the blue line is the random forest. So, so first thing about the blue line, it, it, it also is up and down. Okay. And that that has that means that this random forest is trying to feed the noise in a way, right, because it goes up and down like it tries to feed the noise if I make another simulation, and I repeat this, it will the curve will go somewhere else. So and it basically randomly goes up and down, because I know that's a simulated noise, I know it's a noise. And even bigger problem is if you go to extrapolation space, you have this last element, and then you apply this random forest error formula for uncertainty, and it puts the uncertainty here, it's lower than uncertainty where we have the points, where I have the points that are in the forest, that will get the uncertainty, let's say, relatively correct. But as soon as I move outside, we have a problem. And this is a visual, visual explanation of the two main problems of machine learning that will happen with you if you blindly throw the data and just run random forest without thinking what am I doing here and do I have some clustering problem, do I have some structure that I'm ignoring, etc. How much I'm extrapolate, etc. And so that's a problem. And that's many people say they're really basically criticized, and they really find it, you know, you shouldn't be doing that. And so what is the what is the solution to that? How do we prevent this overfitting? How do we solve ourselves? So many things, you can do many things. Number one thing is to basically to be aware that there's overfitting, and then to study there's extrapolation, and then to try to see how much I extrapolating, extrapolating, how much where will be the extrapolation and how many made this little functionality to estimate so called area of applicability. And this allows you to see which of the pixels were extrapolating. So first you should be aware how much is it like 10% is it 50% you know, sometimes you most of the pixel extrapolating. So you have to be you have to consider. Then the other thing is this book by Chris, Chris Molnard, fantastic book online, you can also get a good book, maybe not Markdown, you can read all the diagnostics you can do to analyze the models and to see whether there's strange things. And one of the classical things you do is partial dependence plots. Yes, so you can take if you fit a model, you zoom in, you do lots of diagnostics after you press the forest, it's okay, I got the model, I got numbers, blah, blah, all automatic. But now I want to see what did I get? What do how does it look like so you can plot this is the machine learning fitted regression. Yes, this line and you see the uncertainty of machine learning. And you see where the extrapolation it grows. So that looks that looks okay. But for example, here, I have this curve going down and it goes up. You see the curve that goes down, and then it goes up. And this doesn't make any sense. I know from physics that once I have, for example, by density, if it goes lower by density, then the carbon goes up, it cannot go by density cannot go up again, if I have more carbon, it doesn't never happens. Almost physically impossible because of just how you compress matter. So I know that this, there is a problem here, this doesn't match any physical law. So I see there's a problem this curve going up, it's a problem. How do I solve it? I don't know now, but at least I detected there's a problem. And one way, you could say one way okay the easiest way if it's machine learning, you can add more points, you can do more measurements where I have a higher organic carbon. And then I add more points so that the models goes okay, and then adjust. Right, that will be one way to feed it with more data. But maybe you could also, maybe you could limit the prediction to the point where you, the model starts going strange. So you could say well I only predict to this values. And I don't predict because I know the model doesn't do well. So these are this remediation techniques. Universally for the board for the extrapolation and for the over fitting, I found out that one mediation technique is just to do ensemble. Just ensemble, the ensemble helps me prevent this overfitting. And I want to share that with this data set. I can show you because now I switched to the MLR and MLR is a framework that I'll use in the afternoon for the ensemble machine learning. And so here I can say, well, I'm not going to do only random forest, I will also do GLM, I will do a gum boost, and I will do a kernel support vector machine. So I do four techniques, which are very different, the very different techniques, they really different ways the data is used and how the, you know, parameters are fitted. So I will use four techniques. And I specify that as a task, a machine learning that's called a task. And then I specify that I want to do a stack learner. Now we'll explain the afternoon what is a stack learner. So, and I specify that I want to do a stacking by a cross validation with the refitting. And then I feed that model. Now it's a bit, there's a bit more code. And it's also a bit more computational, even like the small data I have, it computes maybe two seconds. And here I get this model and this model shows that I have 0.85. And it shows okay, R square is a similar now to the linear model. So, and now I can plot again that I can plot it. And do a prediction. So it's a way more code, so definitely more code, more computing. And then I plot it. And let's say this is now comparison between, if you remember the last model and the ensemble model. So here I go, I do now machine learning, but I do ensemble. One of the models I use has a simple structure. It's a linear model. And this model, of course, when I look at the result of machine learning, so here's the summary. This one. If I look at this model, you see that this stacking, stacking find out that the random forest actually, it's so badly, it's a so least important, that we could remove it. If it doesn't have a one star, it has to have one, two, three stars, that makes it significant. So support vector machines and random forest, I could remove. Yes, but I don't remove it manually. I got it from the modeling. So I could do extra code to remove all the learners that don't match. Then I run it. And then in this case, I didn't remove it because when they have a small, so this beta, you see, beta is very small, 0.7. And the beta for the GLM and for gumboost is 0.4. Sorry, four. So it's about 80 times bigger. Okay. It's about 80 times bigger. So I don't even have to remove it. The beta is so small that it has no impact on prediction, or minor impact. And then when I put it, we get this one. And now let's do some interpretation. So are you more happy with this plot? Why are you more happy? So the uncertainty, this uncertainty, the estimate is really wide. It's really wide, right? Maybe it's even too wide. But let's say in this case, because we want to use nonlinear methods, we don't take out the random forest, we want to use it lonely there. And then we see that between different methods, values go up and down. And then you say, well, the uncertainty is high. The fitting of noise is much, much less. The blue line is almost close to the linear line. It does, however, predict here a bit higher values than linear model, here a bit lower. So it does predict values going this way. If I repeat this many times, every time I will get a bit different curve, you can test it yourself. That's a nice thing to do as I exercise, change the seed value, and just rerun it, change put another number, rerun it, and then you will see the curve changing. You can play with that curve. And this way you learn also, you're basically doing machine learning on yourself. So you learn about how these things change. But definitely sounds like this could be a remedy for the extrapolation and for the overfitting. So it could be an ensemble in itself, it's a good, let's say, remedy. Even better will be if I would add a function here to remove the learners, which have a very small beta. Yes, I could just remove them because they otherwise they increase the uncertainty. Yes. So I could remove random forest support vector machines. Actually, I can do it now here. Let's see. So I basically rerun everything is the same. Everything is the same. I just go in the where I define the learners. I can put a new line. Yeah. Maybe I'll have to restart my session. So this one, I go like this, and I mask out this one. And here I have to now remove, let's say, fire removal ranger. Things are not what you see. Not what you see, what you get. And let's say I put here on the end. Something like this. Okay, so I changed the, I changed the list of learners. And if I then rerun the whole thing, let's see. I have to go step by step and rerun. Ah, okay. And let's see. So I made a typo. Ah, okay. Okay. I'm going to rerun this one. And then I get the summary. Let's see the summary now. It's still, I still have only two learners significant. This end net, it's less significant. But let's see now the uncertainty. Okay. And here's the new plot. So now the uncertainty seats are much, much, much more narrow. Do you see, but I don't use any random forest support vector machines. Do you see this? So uncertainties way more narrow. Okay, so that's the, that's the effect of, if I would, for these data instead of just universal. So it's a universal space time function it will work with energy that are compatible data set time series. Okay. But the force doesn't make sense to use it only for spatial problems. You have to have at least three, four times, you know, then you use it. And also, by the way, do you need to space time. How many time repetitions do you need to do space time? What do you think? How many, how many repetitions certified. If I look at some feature, and it's changing through space time. This is the time right. This is the picture. And this is a space. Now I put it out of Iran and space I put only one dimension like X or Y the latitude. So how many points I need to be able to do space time regression. How many points spread in time. Repetitions or what do you think is it okay to have to have like something like this. So space time regression. So I have like just two years when there's observations or something. Sorry. Well, technically you could, you try to machine learning and you will get a model. But think about this. Imagine, imagine our time is not time, but you want to model distribution between so based on the size of the shoe you want to predict somebody's weight. And there is a longer foot you have the taller you are right. So it is let's say close to linear. So if you have two points you could fit some linear model. But you could make a lot of mistakes, very tricky to do with two points. Right. So, if you have three, three, let's say, three then linear structures fine but as the, let's say the structures are more complex. And then if you don't, if you don't have enough time repetitions, it's really on the edge of doing space time. And I just wanted to be very careful because we did today with the machine learning you can just throw data. You just throw the data and you get a model and people say wow great to go to model without thinking and the many people I see they hurt themselves with machine learning when I say hurt not physically hurt. In a sense they make. They, let's say overshoot with conclusions. They overshoot with confidence, all these things I show you with the simulator data set. It applies also to complex data sets or the underestimating the extrapolation arrow, overfitting the noise, everything applies and creating artifacts. Yeah. And so this all this applies in synthetic data applies to the real data. So that's why be careful. If you have two repetitions time, probably I wouldn't do space time, even though you can fit the model. So the questions can you do it. Yes, you can. But the should you be doing it. 80% of cases I will say no, don't do it 90%. Either we get extra points in time, or you just say I cannot do space time. But maybe in some extreme cases you could allow. If you know that it's a linear structure. Maybe I would allow it. We have the random forest algorithm use of all co artists select some of the build the random for some use all colors or some. You can do a co bar selection is called feature selection. I didn't run it here but there is a code in MLR. Also, if you have 300 covariates, it will run a learning process to remove the covariates that are not significant. It's a similar to a stepwise regression and all these methods. It can be also forward forward, looking or backward looking or forward backward. You run it and often we reduce 50%. When you have a lot of co artists you can remove 50% often. Sometimes you will remove only 10%. What measure of uncertainty this one that I was showing you in this plot. The first one is the prediction error. So that's the error simulated predicting by doing a random sampling, simulating what will be the actual difference between predicted and model. Sorry, predicted and observed. So that's the different funds. There are different measures of uncertainty but most of when you do predictive mapping. Most of us are interested in the prediction error and prediction error is at the location. So, for any new location you would like to know fairly pixel what is the prediction error. If there's no more questions, we can stop with the YouTube. I will stop with the live YouTube. I have lots of in the chat also. Okay, people complain about the video problems with installing the virtual box. Okay, I will come back to your questions, but for now I have to stop. Have to stop with the live sharing. Stop live stream. Okay. I have a lot of questions here if there are questions in the room, please. My hope is that you are seeing a lot of new things. Is that correct. Yes. So that's a good thing so you seem a lot of new things what is mind blowing. What was mind blowing for you to see. Okay. Yes, it's good to visually to remember that. And these are really school examples this plot. This is like a school example everybody uses. But also the one I developed it for the but it's the same problem. It's absolutely also overfitting you know I also was thinking that many techniques are kind of noise proof, but when you, when you fit a model looks like it's not noise proof because as you see it will. It goes through the noise, and we know it's noise because I simulated it so I know it's not. No maybe there's a structure in data know it's noise and then of course cannot. There's a noise proof. Okay. I like the concept of area of ability and it's a good to compute it to check, you know, to plot it to see. But I actually think when you do ensemble machine learning, if you do things properly, the estimate of uncertainty is your area for ability. So it's very simple where the answer to become so high. And this is in this article. And here if you go all the way to the real data set I do also with the mass data set. And as you see here I estimate this ensemble uncertainty. And as you see that's one point it starts growing so high that you could cut out. And that's your ability you just use uncertainty if it's a good estimate. Let's say if it's a robust estimate of uncertainty. And you can say well when the variance goes above the variance of the data. So maybe somewhere here I don't know. I would just cut out and say, don't do prediction here. So I would just mask out the pixels. And then it's a more universal way to do it because therefore, the ability, you know you have to consider that it's a proxy. And this is not a proxy this is a model estimate. So maybe it's easier to compute because this one it's more computational, but this one you can call it then you can automate. This one it's in the, it's in the slides. I think I put it in the slides otherwise fine I will put it here in the, in the metamask. I put it. So, that's the position. Okay, so you can. So it applies to anything. It's called feature space, but in future space you see I can use distances. So it applies to anything. Yes, in the, in the when I use the MLR. I will hear I will set it up. And when I do ensemble. So I specify here. So, I say it's tech CV. So the learner. So it has a two process I'll speak about it in the after that, but it has two steps. And so it's actually it's a lot of testing. There's a lot of model fitted behind that you don't see. And, and so when you do hear the, the splitting, it's done by repeated refitting the stack CV it means repeated refitting five fold, but yes, the way you decrease the overfitting is by doing this stack CV, because you repeated it many times, and then you know. Yes, so then, because with the stacked stacked ensemble stack learner, you have two modeling. So you have these divisional models, and then you pay the many times, and then you have this so called metal learner that learns from this individual models but learns to cross validation of these models. So if one model over fits in the second round, you will see the overfitting and then this values goes down. That's why the random forest the value drops. It's become irrelevant you could remove it. It's insignificant. That's why that's why I find it is a remedy on some machine in itself is a remedy to prevent overfitting. Yes. Yes, and then also remedy for the extrapolation to get the correct estimate of the uncertainty and to also not to have the line going like this but to straighten the line. This kind of whether this example should know space. This is just a model. No space but if you extend it to spatial problems. So like here. Here there is a space in world but I don't use the space in the modeling like I'm the example I showed you this is just distance to the river. So actually it is kind of geography. So here I apply to the real case and now I use it on a actual spatial data. Right but it's the same the code is the same. Many things when you program as a third you in machine learning I have I can do just machine learning, right, and they do machine learning on images photographs whatever. But now I say I want to apply machine to spare spatial problems then I look at coordinates of the distances you know, and then I say oh man I want to space time, then I have this is a knowledge the effect and I have the, you know I have to have a time series of remote sensing data I have to fill in the gaps you know, but eventually when you do everything correctly you finish with the regression classification matrix and there's no change there from machine learning is the same. So it's the same but the preparation of the data is different. And so when you get to prepare it, you should do everything properly. Then it's the same it's just a machine learning problem. You just have to learn the model line avoid overfitting and remedy extrapolation. You know, so it applies with a spatial space time or non spatial. It doesn't matter.
 So welcome back. We did the first session on a short training course on machine learning  for spatial temporal data. The first session was about basically about spatial temporal  data, what's specific about it. And I also show you some examples how you visualize space  time data, what are the strategies to visualize. And then we also spoke about complexity of  time dimension, how you do time series analysis. This is the second block here. We will also  do a bit of introduction to modeling with special temporal data. And specifically, I  will focus on doing spatial interpolation. I call it spatial interpolation, but in general  also later on we will be doing some regression classification. So that's the focus of the  second block and then the third block will be after lunch, we'll be doing basically ensemble  machine learning. So we go another layer of complexity. And for those of you who are  new ensemble machine learning, I will explain you that. I will show you how it's done efficiently,  the framework and how you can basically combine multiple learners and also look at the effects  of that on the accuracy and go step by step. So that's the plan for the second block. So  I will in this session, I will also show you how we can model seasonality effects. And  we have a little formula that allows you to model seasonality at every point in the on  the planet surface. Then I'm going to talk a bit about machine learning, what are the  standard steps. Then I have a really nice example with the synthetic data about the  extrapolation over fatigue of machine learning. Many people start using machine learning without  being aware of all the, let's say, dangers of using machine learning. And then they sometimes  because it's very complex and it looks like a black box. Some I noticed some people make  a prediction or some results and even send a paper to publish. And then they discovered  that there was overfitting or there was some artifacts. And that can be quite a problem,  especially when the paper is already accepted for publication. So, so that's why we're going  to talk about it, how to prevent that. And then I will show you this patient interpolation,  I will show you the thing that I used to do in geostatistics using a creating fitting  viral grams, etc. And I will show you how does it look how you do that in with machine  learning. So something I specialize with and the last paper we published went really viral.  So it receives now over 100 citations a year. So they people got really interested in that.  And we kind of make a bit creating redundant because we have a more flexible method to  do special interpolation. So that's the problem for the second block. So that this is the data  set I was showing you. And this is a this cook farm, it's a four dimensional data set.  And as you see, when you look at many space time data, especially ecological data, you  will see the seasonality effect. And so that's very interesting. So it's very regular, let's  say, but differs a bit. Seasonality also, when you look at the seasonality, if you have a  winter in Europe, in Brazil, it's a summer, some southern Argentina, let's say it can  be summer, so it's completely opposite. So the climate is also even seasonality of climate  is very complex, you know, you have tropics, you have moderate zones, you have a southern  northern hemisphere. So also, even this is anality, it's complex. And so I'm super happy  we published this paper in, let me see 2014. And Milan actually is a first author, it was  part of his PhD. So he tried to tackle that problem of seasonality of daily temperatures.  At that stage, it was just we were just focusing on daily temperatures. And he came to a formula,  I don't know if you ever saw this, but imagine this temperature, daily temperature anywhere  on the planet, actually, it's, you can kind of model it based on the angles. So it's a  kind of sphere geometry. And this is the formula. So the two formulas. And if you put the coefficients  of these formulas, then below here, you can calculate minimum daily temperature and maximum  daily temperature. So you as an input, you only need the latitude, you need the latitude  and you need the day of the year. If you know just the latitude and day of the year, you  can model global temperature and that way you can also model the difference between  southern northern hemisphere tropics, etc. So I don't know if you ever saw this formula.  I'm still not sure if he derived the formula. First time, I think he saw pieces of the formula  and then he combined it together. And then he made it as a single thing. And you can  put it in, you can put it in R. And this is what it looks like in R. So this is a mathematical  notation. And this one is in R. And you can, you can also estimate your own A and B, A  and B parameters. So you can also estimate them. So I think that's kind of some type  of a maximum minimum temperature, a zero latitude or something. But you could put that  to estimate your own parameters. And then if I go to the R studio, so that's this modeling  seasonal components, you can see that we put that formula here. Here's the formula. And  as you see, I can go in, let me see. I can go and run that formula to some arbitrary  day. So let's say if it's a 52 degree, which is we are now 52 degrees, then I get for day  120, I get 8.7 degrees, the minimum daily temperature. But I could also put it somewhere  under 22 degrees. I don't know somewhere in Morocco or something. Let's see. I deleted  something wrongly. Looks at this visual editor. It's not working. So let's say 22. And if  I compute it, I get 23. You see? So it will change. And I can also play with the day of  the year. So I can just keep on changing these numbers. And you can play a bit with it. And  what happens, you get constantly, you have a different number. So that's basically when  you put it in this formula like this, that I take five years, right? I take five years  and I say, I want to look at the 52 degree in Oords and I want to look at all the days.  And then I will get this plot. And so that's the seasonality. And so this formula, by the  way, I can plug in and compute for pixel. So I can also make images because you see it  changes with the latitude and day of the year. And so I could do it for every pixel. And  then I create, I could basically map the temperature or geometric temperature for the Netherlands,  for every moment. So a very useful formula. And we discovered that you can actually use  it universally anywhere you have a day, a day or monthly values. So let's say for the land  cover for these predictions here, we also use this geometric temperature as the covariate.  So we also use it to model the land cover and the forest free species, etc. So we can  use it for forest free space. No, we didn't use it, but because it's annual annual doesn't  make sense to use, but we use it for the land cover because land cover we model, we can model  per season, we know the seasonal changes. So then you can use it as you basically add  it as a covariate. And it helps you, it helps machine learning because if you miss any other  parameter that especially for larger areas that basically represent the latitude correctly,  because if you use the latitude, it's not the same, right? Because latitude, it's like  a X dimension. And then if you go, for example, from a latitude, the 42 to 52, you move for  10 units. But actually that's not one to one to change us in the temperature. You know,  if you move from the 42 degrees to 52 degrees and from 52 to 62 degrees, then from the 52  to 62 degrees, the temperature drops much, much more rapidly. Okay, so you don't model  the latitude and correctly, but if you apply this formula, this trigonometric formulas,  then you can model really seasonality effects much more, I think realistically. And that's  why this formula is super important. And as you see, this is with Milan took the actual  data on the methodological station, this is super interesting. So what you see is, let  me increase this. So what you see here is the gray line is the actual measurements of  daily temperature at the station. So you can see the temperatures daily, they can differ  from day to day. I mean, you can have some rainy periods, cooler periods, warm periods.  So they differ like within a one week, you have a bit cooler, a bit warmer. And then  this one is the modest, the broken one. Modest is a bit overestimating temperature looks like.  So it's a bit above for these two points at least. And then the geometric temperature  is the very smooth line. That's the geometric temperature. And you see the geometric temperature,  the actual daily temperature reflects very well. And he was surprised that when he made  this model for geometric temperature, he took the global data set. And he said this geometric  temperature explains already 80% in variation in values. So you can already with the geometric  temperature, you already, you know, 80% of ellipse you can predict. So that's just a  nice proof of concept of that formula. But, but of course, there's a remaining variation  that you would like to improve. And then the final prediction is to do this linear model  and this linear model. So it uses the geometric temperature and modest, and it makes the prediction  in between. You see that's in between the modest and geometric. And that's kind of already  on sampling, you know, you basically have two predictions of the same thing. And you're  doing on sampling, and then you see that you come closer and closer to daily temperatures.  And that's what he did in this paper, basically, that's the that's the work. And so we really  like that. And it's a really, it's a very good response. And, and we started later on,  we started using that geometric formula, we can use it universally as a covariate for any  modeling where you have seasons or days. Yes, so you can use it universally, basically anywhere,  as long as I mean, if the error is very small, then the effect of latitude will be very small.  Then there's probably no need to use. But as soon as you move to an area, let's let's  say size of Netherlands, then it makes every sense to plug in that as one of the components,  and you just add it to machine learning. Okay, so I started with the geostatistics,  then about seven, eight years ago, maybe with less, I, I kind of started switching to using  machine learning for predictive modeling. And this is the general framework that we still use  basically for, for the machine learning for predictive mapping. It has a, we put about eight  steps and you can read about this in this article. But let's say the first steps will be to define  the area of interest and variables and consider all the inputs that you will need. And then you  base on that you make an implementation plan. And then the second step is to prepare the covariates.  That can take a lot of time. And also to prepare all the training points. And so that could take  a long time because you have to prepare earth observation data, etc. Then you could, if you  are starting, if you don't have the points, then you could do a new sampling, then we use a Latin  hypercube sampling. That's the usual we do because the, let's say a more suited sampling for a  multivariate regression problems, a classification problems. And then we will do the fieldwork,  you collect data. And then you do overlay. In the case of space time, we do space time overlay.  And sometimes you need to also make covariates, which have this, let's say some road network or  accessibility for the Latin hypercube sampling. And then you create this regression classification  matrix. And then from there on, it's just machine learning. So if you do everything properly,  then you have the machine learning. And then you can, you can do generate also maps, eventually.  And then you look at the accuracy and you look at the patterns and you say, well, we have a  problem, this still doesn't work still. Then you could go and collect maybe new points, extra points.  And then you remodel everything. If you call everything, if you do reproducible research,  every step is in code. Then eventually when you finish the coding, if you add more points,  then it's just a rerunning and repredicting. And that's a general framework we use for predictive  mapping and modeling. So more or less all our examples, everything also you see in this training  material, more or less all follows the same principle, except we don't have, often we don't  have an opportunity to go do sampling. So that's a pity. Sometimes the samples, they are basically,  they're defined how they find. So and sometimes they are purposeive sampling, and there's a bit  of clustering and there's a bias over representation of areas. And that's a problem. And there's some  remediation techniques for that in machine learning. But you cannot do sometimes you just cannot do  magic. Sometimes we just do to have to extrapolate and predict outside of the, let's say,  a feature space that we have for training. So that's the general approach. And then this is also  just in text, inside the machine learning. So just if you zoom in on this part, the,  so level seven. So if you zoom in on that part, so what you have here is the,  you do the fine tuning. So you can do fine tuning, you can do, you can do feature selection. So you  try to reduce model complexity. You can also do a learner selection. So you can just test the  learners that are most significant. So you can do many steps. And then it's also very good to do  kind of cross evaluation, cross validation. And then you estimate what is really your  uncertainty and what is the most, most important variables, etc. And then you generate predictions  and then off you go, you make a web GS and you get this thing. And then people can access it and use  it. Notice that most of the GS we make, it has also URL bookmarking system at the moment is without  authentication. So it's really purely open. But we might put some session ID. So at least we know  if you have a bug, you said, I had a session last week and I had these problems. So we'll put some  session ID. But if I take, if I just copy this URL, and I send it to some, yeah, if I open it in a  new browser, if I send it to colleagues, when they click on it, they will get exactly the same view.  And that way you can communicate about the same problems without even understanding  GIS and space time. I don't know. So that makes it very accessible to also to non experts.  So that's something I think we find increasingly important. Okay. When I started using our, I  noticed one of my favorite books at that time was the carer book, car package book. I don't know if  you ever use it, but it's very systematic. And I think it's the best way to understand also as  introduction to machine learning, all the many of the techniques here, they come really from  statistics, right? And this book, the beginning was about classification regression trees,  you know, modeling different type of regressions, which are like can be even just standard GLMs.  And I don't know. So this book, I really enjoyed that book. And I really, it was my,  let's say a mind changer for me. And I started really thinking about, you know, it's, it's very  important to do the cross validation, repeated cross validation, refitting. So, you know,  many times try to do from different angles, you know, and then learn from there. And that's how  machine learning works. Machine learning is a hybrid between computer science and statistics.  And so some techniques, you know, machine learning, actually, they're really just statistical  techniques, just combine into computer science techniques. And like three common computer science  technique in machine learning is, for example, sorting, you know, sorting, then, let's say,  finding a pattern, finding a combination that has a best match, best matching and things. And then  learning is, you know, making decisions based on repeated cases, making some decision based on  some, let's say threshold values, these are like computer science things. But then RMSE,  ordinary, really squares, fitting technique, you know, this is often statistics. Right. So then,  when you combine the two, you get machine learning. So please take a look at the title book, I  highly recommend it. So when you look at this framework, you know, we do what this framework,  this is purely spatial framework, as you see it. So how do you apply that framework to space time?  And there's an extra layer of complexity, how do you do Latin hypercube space time, you know,  it is possible to extend Latin hyper to space time, also how do you do machine learning on  space time data? So it looks like I showed the space time data is no complex than spatial data.  But actually extension, you will see many extensions to space time. They're not so,  they're not so complicated. So as long as you, you pick up the key things that,  that are kind of represented difference. So one of the, one of the things I showed you is the  seasonality. Yes. And I showed you that seasonality, you can represent it with the, for example,  geometric formula, and you can plug it in into a spatial data. So we can also estimate it for  every grid cell. Yes. And then in that way, we represent seasonality in our raster model,  raster stacks. Okay. So that's a very efficient way to then they represent that in the special  temporal data. Then the other thing is, when you do, for example, overlays spatial overlay,  which is relatively simple, let's say, when you do space time overlay, we made this extra  functionality that allows us it's called extract, extract ST. So we made, I made a little function,  which basically, it's a bit of complexity, but allows allows us to do a space time overlay.  So I still use basically, I use a spatial overlay, but I made this little function  that extract ST. So here's the function. So I made this little function that I subset the  points in space time. So if I have a raster, imagine I have a raster and this raster has a  begin and date. So it applies to some begin and date. Let me put it here. So if I have a time  series of rasters, yeah, if I want to do a space time machine learning, obviously,  if I have points, remember points like in a space time cube. So let's say this is the space time  cube. So the points can be different, different places in the space time cube. Yeah, so you have  the image and it's like multiple rasters. And then we have the points. And I want to overlay. So I  need to know for this rasters, I need to know to which time period they apply. So let's say more  these images, temperature images, there's a day. So they apply to the eighth day period. Eight day.  Yes, but my points were daily. Okay, so my points are daily and these ones are eighth day.  So I have to subset first all the points that fall into the time period. Then I get the subset  of points if I have a one million points or 1000 points, then I take the points that fall into  the eighth day period. Okay, and I have X, X, Y, T points from the original X, Y, T that fell.  So that's like a subset. So let's put it something like S. So this is a subset. And these points,  I can now overlay with this raster. And then it becomes just a spatial overlay, you understand?  So actually it's not so complicated, space time overlay, you just have to match, you have to  match points first in the time, then you match them in the space. So first the time,  and then you match in space. And then you do overlay. Yes, and then because it's the large data,  what this function doesn't do any parallelization, this function is just simply extract first subset.  Subset, subset here is the select, select is the subset. So I said, it has to be  less than the upper date and more than the lower date. Then it's subset the points. And then I do  extract, which is very fast. Yes, there are, there are package extract. So the just the points over  the raster. And then because it's a very computational to do it sequentially, then I make a function  that goes and does that. So let me see where was I showing that here. Sorry. Then I make a,  then I make it in parallel with the parallel package. There's a MCL apply. I make it in  parallel. So if I have a time series of 1000 images, then I will just run it as fast as possible.  And then my overlay will be as fast as possible. And you see, it takes literally to overlay  155 points going through one year data. So lots of space time points. I can overlay within seconds.  And that's the, that's the, that's what we do. That's, that's the stuff I kind of created over  the years. And now you can just go and use it. It will allow you that in all you can do as fast  as possible. I don't know if it can get faster. Trust me, I try many things. This is the fastest  with using the default packages for spatial analysis. This is as fast as you can get.  And you can do the space time overlay. And the only as an input, you only have to say,  you have to say what is the, for every thief, you have to say, what is the begin time of a  thief and the end time of a thief. That's the only thing you have to say is this. And then you put  it here, you say, this is the list of images. And then for each image you do overlay versus the  meta values. And this is the begin time of a thief and time of a thief. And then you do the  overlay. But we're going to show you that we're going to run that example in the last block.  And then you can, you can look at that. So, so this is the way it's another way to move from  space to space time. So we do a bit. So we are the seasonality. And then we do the space time  overlay. And then once you get the space time regression matrix, actually then machine learning,  it becomes the, actually it's the same or more or less the same, except you have to be aware it's a  space time, always, but more or less the same principle supplies. So you solve couple of  methodological problems, and you get to the machine learning regression classification  matrix. And from there it becomes more as a central mellow. Can you please close the door?  Then then the issue becomes that the data becomes very, you know, it becomes very large. That is  one of the issues. And that's what I told you. And you will see it in this, when we do here do  training. I mean, look, I have 155 stations, but it's a space time problem. Suddenly I have almost  45,000 values. And you say, okay, 45,000 nothing, but imagine when we had seven million space time  points, and we have seven million rows with 300 covariates. And that's a problem. I cannot open  that in laptop. You know, I cannot do machine learning on my laptop. Then you slowly have to  move from using just PCs, whatever to using high performance computing infrastructure. And we don't,  of course, any of our project, we don't compute anything on computers. We use all the servers,  right? But we will test, we will make the core development. We subset this space time regression  matrix, we subset 2%. And we do core development, you can do it on laptop. That's fine. And once  you're done with that, then you put the 100%, you put on some computing infrastructure. And computing  infrastructure, everything we use also answer, Buntu, actually, you have a GUI, you can, you log  into this computing infrastructure, it also looks like a GUI and except it's the servers that cost,  I don't know, 50,000 US. So, so you have to be careful what you do. But this is the space time  going from machine learning to space time. So I got really interested, of course, in machine  learning. One of the reasons automation, why automation? Well, you want to retire earlier  and have more free time, obviously. That's why we like automation. So we have to do, well,  you definitely will automation, you want to do less repetitive work yourself. Okay, everything  basically in science and in life, anything you can automate, you should automate doesn't,  I think repetitive processes, they are good, sometimes for meditation, they're good for  relaxation, I don't know. But they're not good for creative growth. So they are not good. So any work,  anything we go, like if you imagine today, people sit in the car and they drive to work and drive back,  you know, it's really bad. It's really bad for brain. I mean, unless you feel meditate in the car.  But let's say there's nothing to meditate, and there's no interesting radio station, you know,  it's a waste. So it's best that we use our brain only for the creative things. And so anything  you can automate, you should automate. So I really like that about automation. Then machine learning,  it's also, I think, sometimes we have been statistics and in the classical methods,  they have, they're very limited. So they're not so flexible. So sometimes I heard in sciences  that even people look for data sets that match their methods, right? There are people who say,  well, I have this method, I know how to use it, but it has limits. So they will only look for  the data sets that match their methods. And of course, that's a wrong way to go in science.  And science, the right way to do, of course, is to look at the complexity of world,  and then see, okay, how do I represent that complexity? So like if you look at T-R-F-K-R-O-S,  if you look at, I don't know, chaotic behavior of stock exchange or climate, and I don't know.  So you say, how do we model that? That's a much better way. And if you look at the many things  in the world, you need models, which, for example, can represent normal linear relationship,  different distributions, different types of covariates, many covariates, everything influence  everything. And that's where this models come, the neural nets, random forests. These models  have a capacity to have many variables overlap, the relationship can be complex. And actually,  when you look at just the random forests, actually, so generic, you can apply it to regression  problems, classification problems, survival, modeling, whatever. So it's also very generic in  sets of how you can apply to data. And that's what I got really interesting. The problem with  machine learning is computational complexity. So the computational complexity I had when I did some  linear statistics and geostatistics, as I said, it was an order of magnitude less,  even sometimes 1,000 times more computing. But computers, if you look at them,  you know, we are now somewhere 2021. And, you know, when they started, I don't know,  this was the first computer, IBM tabulator, I don't see the year, wait. So this is something 1930  or something. And then the first IBM computers 1950, and you know, the operations they could do.  So they were very, very basic. And now they say, yeah, you know, this Apollo,  the space mission, you know, that also computers that spend billions of dollars. And now you have  the same computing capacity on one iPhone, you know, the same computing capacity from 1960s.  So when you look at further down the road, 2045, we are now getting closer to the brain power  computing. I don't know how they estimate this computing. I don't know what is here on the  white line. But apparently by 2000, let's say, 30 or something, we will surpass the capacity  of brain computing in 2045. We will surpass the because it follows this more slow exponential  growth. By 2045, we have computers that can take it all people ever born at once. So and that's a  bit scary. And there's even this Ray Kurzweil wrote this, have you ever seen this? It's like a  six epoch. And so we are now somewhere in the between epoch four and five, where technology is basically  combined with biological life. And then when it gets to the high level, so you will see the  biological life will disappear slowly, and you will have just the technology. And then you will  have this highest level of intelligence, and it will have a capacity to spread around universe  and also terraform the universe, not by traveling, not people traveling in sci fi movies, but just by  electromagnetic light, it will perform the whole universe and turn it into a supercomputer. So that's  the six epoch and that's called universe lake wakes up. Anyway, read about it. But that's a  real extrapolation of more slow and many people don't buy it. Not every exponential growth is good.  The CO2 emissions, very sad to see, you know, it's very exponential plot also. The CO2 emissions  and population growth, population growth a bit slowing down, at least they think it's going to  slow down. But CO2, it's been now from, I don't know, this real meeting and Kyoto protocol as all  these meetings and we still admit CO2 exponentially more. Now the exponential growth started slowing  down, but we still meet every year more and more. So that's a sad thing to see. But me back in  2019, I was very excited about geostatistics. Okay, so when I started out creating and I was  learning I followed two courses. And then I published some work on maybe combining creating and  regression and applying it also to space time problem. So I slowly published. And then we  started doing bigger and bigger data sets. And then we switch I switched 2013. I first time I  switch I wanted to do geostatistics of Africa. And then I saw, I don't know, I'm going to do this  and points are clustered. So we just did the random forest and I saw random forest where I get the  better accuracy. So it looks like it's more suited for these type of problems. Then I did some global  data sets. And from about 2018, I stopped, I don't use so much geostatistics only use it for teaching.  The problem is in science is that my most cited work is in geostatistics. So when people see me  now, they say, I'm the regression cleaning guy. So that's what people tell me. You know,  in science, when you reach a level, people will publish people site you then you become the term,  you know, you're not your name or something. You know, you are the, you are this guy or that guy,  you become a term. So I was there, the regression king guy, and I said, look, I stopped using in  2018. So officially, I'm not, you know, it's like with the other and our people, they made this SP  package and that, but they would drop it a bit. So now you're the SP guy, no, I'm the SF guy, right.  And also Robert Heimers, he made the roster package. So you're the roster package guy. No, I'm the  terror package guy. So, so these things also change. But unfortunately, there's a bit of echo  delay. And so I'm definitely not using a random regression cleaning and stuff. So I'm much more  interested in, in machine learning. This is just to show you, like at that stage, you know, I looked  at the ordinary cleaning. And, and I made this paper called mapping efficiency information content.  And so what we saw that if you plug in this regression part, and you combine it with cleaning,  you get the map on the right. And then the uncertainty drops here, we kind of visualize  the answer to the very simple, we mask out the pixels that have the error larger than the variance.  So we kind of mask out this pixel because it means like you don't create a nice mythic  and prediction. And when you switch from this model to this, you can see that you can map more  pixels. So very simple. And you can see the patterns, you know, they're a bit changing,  it becomes like more and more detailed. So that was this paper. So that was that time I was really  excited about that. And then we started looking at this using a random for us to do interpolation.  So basically, not as a combination of geostatistics, but really just as a replacement.  So I just want to show it. So this is like this creating technique.  Do you use creating? Have you screenings? Yes. One, two, no, no. So so in creating, so you have  the points and you fit a variogram, a model of the spatial dependence, you need to decide about  transformation to understand your statistics, you cannot just like I'm just going to do  geostatistics. So you have to know what you're doing, but the many choices are new. So you have  to pick up the transformation of data, you have to pick how you're going to fit the variogram.  You want to do an isotropy, yes, no, tata, tata. So you have to pick that all the top and then you  fit a model. And then you get the predictions. And let me show you. And then the regression  creating is to add to that also regression model. But let me now show you how it looks like.  So when you do model based statistics, you know, you say it's a GLM,  and you have the transformation, you have the, you have to specify the co-variates you're going to use,  and you have to specify this link function. So you have to put all this up. This is this  model based geostatistics or geostatistics. And then machine learning, very different. You just say,  I want a quantum regression random forest. And off you go. Okay, so very, very different. And  there's a famous article by Leo Breiman called two cultures. And you should read that it's a really  classic. If you want to get into machine learning without leaving that article. And also, the other  articles that follow discussion articles, you will, you will, you will never get to machine  learning. So please read that. Leo Breiman two cultures. And basically, so what he says that  these two cultures just to summarize it. In both cultures, they have different obsessions, let's  say, and they see the aesthetics of things differently. So when you do model based  statistics, let's say, so like just when you think about Gaussian distribution,  Gaussian distribution, you represent it with how many coefficients. You have the mathematical  formula for Gaussian distribution, and then how many coefficients you have?  Do you know? So it has two coefficients, the mean and the standard deviation. So you have  mathematical formula, and you have two coefficients. And this is beautiful. You have something which is  using statistics all across, right? And universally many fields. And it's a represent you can  represent it with two coefficients. And the beauty of that is the simplicity to coefficients in your  model, different distributions, then you can have maybe a skew distribution, I don't know,  you can add a coefficient. But otherwise, the most simple case, we have two coefficients, and you  model the distribution of errors, the distribution of anything you measure. So it's really beautiful  when you think about this. So the beauty in the model based statistics in the figure out what is  the simplest possible way to present something universally. And that's in the case of for  Gaussian distribution, that formula that Friedrich Gauss made, and he says it's two coefficients.  I mean, maybe it could be done in one coefficient, I don't know, but two coefficients already,  it's very simple. When you do machine learning, it's a different direction. You don't mind that this  random forest, it can have 1000 parameters, you don't mind, it can have many parameters.  Because the beauty is the how good you learn, how good you match the data. That's why it's called  also data science. It's not the essence is to understand the models. You have very generic  algorithms that come from mixture of computer science statistics, you have very generic algorithms,  and then it's just how efficient are you in finding the structures and data. Yes, and that's  the second culture. You don't worry about the beauty of the models, you don't worry if there's  like, if it's Friedrich Gauss or if it's 1000, if it's two parameters, no problem. Yeah, but you  do way more computing. So these are these two cultures and I kind of switch to that. Yes, question.  Yes.  I don't think you can easily say this technique is data science and this technique is model based  statistics. Sometimes there's a transition. So there's some things that have elements of learning  and repetition and sorting and then some has elements of just, no, this is the model and we  just define the minimal complexity model and that's fine. So I don't know if there's a straight line  that I could say, I don't think so. So it's a bit of fuzzy term, you know, but in right, yeah,  super interesting. And in right, I think you can combine it in an ensemble system, you can put it  in a sample system, and then becomes one of the learners. And then you apply this data science  machine learning on top of it. And that's fine. So that's how I would do it. Right. So if it does  work, but I think this by using methods as far as understand, you need to know for every  observation, you need to also know some uncertainty or some prior, I don't know,  and that makes it complex because you don't have it from any data. So it is, it's a specific  method that you can assume that it's all the same or something, I don't know. But the real  effect if I understand that by easy and correctly, the real effect is if you know these priors, if  you have them like first estimate or if you have the worst uncertainty data, you know, then you  have the real effect because that's the thing you don't get so much good in machine learning.  Right. That's how I see it. But I haven't used in law. So I cannot, you know, give you a more  detailed answer. I haven't personally used in law. This is the geo package, just statistics.  So the same thing as I was showing here more or less, but this was just a regression. Now in  geo statistics, you specify the lambda, which is if it's a lambda zero, it means a low normal  variable. And then you have to say this range of you want to use to fit the diagram. And you have  to say which a variable model use. So you have to specify these things. And then you fit the  diagram and then you do the cringing. And then you have all the steps that goes fit the model,  do the box, cross transformation, then on the cringing, et cetera. And then why are you sometimes  you have to do on isotropy. So you consider also extra parameters of higher ground on isotropy.  And you can see that it is sometimes in different directions, the diagrams can be different,  like the red and the blue, they're quite different. So you sometimes you have to do all these things.  And so what I started looking at, okay, let me now look, what if I just, I just do like this,  it's a, I just take a buffer distances to points. I take the points and I take buffer  distance to all the points. So there's this formula. And then I run random forest on this.  So no, I don't have to know any geo statistics. I don't have to pass any parameters. There's  nothing is just fully totally automated. Nothing. I don't have to know model, I don't know modeling  assumptions, no distribution, nothing. Okay, I just say, here the points buffer distances.  And this is where they go to the beginning. This, the left one is the state of the art cutting edge  geo statistics. The one on the right is just random forest on distances. And I showed this  to a few geo statisticians and they're, wow, this is, wow, this looks quite similar.  Right. And first, what is similar as smoothness. So machine learning finds the smoothness in the  values, right? The values, they could be changing like, or the shorter distance or longer distances.  So it finds like, it's almost like recognize the, the, the, the, you know, and it matches it.  And then it's, it's in between the values, it's kind of like the values like interpolating.  And, and I repeated that for many datasets. And so again, you're also creating random forest.  Here you have even an, an isotropy. It's a one famous dataset. It's a very anisotropic. So you  have to model an isotropy in Varder. And then the machine learning find also this anisotropy.  You see, it's refined that the, the values change more in this direction than in this direction.  Okay. So, and, and you can estimate also the, the prediction error, which can be quite different  from a key. So get more detail. So I got very interested in that. So I said, wow, this,  with machine learning you, with the key, you have this stationarity requirements, you, you,  the distribution should be normal. Then you have to fit the Varder. Then you might have some hot  sports and they can be a problem and you have to do some, something in geostatistics. And with  machine learning, none of this actually don't even have to fit any variety of nothing. And  actually you can also extend it. You can add the more complex distances. You can add also  distances in the watershed. You can add distances to some sources of pollution or something. So  you can put everything together, which you cannot do in the geostatistics. So you just,  you can just throw it on a pile. It's unbelievable. And even when you look at this, even 155 points,  I calculate 155 buffers, this one co-radiates for one point. And for machine learning, it's no  problem. So technically, I can run it here in the code. Let me see.  So here's the buffer distances, right? Let me run this code. This takes a bit of a time. You see,  it takes, and it pulls up in a RAM, I think. It will pull up in RAM because I'm not running it  in parallel. No, actually, it is running in parallel. But it takes five seconds because I have to,  I create 155 maps. If I have 1000 points, I need to create 1000 maps. And let's plot one map.  So let's plot this one.  So this is a buffer distance from a point somewhere, I don't know, here, this point.  So I have 155 buffer distances. I mean, I can go to some other point, I don't know, 28.  And it's all different. So you're moving around, right? So these are all just buffer distances.  Yeah. Yes, here, no, nothing. That's random forest. As long as if it's non-linear or skewed,  it will try to get to the data. No, nothing. You just try it, and you get, often, you get a higher  accuracy. So here's the model. And it estimates that it explains 50% of variation based on out-of-bag,  which is already parallel with the estimate. And then you make a map. And so it looks like this.  So here's the, so it looks like basically like a crevice.  And I can plot it already in the, I did plot it in the, let me see.  I didn't check with the land or the, what's up with the, but I will just copy here.  So let me just copy all this stuff.  And we can open it here in Google Earth.  Let me drop it from the, so here's the prediction. I just plot it now in the Google Earth.  And we can see the points and the interpolated values. These are the,  basically observe the values. And this is the interpolated.  And so it looks like it's creating interpolation. Even this point, you see, it's a higher point.  It tries to come close to that point. So it's a really magical that without any  geostatistics, you can go and make predictions. I can play with this. It was very nice in  Google Earth, of course, to get this super high resolution. So if I set up a transparency,  you could actually, let's say if this was heavy metals, and you can see whether there's a town  or something. So you say, oh, wow, we have a really, people have to be careful here. Right.  So you can see where they're to last the town. Also, that's very nice. You see here, predict  water. Then you say, what's going on here? Why did you predict water? Well, we can look at the  time scale. Let's say here. So we can go back to the time and we can see that actually this  used to be very dynamic. So sometimes it floods and the water goes up and down. And sometimes the  since it's very dynamic, this area. And if I go maybe way back, when they did the sampling, maybe  then maybe it wasn't maybe it wasn't water body. So you can also scroll through. You see how the  water changes. So that's also good to important to be able to interpret the results and see how  do they match with the actual space. And you can you can play with looking at it 3D or space time.  So so that's that's this data set. So you never you said you never experience anything like that.  And yeah, we made a paper as I said, and this is this paper.  Where was it that paper? Or is it?  We made the paper and then we also made this tutorial. And then we kind of worked out, okay,  that was just a quantity variable. Now let's look at the categorical variable. Let's look at the  extreme values. Like look at the space time, multi-variate problems. And then I saw, wow,  this random forest, you can use it as a really generic case. So you can use it generic case for  a spatial interpolation. And it's horribly direct, you know, it's kind of demotivates you to learn  geostatistics, which I don't I don't recommend. I think I still I like the idea of fitting  variogram, you know, variography. It's very good because you can you fit something and you understand,  and you have a proof in the model that there's some spatial dependence or people what trains  etc. So I do still fit variograms. But I don't use it to make the predictions. I don't use it so much  this paper here. So that that's the paper we publish and it has a lot of attention and many  people now test similar approaches. And the other approach is even more brilliant, calling from  Belgrade. He made this approach with using distances to the neighbors. So you calculate  the distance to the second. So the first second, third up to 10th neighbor, and then also the  values of the neighbors. So and that even closer to geostatistics, but because you actually,  you use the values of the neighbors and the distance to neighbors, I just use no distances,  right? It's a bit primitive, let's say, but he uses also the values. He also got good results.  And it's also super simple and he applies it to space time also. Yes.  Yes, yes, very similar to Kriging, but it you fit a random forest model. I'll show you also how  this is done and you comply to your data. So at the same time, when I got interested in machine  learning, this world champion in Go, he was, have you watched that documentary? It's excellent,  one of the best documentaries I've ever saw. It's very tense, you know, there's the Google team  and this person, I don't know his name, but he was the world master and like Lee Seedle, Lee Seedle.  So and he said, I'm going to be a bit, beat you guys five zero. Yes. And you know what was the score  in the end? It was a four one for Google computer, the deep mind. And he was surprised, he couldn't  believe it. He couldn't believe he's losing you, but he won one game and he made some moves.  They call it the move for God. He had to make some moves that had never been made.  That's the only way you can beat computer. So you go really beyond and he made some moves and the  computer couldn't never had that because computer is, I mean, it's not, there's no intelligence  in computer yet. We don't have any intelligent computer, right? It's just if he has enough data  that those data mining and fine stuff, then it can, you know, implement that knowledge. But  this guy had to do completely new move and that's how he won one game. Then after that they played  some much more complex starcraft. And this one, they said no way computer can beat and also they  lost because this is way more complex, like the chess, you know, the like strategies and rules  that you could explain them on a book on 20 pages. When you play something like this, I mean,  the strategy is like 10,000 pages. Yeah. And even more, there's no end, I don't know. And they're  also, they also won. And then even these people made the software after this game that they won,  they published a nature of paper where they trained the machine to play with itself.  It learns by teaching itself. It doesn't even human knowledge anymore. It will generate,  it will simulate things and it teaches itself. So it plays the games that human, it didn't even  play ever. And it learns from that. It's really scary when you think about it. And so the things  are really taking, so we are somewhere here now. We are somewhere here. And so let's say when the  things cross, I don't know what's going to happen. But again, in that way, when you look at it,  it's most likely going to be some merger, I don't know. So this is all like really great, right?  It's fantastic when you think about this. Very simple. You plug in the data and off you go.  You don't have to think about anisotropy, modeling, parameters, distribution, nothing.  But it doesn't work always. So there's also cases when the machine learning doesn't work. And I  also, I burned my fingers. I did some maps where I did extrapolation, I created artifacts. And we  did some overfitting. And so I learned the hard way about also drawbacks of machine learning.  So number one drawback is the extrapolation problems. Then you have also bias in training data.  Data science means data is super important. As you see the metals are secondary. The data is  important. So if you have a blunders in data, you have a typo, you know, somebody puts instead of  zero comma, and you get like instead of five, you get 5000. And when you think it's a small error,  right? But on machine learning, it has huge impact. So you can really, just from few blunders, like  a 0.1% of data has blunders, they can mess up 80% of predictions. It's very sensitive to blunders.  You have to spend more time with the point data. So you spend instead of like in data science, you  spend in, sorry, in model based statistics, you spend a lot of time conceptualizing, picking up  the model, do I use this by easy and do I use this? That's what the learner, the analyst spend time.  With the machine learning, you have to spend a lot of time cleaning the data. So you do extra,  actually, data mining, and you look, is this an artifact or not? Is it, you know, checking the  data? So you have to spend much more time in the beginning and on the after the machine learning.  So in a model base, you know, you kind of in the center, you spend the most time the modeling.  But as I showed you here, you don't have to do any more than just show the data, you get the  values, but you spend more time in the beginning on the end. So checking the input data, and then  after doing expectation and seeing whether it matches the everything matches the input data  and relationships, etc. And so also I did the overfitting, overfitting also a big problem.  And then you get the bias. If you overfit to get the bias and, and also then in AI now,  they say, yeah, there's a problem in any machine learning you train with the bias data.  Yeah, this bias is going to propagate. And who is going to be to blame when you run the machine  learning and you publish the paper, but the biases in the training data, you are responsible.  Trust me, everybody call me responsible. They never said, there, it's fine. No worry,  it's the data. No, they say, what did you publish? It was your responsibility to check the bias.  Yes. And now also Facebook and all these companies that use this automated phase recognition,  whatever, you know, for job, you know, they had the model where they say they will predict where  the candidate will get the job or not. Did you hear about that? And the model just went, yeah,  okay, the first thing most important variable is the sex. If it's a female cut it down, just take  out the males. So you'll get arrested for discrimination, right? But the models was  fine. I mean, it was because that was the data, but the data comes from the past. And you apply  that model to the future. And that was a problem. When you do a special prediction, these are very  memorize this figure. When you have a clustering of the points, and you fit a model with the cluster  points, and then you slowly start removing for cross validation, you start moving the using  only the neighbors which are further and further away to check the models. Then they got the plot  like this. So if you take the cluster points, you fit a model, you get a 0.96 accuracy. And then  you slowly start testing a model. How does it work when I separate the training points from  validation points in space? Let's call a spatial cross validation. And they saw this that it actually  drops to 0.5. So it's a really big difference. Imagine here, 97, 52. It's a 50% overshooting.  But later on they saw, okay, this thing when you move points further and further, it's also  eventually you create a two pessimistic estimate of your model. It can be two pessimistic. You  force a model to be too strict. But let's say the truth is somewhere in between. So probably their  truth is that they have maybe 60% accuracy. If they would just randomly throw the points  in an area, an estimate accuracy, they will get maybe 60%. But they got 97. So that's why it's  also important to apply that. Now, this is a famous example. It's called, it's a little board that  person wrote, Peter Ellis. And it says a random for extrapolation stuff for the trees. Have you  ever seen this? So what you see here, this is a similar to data set point data set. And if you  fit a linear regression, you can predict out of the space, and you will predict the values are here.  If you do random forest, random forest, it has no structure basically assumes no structure. That's  why it's very nice. It's very flexible. But the door back is because it assumes no structure.  When you come to the edges of the training space, if it goes beyond the edges, it can only use the  last element. So it says, well, this is the last element. So it must be this. And basically,  it predicts all the weather from the last element. And this, these things people don't like. Of course,  because you see, look, this data has a structure. That's very clear structure. And the random forest  predicts this. So it loses that structure. And we feel like, Oh, this is over shooting. It means  anybody applying this model to decisions with extrapolation, it's making a mistake. Well,  most likely we don't know in this case, you know, it could be some models, they had this break, you  know, but let's say if you have 100 people and you won't tell them what which model would you use  100 people, 100 people will say I use the model on the left. Okay. So we can do that. We can play  with that this thing. So this is the extrapolation overfitting. So we can play with that. And I wrote  this little post in the medium. So let's see. So here's this post you can read about this. I  basically I don't make it as the code, but the more like in a human language, explain all the  things. And but we can also play with this in the the studio. And so we basically this, this is the  simulation of this model. And so what do I say, it's equal to x, x is y is equal to x plus the  noise. Very simple. That's basically I simulate a linear, perfect linear model with just the noise  component. Yes, and then when I go and I fit a linear model, I will get the R square of, let's  say 0.85. And you see the RMSE 11.8. You see this RMSE should match this one here. So it's almost  close to the 15. So it's approved that the linear model for linear structure, it will do exactly  what you expect. So it will estimate the correct RMSE. And it will say that yeah, 85% of the signal  is the noise. I could have increased this noise if I had a higher standard deviation,  then the noise gets noise component gets bigger and bigger. So that's a very simple way to reconstruct  that. And then we fit a random forest. Yeah, so imagine so now I'm teaching you random forest,  great, you can automate it, you can retire earlier, just show your data, and you come back and oh,  here, projects delivered. But look at this now, we get about the same variance explained. And  there is a package called random forest allows you to also estimate the prediction errors for  anywhere in the plot. So we do this random forest to fit it. We actually get the RMSE of the random  forest is a bit higher, 14. But it's about the same, it corresponds to the 15 simulated. And then  when we plot it, he's a nice plot. So you see the uncertainty of the linear model. So not necessarily  you see the linear model fit the red line. And now what happened and this is where the problem  starts happen. So you see the blue line is the random forest. So, so first thing about the blue  line, it, it, it also is up and down. Okay. And that that has that means that this random forest  is trying to feed the noise in a way, right, because it goes up and down like it tries to  feed the noise if I make another simulation, and I repeat this, it will the curve will go somewhere  else. So and it basically randomly goes up and down, because I know that's a simulated noise,  I know it's a noise. And even bigger problem is if you go to extrapolation space, you have this  last element, and then you apply this random forest error formula for uncertainty, and it  puts the uncertainty here, it's lower than uncertainty where we have the points, where I  have the points that are in the forest, that will get the uncertainty, let's say, relatively correct.  But as soon as I move outside, we have a problem. And this is a visual, visual explanation of the  two main problems of machine learning that will happen with you if you blindly throw the data  and just run random forest without thinking what am I doing here and do I have some clustering  problem, do I have some structure that I'm ignoring, etc. How much I'm extrapolate, etc.  And so that's a problem. And that's many people say they're really basically criticized, and they  really find it, you know, you shouldn't be doing that. And so what is the what is the solution  to that? How do we prevent this overfitting? How do we solve ourselves? So many things,  you can do many things. Number one thing is to basically to be aware that there's overfitting,  and then to study there's extrapolation, and then to try to see how much I extrapolating,  extrapolating, how much where will be the extrapolation and how many made this little  functionality to estimate so called area of applicability. And this allows you to see  which of the pixels were extrapolating. So first you should be aware how much is it like 10%  is it 50% you know, sometimes you most of the pixel extrapolating. So you have to be you have  to consider. Then the other thing is this book by Chris, Chris Molnard, fantastic book online, you  can also get a good book, maybe not Markdown, you can read all the diagnostics you can do to  analyze the models and to see whether there's strange things. And one of the classical things  you do is partial dependence plots. Yes, so you can take if you fit a model, you zoom in,  you do lots of diagnostics after you press the forest, it's okay, I got the model, I got numbers,  blah, blah, all automatic. But now I want to see what did I get? What do how does it look like so  you can plot this is the machine learning fitted regression. Yes, this line and you see the  uncertainty of machine learning. And you see where the extrapolation it grows. So that looks that  looks okay. But for example, here, I have this curve going down and it goes up. You see the  curve that goes down, and then it goes up. And this doesn't make any sense. I know from physics  that once I have, for example, by density, if it goes lower by density, then the carbon goes up,  it cannot go by density cannot go up again, if I have more carbon, it doesn't never happens.  Almost physically impossible because of just how you compress matter. So I know that this,  there is a problem here, this doesn't match any physical law. So I see there's a problem this  curve going up, it's a problem. How do I solve it? I don't know now, but at least I detected  there's a problem. And one way, you could say one way okay the easiest way if it's machine learning,  you can add more points, you can do more measurements where I have a higher organic  carbon. And then I add more points so that the models goes okay, and then adjust. Right,  that will be one way to feed it with more data. But maybe you could also, maybe you could limit  the prediction to the point where you, the model starts going strange. So you could say well I  only predict to this values. And I don't predict because I know the model doesn't do well. So these  are this remediation techniques. Universally for the board for the extrapolation and for the over  fitting, I found out that one mediation technique is just to do ensemble. Just ensemble, the ensemble  helps me prevent this overfitting. And I want to share that with this data set. I can show you  because now I switched to the MLR and MLR is a framework that I'll use in the afternoon for the  ensemble machine learning. And so here I can say, well, I'm not going to do only random forest,  I will also do GLM, I will do a gum boost, and I will do a kernel support vector machine. So I do  four techniques, which are very different, the very different techniques, they really different  ways the data is used and how the, you know, parameters are fitted. So I will use four techniques.  And I specify that as a task, a machine learning that's called a task. And then I specify that I  want to do a stack learner. Now we'll explain the afternoon what is a stack learner. So, and I  specify that I want to do a stacking by a cross validation with the refitting. And then I feed  that model. Now it's a bit, there's a bit more code. And it's also a bit more computational,  even like the small data I have, it computes maybe two seconds. And here I get this model and this  model shows that I have 0.85. And it shows okay, R square is a similar now to the linear model.  So, and now I can plot again that I can plot it. And do a prediction. So it's a way more code,  so definitely more code, more computing. And then I plot it. And let's say this is now  comparison between, if you remember the last model and the ensemble model. So here I go,  I do now machine learning, but I do ensemble. One of the models I use has a simple structure.  It's a linear model. And this model, of course, when I look at the result of machine learning,  so here's the summary. This one. If I look at this model, you see that this stacking,  stacking find out that the random forest actually, it's so badly, it's a so least important,  that we could remove it. If it doesn't have a one star, it has to have one, two, three stars,  that makes it significant. So support vector machines and random forest, I could remove.  Yes, but I don't remove it manually. I got it from the modeling. So I could do extra code to  remove all the learners that don't match. Then I run it. And then in this case, I didn't remove it  because when they have a small, so this beta, you see, beta is very small, 0.7. And the beta for the  GLM and for gumboost is 0.4. Sorry, four. So it's about 80 times bigger. Okay. It's about 80 times  bigger. So I don't even have to remove it. The beta is so small that it has no impact on prediction,  or minor impact. And then when I put it, we get this one. And now let's do some interpretation.  So are you more happy with this plot?  Why are you more happy?  So the uncertainty, this uncertainty, the estimate is really wide. It's really wide, right? Maybe  it's even too wide. But let's say in this case, because we want to use nonlinear methods, we  don't take out the random forest, we want to use it lonely there. And then we see that between  different methods, values go up and down. And then you say, well, the uncertainty is high. The  fitting of noise is much, much less. The blue line is almost close to the linear line. It does,  however, predict here a bit higher values than linear model, here a bit lower. So it does predict  values going this way. If I repeat this many times, every time I will get a bit different  curve, you can test it yourself. That's a nice thing to do as I exercise, change the seed value,  and just rerun it, change put another number, rerun it, and then you will see the curve changing.  You can play with that curve. And this way you learn also, you're basically doing machine learning  on yourself. So you learn about how these things change. But definitely sounds like this could be  a remedy for the extrapolation and for the overfitting. So it could be an ensemble in itself,  it's a good, let's say, remedy. Even better will be if I would add a function here to remove the  learners, which have a very small beta. Yes, I could just remove them because they otherwise  they increase the uncertainty. Yes. So I could remove random forest support vector machines.  Actually, I can do it now here. Let's see. So I basically rerun everything is the same.  Everything is the same. I just go in the where I define the learners.  I can put a new line.  Yeah. Maybe I'll have to restart my session. So this one, I go like this, and I mask out this one.  And here I have to now remove, let's say, fire removal ranger.  Things are not what you see. Not what you see, what you get.  And let's say I put here on the end.  Something like this. Okay, so I changed the, I changed the list of learners.  And if I then rerun the whole thing, let's see. I have to go step by step and rerun.  Ah, okay. And let's see.  So I made a typo. Ah, okay.  Okay.  I'm going to rerun this one.  And then I get the summary.  Let's see the summary now. It's still, I still have only two learners significant.  This end net, it's less significant. But let's see now the uncertainty.  Okay.  And here's the new plot. So now the uncertainty seats are much, much, much more narrow.  Do you see, but I don't use any random forest support vector machines. Do you see this? So uncertainties way more narrow.  Okay, so that's the, that's the effect of, if I would, for these data instead of just universal. So it's a universal space time function it will work with energy that are compatible data set time series.  Okay. But the force doesn't make sense to use it only for spatial problems. You have to have at least three, four times, you know, then you use it.  And also, by the way, do you need to space time. How many time repetitions do you need to do space time?  What do you think?  How many, how many repetitions certified. If I look at some feature, and it's changing through space time.  This is the time right.  This is the picture. And this is a space. Now I put it out of Iran and space I put only one dimension like X or Y the latitude.  So how many points I need to be able to do space time regression. How many points spread in time.  Repetitions or what do you think is it okay to have to have like something like this.  So space time regression.  So I have like just two years when there's observations or something.  Sorry.  Well, technically you could, you try to machine learning and you will get a model.  But think about this. Imagine, imagine our time is not time, but you want to model distribution between so based on the size of the shoe you want to predict somebody's weight.  And there is a longer foot you have the taller you are right.  So it is let's say close to linear. So if you have two points you could fit some linear model.  But you could make a lot of mistakes, very tricky to do with two points. Right. So, if you have three, three, let's say, three then linear structures fine but as the, let's say the structures are more complex.  And then if you don't, if you don't have enough time repetitions, it's really on the edge of doing space time. And I just wanted to be very careful because we did today with the machine learning you can just throw data.  You just throw the data and you get a model and people say wow great to go to model without thinking and the many people I see they hurt themselves with machine learning when I say hurt not physically hurt.  In a sense they make.  They, let's say overshoot with conclusions. They overshoot with confidence, all these things I show you with the simulator data set. It applies also to complex data sets or the underestimating the extrapolation arrow, overfitting the noise, everything applies and creating artifacts.  Yeah. And so this all this applies in synthetic data applies to the real data.  So that's why be careful. If you have two repetitions time, probably I wouldn't do space time, even though you can fit the model. So the questions can you do it. Yes, you can.  But the should you be doing it.  80% of cases I will say no, don't do it 90%.  Either we get extra points in time, or you just say I cannot do space time.  But maybe in some extreme cases you could allow. If you know that it's a linear structure. Maybe I would allow it.  We have the random forest algorithm use of all co artists select some of the build the random for some use all colors or some.  You can do a co bar selection is called feature selection. I didn't run it here but there is a code in MLR. Also, if you have 300 covariates, it will run a learning process to remove the covariates that are not significant.  It's a similar to a stepwise regression and all these methods. It can be also forward forward, looking or backward looking or forward backward.  You run it and often we reduce 50%. When you have a lot of co artists you can remove 50% often. Sometimes you will remove only 10%.  What measure of uncertainty this one that I was showing you in this plot.  The first one is the prediction error. So that's the error simulated predicting by doing a random sampling, simulating what will be the actual difference between predicted and model.  Sorry, predicted and observed.  So that's the different funds. There are different measures of uncertainty but most of when you do predictive mapping. Most of us are interested in the prediction error and prediction error is at the location.  So, for any new location you would like to know fairly pixel what is the prediction error.  If there's no more questions, we can stop with the YouTube.  I will stop with the live YouTube.  I have lots of in the chat also.  Okay, people complain about the video problems with installing the virtual box.  Okay, I will come back to your questions, but for now I have to stop.  Have to stop with the live sharing.  Stop live stream. Okay.  I have a lot of questions here if there are questions in the room, please.  My hope is that you are seeing a lot of new things. Is that correct. Yes. So that's a good thing so you seem a lot of new things what is mind blowing.  What was mind blowing for you to see.  Okay.  Yes, it's good to visually to remember that. And these are really school examples this plot.  This is like a school example everybody uses. But also the one I developed it for the but it's the same problem.  It's absolutely also overfitting you know I also was thinking that many techniques are kind of noise proof, but when you, when you fit a model looks like it's not noise proof because as you see it will.  It goes through the noise, and we know it's noise because I simulated it so I know it's not. No maybe there's a structure in data know it's noise and then of course cannot.  There's a noise proof.  Okay.  I like the concept of area of ability and it's a good to compute it to check, you know, to plot it to see.  But I actually think when you do ensemble machine learning, if you do things properly, the estimate of uncertainty is your area for ability.  So it's very simple where the answer to become so high. And this is in this article.  And here if you go all the way to the real data set I do also with the mass data set.  And as you see here I estimate this ensemble uncertainty. And as you see that's one point it starts growing so high that you could cut out.  And that's your ability you just use uncertainty if it's a good estimate. Let's say if it's a robust estimate of uncertainty.  And you can say well when the variance goes above the variance of the data. So maybe somewhere here I don't know. I would just cut out and say, don't do prediction here.  So I would just mask out the pixels.  And then it's a more universal way to do it because therefore, the ability, you know you have to consider that it's a proxy. And this is not a proxy this is a model estimate.  So maybe it's easier to compute because this one it's more computational, but this one you can call it then you can automate.  This one it's in the, it's in the slides.  I think I put it in the slides otherwise fine I will put it here in the, in the metamask.  I put it.  So,  that's the position.  Okay, so you can.  So it applies to anything. It's called feature space, but in future space you see I can use distances.  So it applies to anything.  Yes, in the, in the when I use the MLR.  I will hear I will set it up.  And when I do ensemble. So I specify here.  So, I say it's tech CV. So the learner. So it has a two process I'll speak about it in the after that, but it has two steps.  And so it's actually it's a lot of testing. There's a lot of model fitted behind that you don't see.  And, and so when you do hear the, the splitting, it's done by repeated refitting the stack CV it means repeated refitting five fold, but  yes, the way you decrease the overfitting is by doing this stack CV, because you repeated it many times, and then you know. Yes, so then, because with the stacked stacked ensemble stack learner, you have two modeling.  So you have these divisional models, and then you pay the many times, and then you have this so called metal learner that learns from this individual models but learns to cross validation of these models.  So if one model over fits in the second round, you will see the overfitting and then this values goes down.  That's why the random forest the value drops.  It's become irrelevant you could remove it. It's insignificant.  That's why that's why I find it is a remedy on some machine in itself is a remedy to prevent overfitting.  Yes.  Yes, and then also remedy for the extrapolation to get the correct estimate of the uncertainty and to also not to have the line going like this but to straighten the line.  This kind of whether this example should know space.  This is just a model. No space but if you extend it to spatial problems. So like here.  Here there is a space in world but I don't use the space in the modeling like I'm the example I showed you this is just distance to the river. So actually it is kind of geography.  So here I apply to the real case and now I use it on a actual spatial data. Right but it's the same the code is the same.  Many things when you program as a third you in machine learning I have I can do just machine learning, right, and they do machine learning on images photographs whatever.  But now I say I want to apply machine to spare spatial problems then I look at coordinates of the distances you know, and then I say oh man I want to space time, then I have this is a knowledge the effect and I have the, you know I have to have a time series of remote sensing  data I have to fill in the gaps you know, but eventually when you do everything correctly you finish with the regression classification matrix and there's no change there from machine learning is the same.  So it's the same but the preparation of the data is different. And so when you get to prepare it, you should do everything properly. Then it's the same it's just a machine learning problem. You just have to learn the model line avoid overfitting and remedy extrapolation.  You know, so it applies with a spatial space time or non spatial. It doesn't matter.
A1_0_251.png A1_252_1043.png A1_1044_6875.png A1_6876_8423.png A1_8424_8531.png A1_8532_8591.png A1_8592_9263.png A1_9264_9371.png A1_9372_9515.png A1_9516_11327.png A1_11328_12779.png A1_12780_14963.png A1_14964_22355.png A1_22356_22415.png A1_22416_22511.png A1_22512_22559.png A1_22560_23327.png A1_23328_23519.png A1_23520_23675.png A1_23676_24635.png A1_24636_25475.png A1_25476_25559.png A1_25560_25979.png A1_25980_26039.png A1_26040_29123.png A1_29124_31343.png A1_31344_31451.png A1_31452_31715.png A1_31716_32015.png A1_32016_37235.png A1_37236_40103.png A1_40104_50495.png A1_50496_50903.png A1_50904_55391.png A1_55392_56171.png A1_56172_62015.png A1_62016_62231.png A1_62232_65051.png A1_65052_65231.png A1_65232_65327.png A1_65328_66083.png A1_66084_66731.png A1_66732_71243.png A1_71244_73943.png A1_73944_74219.png A1_74220_74375.png A1_74376_75299.png A1_75300_75419.png A1_75420_75599.png A1_75600_75731.png A1_75732_76235.png A1_76236_76523.png A1_76524_76775.png A1_76776_77111.png A1_77112_77159.png A1_77160_77375.png A1_77376_77435.png A1_77436_77747.png A1_77748_78071.png A1_78072_78611.png A1_78612_78731.png A1_78732_78803.png A1_78804_78863.png A1_78864_78911.png A1_78912_79043.png A1_79044_79511.png A1_79512_80063.png A1_80064_85499.png A1_85500_94271.png A1_94272_96683.png A1_96684_96839.png A1_96840_97151.png A1_97152_97715.png A1_97716_97787.png A1_97788_98027.png A1_98028_98759.png A1_98760_99695.png A1_99696_102263.png A1_102264_102347.png A1_102348_102587.png A1_102588_103775.png A1_103776_107075.png A1_107076_108563.png A1_108564_108779.png A1_108780_109559.png A1_109560_109907.png A1_109908_110063.png A1_110064_110591.png A1_110592_111527.png A1_111528_112091.png A1_112092_112211.png A1_112212_112259.png A1_112260_112751.png A1_112752_115031.png A1_115032_115295.png A1_115296_117647.png A1_117648_118331.png A1_118332_118403.png A1_118404_118679.png A1_118680_118919.png A1_118920_119135.png A1_119136_119231.png A1_119232_119759.png A1_119760_129419.png A1_129420_129983.png A1_129984_131711.png A1_131712_131795.png A1_131796_132659.png A1_132660_133631.png A1_133632_134567.png A1_134568_136019.png A1_136020_136079.png A1_136080_138587.png A1_138588_138803.png A1_138804_138935.png A1_138936_139763.png A1_139764_140366.png