Big Earth Observation- and Geodata analysis with actinia and GRASS GIS
CC Attribution - NonCommercial - NoDerivatives 4.0 International:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Adams, Till
We present a cloudbased technology, that allows “big geodata analysis” in process chains, that make use of actinia and GRASS GIS. Geo- and EO-data are more and more available, especially the growing Open Data policy of organisations fuels this process. With this awareness and demand for geo-based decision support increases. We want to present an implementation of the new paradigm of “bring processes to the data”. In the talk we present how big geodata analysis process chains could be easily implemented in a cloud environment by using GRASS GIS and actinia (OSGeo project & OSGeo Community project, respectively) for the process algorithms. The combination of these allows cloud optimized geodata processing triggered through simple API-calls. In the first part we show the architectural design and the interactions between actinia and GRASS GIS. In the 2nd part we present examples of process chains. These are a) automated surface-type detection from orthophotos, b) on-demand creation of cloud free Sentinel-2 scenes using temporal interpolation, and lastly c) gap-filling of high-resolution land surface temperature data from MODIS-satellite data.
 Hi everybody, I'm talking from Bonn. Some of you might remember that conference over here, 2016. I think a lot of people have been to Bonn that time. Yeah, my name is Till Adams. I was, I had the honor to chair the Global Phosphor G in Bonn. I'm also quite active in OSGU since few years. Since last year, I'm a member of the OSGU board, which is great fun, I must say. So, and beside all the time I spent in community building, community stuff, I also have some regular work and I work mainly as a consultant and agile coach. For two companies, I founded the company of Terrestris in 2002 and in 2015, we founded a second company, it's called Mondialis and that was me, my partner from Terrestris and also Marcus Nittola. I think quite a few of you might know Marcus, he's founding member of OSGU and I think widely known as Mr. Grassjie. So, that's more or less the background where I come from. Today I talk for the company of Mondialis and yeah, first of all, we started, when we started Mondialis, we had something like an idea. I think one of the reasons why we came to the idea, a conclusion to found a second company and not to put that topic into Terrestris as well, was the earth observation data and the release of the Sentinel data for free from the European community and since then, I think a lot of organizations, even in Germany, which are quite conservative on that part, released their data as open source. So, there are tons of data on the one side, there's stored on cloud storages and whatever, but you always have the problem if you want to work with this data, if you want to extract information out of this data, you have to download the data to your desktop and that was our idea. What if you could bring all these fantastic grassjie as modules, processing chains and whatever to the tons of data which is lying around in the cloud. So, that was one of the original ideas we had that time and yeah, we thought a little bit about that and said, okay, if it would be possible, if you could use grassjie as functions, wire and HTTP REST API call and so bring the algorithms to the data, then we could do something that we originally called grass, which is the acronym for grassjie as a service. So, that turned out later to the name Actinia, which is part of the title of this talk and the underlying idea was of course because grassjie as is open source. So, we have no license restriction in just deploying grassjie as anywhere in the cloud. We could even deploy 20, 30, 40 cores and didn't even have to care about or think about any license restrictions, which is I think really an advantage we have in the open source world that we just don't have to think about that. So, these were the initial ideas we had and yeah, before I go in a little bit deeper in that, normally if I would stay on stage now, I would please you to raise your arms, whether you already worked with grassjie as or not, but for everybody to follow that talk, I think it's quite important to have two slides about some grassjie as basic so that you get an idea how grassjie as in general things. So, if you installed grassjie as there is something called the grass database or grassjie as database, where all the data you're working on is put into a pack together and you can imagine it a little bit like a file browser. So, you have below the grassjie database, you have locations, which are kind of folders with the same EPSG code, the same area you're working on. And below the locations you have different mapsets and in your mapsets you have what we normally call layers. So, you have vector tiles, raster tiles, the evaluation images, which are of course mainly raster tiles. You have spatial temporally data, whatever you want to have. So, that's the structure how grass is handling its data and when you work in grassjie as you work always on a map set. The second thing which is important to know is that grassjie as consists of hundreds of modules, grass modules. Grass module is anything that does anything, it could display something. If you look at the slide you see something called u.clip, which as the name says helps you to clip out a small extent of your vector data. That's the u standing for. If you look at the bottom of the slide, there's a module called airslope.aspect, which helps you to calculate slopes and aspects based on digital elevation models. There is an R before, so that applies to raster data and stuff like that. So, that's the way how grassjie as things and what you do when you work with grassjie as you have a either you have a command line tool and then you just call the commands and give some parameters into that input data and name the output data, give in some more parameters, whatever. And the second really cool thing you can do with grassjie as you can chain these modules. So, you can for instance write a shell script or something like that. And in the shell script you can chain different modules so that the output of the first module gets the input of the second module and so on and so on. So, that's kind of process chains you have in grassjie as. So, just a number I think in the moment there are around 400 grass modules, smaller ones, bigger ones, stuff like that. So, this little bit background about grassjie as and then we had some thoughts about what we might have to do in order to port grassjie as into the web. And the first thing of course we must enable the user to list all the locations, the mapsets and the data as resources. Of course we would have to enable the usage of all the grassjie as modules via an HTTP call so that you directly could access remote installed grassjie as via and rest API. But we have to do something more. If you think about the cloud, every calculation process in the cloud costs you money in the end because every CPU that is running, every storage you use costs money per time. So, we need something like a user administration that administers groups and roles and that enables you to restrict users to use only a certain amount of pixels or a number of processes, number of nodes, whatever. The second thing we had to do is to think about the grassjie as database management and especially we had to think about that we must lock mapsets. If you have grass on your desktop nobody cares because it's just your data you're working on but if grass is on the clouds and everybody could access it you have to lock at least the mapsets in order to mind the conflicts that might appear on that part. And yeah the last thing which is really really important because it lets you really use the power of cloud systems is to make grassjie as scalable. And this could be in two made in two kinds. The one thing is that you enable grassjie as to split up huge processes for instance spatially so that you have a lot of workers that all work on smaller parts of your job especially usable if you work with big earth up the wation data and then at the end put all the data back together into one file and give back the result to the user. Or the other thing is just to say okay we want to have several nodes in a cloud environment but somehow the nodes must work together in a certain way. I could explain that better with the next slide. So if you look at that slide you see on the right-handed side you see some grassjie as actinia nodes which are deployed in the clouds and on the left-hand side you have somebody calling process chains or grass modules or whatever. And if you think that user A sends a request to the grass and says okay I want would like to run this module on this data then it gets back a number an id and the actinia nodes directly writes this process into the redis database you see on the right-handed side of the slide. So and then the process runs and if you have big processes it might take hours to finish them so there's a mechanism you can always pull the actinia node and ask okay what's up with my process it's up with my process and then it reports back yeah it's still running it's still running and then it's finished get your data here whatever you want to have but if you pull the whole installation it might be that you won't hit the node which actually is really processing your process node but every node is capable to look into the redis database and get out the information and give that back to you. So that is how it works together scalable on several nodes and another very important part is if you want to deploy the software in a cloud environment it's very clever nowadays to pack them into into docker files and that is what we did so that you have a really easy deploy mechanism to start up workers in the cloud to shut them down delete them in order to save money on the cloud environment so we packed that into into docker files use docker swarm with docker compose to release them or other frameworks we listed up below like open shift or kubernetes or terraform so that helps you really for a fast deployment just out of your github of the the whole thing in a in a in a cloud environment. The next slide is a little bit more practical so you directly could see what how you interact with an actinia so you send a request like htdpi myurl to actinia slash locations and then you get you receive back a jason file which then lists up the locations so you can see here it's a location one is nc spm08 and the second one is ecad that's the name of these locations you always get back also the status of your request that one was successful and then you can go deeper into that and then you add to your requests after locations you add the name of the location and then you write for instance mapsets and then you get back a list on like on the right-handed side a jason file which tells you about the the mapsets which are in that in that location a very cool thing and that's what we really use in our daily work in our project are process chains i told you before that in grass gs you're capable to chain several grass gs modules for instance by a shell script or something like that what you also could do on the actinia side is to roll your own process chain this is just writing a jason file like that one which is a short one we have some that have a few hundred lines of code so you can chain different grass modules together into one big process which is especially handy if you really work on earth evaluation data so you can do for instance the first one is to to access a central two scene the second one is to do a classification on that the third process in your chain would be a calculate n dvi or something like that and and these process chains are pushed through post via the actinia api and then actinia will will talk to grass gs and yeah let your process chain run so that's really really interesting and and good functionality you have here so if you're more interested about this theoretically stuff i'm not going over to to that website here there is a website actinia dot mondialis dot de there is a ap documentation of everything you can do with actinia or the the modules crash modules how we talk to them some documentation there's some examples so if i think it's much more better explained there than i could do here just in a few minutes so keep that keep that website in in mind five minutes okay um yeah sorry elastor i need another 25 for that slide to explain that but now it's just to give you a short overview about what we did in the core we have this actinia grass gs we have several interfaces like the rest api i'm i'm talking about there is also an open eo api we always think about a wps api we have some plugins that directly let you connect to several kinds of data like sentinel data so that you don't have to import the data into your grass database you just can mount it on the cloud system where it is we have we are working on a connection to geoserver we have a connection to a metadata node in order to get out data and integrate it in processes so a lot of things we we did all around this actinia and actinia core in the in the past year the good thing is for you actinia is free and open source software you find it on github since march 2019 actinia is also recognized as an osgeo community project so we are very proud about that and yes and i want to finish with some examples i shortened it already so i have only two of them and not three as mentioned in the description of the call and the first one is a huge project which is run by a big telecommitation company over here in germany you know if you look at the fiber cable connections in europe if you look at the table some countries like estonia they are near to championship and germany is close to league two so there's a lot of work to do on that and they are setting up a huge mainly open source gs based planning tool and one of the core functionalities about that is that we deliver a roughly planned pre-planned fiber cable network for certain areas and you must know that the cable installation costs depend really heavily on the surface type because if you dig into green land it's it's much more cheaper than you have to dig into roads or pavements or whatever but especially this data is not available in in germany so what did we do we put a lot of data together like aerial image, autofoto, slider data implemented in actinian process that let us automate to create training areas and then we do object-based supervised classification using machine learning where we put all this data in the training data in and what comes out is something like that this is a short piece of a village very close to to bonn here and if it's this autofoto went through our process you get something out like like that one where you can clearly see okay we have buildings we have green land we have woodland we have the roads and that is for that process the base to to calculate the rough network so this is all done via actinia which is integrated in a huge huge system on on that. The second example is more about to show which amounts of data is possible to process with with actinia this is a work that mainly marcus is working on for more than 10 years there's an us american satellite called modis which is a sender that detects surface temperature land surface temperature and the problem is if there are clouds like today here in bonn you you don't get any data and the task was to interpolate these holes and to deliver a scene with completely data on it like you can see on the on the right-handed side and this is really interesting if you look at this modis lst data because they already cover 18 years so the the satellite is crossing the earth for 18 years now and delivers four corriges per day four not five four with a five 250 meter resolution so if you put all that together you have about 26,280 data sets covering the whole world i don't want to i even can't in english name the number of pixels i just wrote it below but it's it's quite a lot of data and and that's if you look at that if you would calculate the data on a on a one cpu 400 megabyte ram machine and you say okay it takes nine minutes per data set then it would take you 164 days and six hours to calculate all this data and remember every day four new scenes come in so you probably never get finished but if you able to scale that up and do a parallelization of that calculation and processing in for example two eight core engine you can you can lower that down to much less much less time so yeah that's yeah just the end of the story if you're interested in that data set i put the link to to a metadata set describing the land surface temperature data on the slides i'm quite sure that we're going to share the slides afterwards and yeah that's that's it that's a short story about uh actinia grass jess uh for now thank you brilliant thank you very much tell those really really interesting um i've got a host of questions here um first off i think we'll start with two very similar questions so from uh usif who asks what kind of machine learning algorithms uh can we have using grass and actinia and also linked to that is from uh david who says uh are the machine learning algorithms you used for surface types available in an open format too um i start with the second question i would say yes look at the look at the github page if not drop me a question and um the first question that's really cool for me because that's the reason why i put on the slide that i'm more of working as a consultant actually i have no idea which which algorithm actually i used i have to talk to the developers but also you can send me just an email asking that and i could deliver that afterwards after the talk because i'm i'm not the person who really puts his fingers into that i'm more the presenter okay cool um and then from calid we've got do you do you think we can process large raster geospatial data on the fly instead of firing pipelines for back end processes that might take hours especially for UAV datasets there can be tens of gigabytes for a single file i would say yes i think it's it's more or less uh the idea how you scale that how you split up the process so it's not real time but near real time could be possible but of course depends on the data depends on data format depends on the amount of data and stuff like that but i would give it a try because the engine is is ready for that and again if if you need some advice here just come back to me and i will connect you to the appropriate person in the in the company that's what i'd like the type of uh sort of gun her out of you i'll give it a try and see what happens then report next for sfuggi yes exactly um we've got one from james here and it sort of links into something i was going to ask you as well so james asks are you involved in the ogc api process in standard development and i was going to just ask how what you're developing with actinia might sit in with say some of the wps stuff that exists already yes so we are not involved in that we are involved in the open eo project which is to mate it's still it's already running for two years i think which has the idea of setting up a unique standard for processing api stuff like that but not in the ogc stuff and the wps um if you if you looked at the slides maybe i go back to that um it was uh it was a red box oh i was too fast sorry that one so um we always thought about um implementing a wps api on that we have a actually we run a pro project with mondialis and terrestris where we connect the actinia stuff to a web interface which mainly is stuff that terrestris does and there we had huge discussions about whether implementing wps api or just using the rest api it ended up that we for now use the rest api so we didn't really implement the wps um api yet but i think maybe it's not worth doing that as all the ogc apis in a uh change in the moment to a new version so maybe it's better to to keep that on hold for for a couple of months and then look how it turns out with the new wps replacing api okay cool um let me just quickly get so andy is asking about the use of analysis ready data in these tools um so i don't know whether that's whether or not they can be used or whether or not um you can process it up to to analysis ready level you could do both of course you can do your process chains and bring your data in the shape you need it or if you directly have the data um if you look at that slide already i've mentioned that as well we we created some some plugins especially sentinel plugin which enables you directly to make use of cloud's stored sentinel data without having to import it into the grass database which would be an additional step and i could imagine of something like a plugin that directly connects to to your data into that it's all written in in python by the way okay cool um so i think as we're coming towards the end of the session i just i've got one question and i don't know whether you'll be whether you'll know the answer to this one but it was you mentioned at the beginning that grass has 400 modules i think you said um do you have any information about um the proportion of use for each of those different ones because i'm guessing some are used by all the grass users all the time are there are there other ones that you sort of think or maybe we could just put put out to grass hey i i think it's impossible to to figure that out because normally people use grass gs uh just on the desktop so we have no idea maybe it's you get a clue about that if you go into the github and see which modules are updated and with many developer stuff like that i could ask marcus he's not actually he's not in the office but i think he should know better a little bit but even marcus i'm sure does not know every of these hundreds of modules in his brain
 Hi everybody, I'm talking from Bonn. Some of you might remember that conference over  here, 2016. I think a lot of people have been to Bonn that time. Yeah, my name is Till Adams.  I was, I had the honor to chair the Global Phosphor G in Bonn. I'm also quite active  in OSGU since few years. Since last year, I'm a member of the OSGU board, which is great fun,  I must say. So, and beside all the time I spent in community building, community stuff, I also  have some regular work and I work mainly as a consultant and agile coach. For two companies,  I founded the company of Terrestris in 2002 and in 2015, we founded a second company,  it's called Mondialis and that was me, my partner from Terrestris and also Marcus Nittola. I think  quite a few of you might know Marcus, he's founding member of OSGU and I think widely known  as Mr. Grassjie. So, that's more or less the background where I come from. Today I talk for  the company of Mondialis and yeah, first of all, we started, when we started Mondialis,  we had something like an idea. I think one of the reasons why we came to the idea,  a conclusion to found a second company and not to put that topic into Terrestris as well, was  the earth observation data and the release of the Sentinel data for free from the European community  and since then, I think a lot of organizations, even in Germany, which are quite conservative on  that part, released their data as open source. So, there are tons of data on the one side,  there's stored on cloud storages and whatever, but you always have the problem if you want to work  with this data, if you want to extract information out of this data, you have to download the data  to your desktop and that was our idea. What if you could bring all these fantastic grassjie as  modules, processing chains and whatever to the tons of data which is lying around in the cloud.  So, that was one of the original ideas we had that time and yeah, we thought a little bit about that  and said, okay, if it would be possible, if you could use grassjie as functions, wire and HTTP  REST API call and so bring the algorithms to the data, then we could do something that we  originally called grass, which is the acronym for grassjie as a service. So, that turned out later  to the name Actinia, which is part of the title of this talk and the underlying idea was of course  because grassjie as is open source. So, we have no license restriction in just deploying grassjie  as anywhere in the cloud. We could even deploy 20, 30, 40 cores and didn't even have to care about  or think about any license restrictions, which is I think really an advantage we have in the  open source world that we just don't have to think about that. So, these were the initial ideas we  had and yeah, before I go in a little bit deeper in that, normally if I would stay on stage now, I  would please you to raise your arms, whether you already worked with grassjie as or not,  but for everybody to follow that talk, I think it's quite important to have two slides about  some grassjie as basic so that you get an idea how grassjie as in general things. So, if you  installed grassjie as there is something called the grass database or grassjie as database,  where all the data you're working on is put into a pack together and you can imagine it a little  bit like a file browser. So, you have below the grassjie database, you have locations,  which are kind of folders with the same EPSG code, the same area you're working on.  And below the locations you have different mapsets and in your mapsets you have what we normally call  layers. So, you have vector tiles, raster tiles, the evaluation images, which are of course mainly  raster tiles. You have spatial temporally data, whatever you want to have. So, that's the structure  how grass is handling its data and when you work in grassjie as you work always on a map set.  The second thing which is important to know is that grassjie as consists of hundreds of modules,  grass modules. Grass module is anything that does anything, it could display something. If you look  at the slide you see something called u.clip, which as the name says helps you to clip out a small  extent of your vector data. That's the u standing for. If you look at the bottom of the slide,  there's a module called airslope.aspect, which helps you to calculate slopes and aspects based  on digital elevation models. There is an R before, so that applies to raster data and stuff like that.  So, that's the way how grassjie as things and what you do when you work with grassjie as you have a  either you have a command line tool and then you just call the commands and give some parameters  into that input data and name the output data, give in some more parameters, whatever. And the  second really cool thing you can do with grassjie as you can chain these modules. So, you can for  instance write a shell script or something like that. And in the shell script you can chain different  modules so that the output of the first module gets the input of the second module and so on and so  on. So, that's kind of process chains you have in grassjie as. So, just a number I think in the  moment there are around 400 grass modules, smaller ones, bigger ones, stuff like that. So, this  little bit background about grassjie as and then we had some thoughts about what we might have to  do in order to port grassjie as into the web. And the first thing of course we must enable the user  to list all the locations, the mapsets and the data as resources. Of course we would have to enable  the usage of all the grassjie as modules via an HTTP call so that you directly could access  remote installed grassjie as via and rest API. But we have to do something more. If you think about  the cloud, every calculation process in the cloud costs you money in the end because every CPU that  is running, every storage you use costs money per time. So, we need something like a user  administration that administers groups and roles and that enables you to restrict users to use  only a certain amount of pixels or a number of processes, number of nodes, whatever.  The second thing we had to do is to think about the grassjie as database management and especially  we had to think about that we must lock mapsets. If you have grass on your desktop nobody cares  because it's just your data you're working on but if grass is on the clouds and everybody could  access it you have to lock at least the mapsets in order to mind the conflicts that might appear  on that part. And yeah the last thing which is really really important because it lets you  really use the power of cloud systems is to make grassjie as scalable. And this could be in  two made in two kinds. The one thing is that you enable grassjie as to split up huge processes  for instance spatially so that you have a lot of workers that all work on smaller parts of your  job especially usable if you work with big earth up the wation data and then at the end put all  the data back together into one file and give back the result to the user. Or the other thing is  just to say okay we want to have several nodes in a cloud environment but somehow the nodes must  work together in a certain way. I could explain that better with the next slide. So if you look  at that slide you see on the right-handed side you see some grassjie as actinia nodes which are  deployed in the clouds and on the left-hand side you have somebody calling process chains or grass  modules or whatever. And if you think that user A sends a request to the grass and says okay I want  would like to run this module on this data then it gets back a number an id and the actinia nodes  directly writes this process into the redis database you see on the right-handed side of the slide.  So and then the process runs and if you have big processes it might take hours to finish them  so there's a mechanism you can always pull the actinia node and ask okay what's up with my  process it's up with my process and then it reports back yeah it's still running it's still running  and then it's finished get your data here whatever you want to have but if you pull  the whole installation it might be that you won't hit the node which actually is really processing  your process node but every node is capable to look into the redis database and get out the  information and give that back to you. So that is how it works together scalable on several nodes  and another very important part is if you want to deploy the software in a cloud environment  it's very clever nowadays to pack them into into docker files and that is what we did so that you  have a really easy deploy mechanism to start up workers in the cloud to shut them down delete them  in order to save money on the cloud environment so we packed that into into docker files use docker  swarm with docker compose to release them or other frameworks we listed up below like open  shift or kubernetes or terraform so that helps you really for a fast deployment just out of your  github of the the whole thing in a in a in a cloud environment.  The next slide is a little bit more practical so you directly could see what how you interact with  an actinia so you send a request like htdpi myurl to actinia slash locations and then you get  you receive back a jason file which then lists up the locations so you can see here it's a  location one is nc spm08 and the second one is ecad that's the name of these locations you always  get back also the status of your request that one was successful and then you can go deeper into that  and then you add to your requests after locations you add the name of the location and then you  write for instance mapsets and then you get back a list on like on the right-handed side  a jason file which tells you about the the mapsets which are in that  in that location a very cool thing and that's what we really use in our daily work in our project  are process chains i told you before that in grass gs you're capable to chain several grass gs modules  for instance by a shell script or something like that what you also could do on the actinia side  is to roll your own process chain this is just writing a jason file like that one which is a  short one we have some that have a few hundred lines of code so you can chain different grass  modules together into one big process which is especially handy if you really work on earth  evaluation data so you can do for instance the first one is to to access a central two scene  the second one is to do a classification on that the third process in your chain would be a calculate  n dvi or something like that and and these process chains are pushed through post via the actinia  api and then actinia will will talk to grass gs and yeah let your process chain run so  that's really really interesting and and good functionality you have here so if you're more  interested about this theoretically stuff i'm not going over to to that website here there is a  website actinia dot mondialis dot de there is a ap documentation of everything you can do with  actinia or the the modules crash modules how we talk to them some documentation there's some examples  so if i think it's much more better explained there than i could do here just in a few minutes  so keep that keep that website in in mind five minutes okay um yeah sorry elastor i need another  25 for that slide to explain that but now it's just to give you a short overview about what we did  in the core we have this actinia grass gs we have several interfaces like the rest api i'm  i'm talking about there is also an open eo api we always think about a wps api we have some plugins  that directly let you connect to several kinds of data like sentinel data so that you don't have to  import the data into your grass database you just can mount it on the cloud system where it is we  have we are working on a connection to geoserver we have a connection to a metadata node in order to  get out data and integrate it in processes so a lot of things we we did all around this actinia  and actinia core in the in the past year the good thing is for you actinia is free and open  source software you find it on github since march 2019 actinia is also recognized as an osgeo  community project so we are very proud about that and yes and i want to finish with some examples  i shortened it already so i have only two of them and not three as mentioned in the description of  the call and the first one is a huge project which is run by a big telecommitation company over here  in germany you know if you look at the fiber cable connections in europe if you look at the table  some countries like estonia they are near to championship and germany is close to league two  so there's a lot of work to do on that and they are setting up a huge mainly open source gs based  planning tool and one of the core functionalities about that is that we deliver a roughly planned  pre-planned fiber cable network for certain areas and you must know that the cable installation costs  depend really heavily on the surface type because if you dig into green land it's it's much more  cheaper than you have to dig into roads or pavements or whatever but especially this  data is not available in in germany so what did we do we put a lot of data together like aerial  image, autofoto, slider data implemented in actinian process that let us automate to create  training areas and then we do object-based supervised classification using machine learning  where we put all this data in the training data in and what comes out is something like that  this is a short piece of a village very close to to bonn here and if it's this autofoto went  through our process you get something out like like that one where you can clearly see okay we  have buildings we have green land we have woodland we have the roads and that is for that process the  base to to calculate the rough network so this is all done via actinia which is integrated in a huge  huge system on on that. The second example is more about to show which amounts of data  is possible to process with with actinia this is a work that mainly marcus is working on for more  than 10 years there's an us american satellite called modis which is a sender that detects surface  temperature land surface temperature and the problem is if there are clouds like today here in  bonn you you don't get any data and the task was to interpolate these holes and to deliver a scene  with completely data on it like you can see on the on the right-handed side and this is really  interesting if you look at this modis lst data because they already cover 18 years so the the  satellite is crossing the earth for 18 years now and delivers four corriges per day four not five  four with a five 250 meter resolution so if you put all that together you have about 26,280 data  sets covering the whole world i don't want to i even can't in english name the number of pixels i  just wrote it below but it's it's quite a lot of data and and that's if you look at that if you  would calculate the data on a on a one cpu 400 megabyte ram machine and you say okay it takes  nine minutes per data set then it would take you 164 days and six hours to calculate all this data  and remember every day four new scenes come in so you probably never get finished but if you  able to scale that up and do a parallelization of that calculation and processing in for example two  eight core engine you can you can lower that down to much less much less time  so yeah that's yeah just the end of the story if you're interested in that data set i put the link  to to a metadata set describing the land surface temperature data on the slides i'm quite sure  that we're going to share the slides afterwards and yeah that's that's it that's a short story about  uh actinia grass jess uh for now thank you brilliant thank you very much tell those really really  interesting um i've got a host of questions here um first off i think we'll start with two very  similar questions so from uh usif who asks what kind of machine learning algorithms uh can we  have using grass and actinia and also linked to that is from uh david who says uh are the  machine learning algorithms you used for surface types available in an open format too um i start  with the second question i would say yes look at the look at the github page if not drop me a question  and um the first question that's really cool for me because that's the reason why i put on the slide  that i'm more of working as a consultant actually i have no idea which which algorithm actually  i used i have to talk to the developers but also you can send me just an email asking that  and i could deliver that afterwards after the talk because i'm i'm not the person who really  puts his fingers into that i'm more the presenter okay cool um and then from calid we've got do you  do you think we can process large raster geospatial data on the fly instead of firing pipelines for  back end processes that might take hours especially for UAV datasets there can be tens of gigabytes  for a single file i would say yes i think it's it's more or less uh the idea how you scale that how  you split up the process so it's not real time but near real time could be possible but of course  depends on the data depends on data format depends on the amount of data and stuff like that  but i would give it a try because the engine is is ready for that and again if if you need some  advice here just come back to me and i will connect you to the appropriate person in the in the company  that's what i'd like the type of uh sort of gun her out of you i'll give it a try and see what happens  then report next for sfuggi yes exactly um we've got one from james here and it sort of links into  something i was going to ask you as well so james asks are you involved in the ogc api process  in standard development and i was going to just ask how what you're developing with actinia might  sit in with say some of the wps stuff that exists already yes so we are not involved in that we are  involved in the open eo project which is to mate it's still it's already running for two years i think  which has the idea of setting up a unique standard for processing api stuff like that but not in the  ogc stuff and the wps um if you if you looked at the slides maybe i go back to that um it was uh  it was a red box oh i was too fast sorry that one so um we always thought about um implementing a  wps api on that we have a actually we run a pro project with mondialis and terrestris where we  connect the actinia stuff to a web interface which mainly is stuff that terrestris does  and there we had huge discussions about whether implementing wps api or just using the rest api  it ended up that we for now use the rest api so we didn't really implement the wps um api yet  but i think maybe it's not worth doing that as all the ogc apis in a uh change in the moment  to a new version so maybe it's better to to keep that on hold for for a couple of months and then  look how it turns out with the new wps replacing api okay cool um let me just quickly get so andy  is asking about the use of analysis ready data in these tools um so i don't know whether that's  whether or not they can be used or whether or not um you can process it up to to analysis ready level  you could do both of course you can do your process chains and bring your data in the shape you need  it or if you directly have the data um if you look at that slide already i've mentioned that  as well we we created some some plugins especially sentinel plugin which enables you directly to  make use of cloud's stored sentinel data without having to import it into the grass database which  would be an additional step and i could imagine of something like a plugin that directly connects to  to your data into that it's all written in in python by the way okay cool um so i think as we're  coming towards the end of the session i just i've got one question and i don't know whether you'll  be whether you'll know the answer to this one but it was you mentioned at the beginning that grass  has 400 modules i think you said um do you have any information about um the proportion of use  for each of those different ones because i'm guessing some are used by all the grass users all  the time are there are there other ones that you sort of think or maybe we could just put put out to  grass hey i i think it's impossible to to figure that out because normally people use grass gs  uh just on the desktop so we have no idea maybe it's you get a clue about that if you go into  the github and see which modules are updated and with many developer stuff like that i could  ask marcus he's not actually he's not in the office but i think he should know better a little bit  but even marcus i'm sure does not know every of these hundreds of modules in his brain
FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_0_107.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_108_503.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_504_2291.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_2292_3287.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_3288_4007.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_4008_5723.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_5724_7667.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_7668_10199.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_10200_14135.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_14136_16475.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_16476_17807.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_17808_19391.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_19392_21563.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_21564_22463.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_22464_23975.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_23976_24431.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_24432_24755.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_24756_26183.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_26184_26603.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_26604_26903.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_26904_27143.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_27144_27707.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_27708_28775.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_28776_29663.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_29664_31319.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_31320_36059.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_36060_36107.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_36108_36155.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_36156_36203.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_36204_36251.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_36252_39767.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_39768_39815.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_39816_39863.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_39864_40931.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_40932_40979.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_40980_41087.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_41088_41135.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_41136_41183.png FOSS4GUKOnline_2020_Sentinel_Adams_D3PeOmH-ge8_41184_41201.png