A New GIS Toolbox For Integrating Massive Heterogeneous GIS Data For Land Use Change Analysis
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Laggner, Birgit (Thünen-Institute of Rural Studies)
Orthen, Natascha (Thünen-Institute of Rural Studies)
Agricultural land use in Germany and related impacts on the environment and the use of natural resources are key research topics at the Thünen-Institute of Rural Studies. As spatial context is essential for the analysis of causal connections, GIS data regarding all necessary information was gathered during different research projects and prepared for processing in a database. In particular, the Integrated Administration and Control System, which was available for certain project purposes for several Federal Laender and years, serves as a very detailed data source for agricultural land use. We use different Open Source GIS software like PostgreSQL/PostGIS, GRASS and QuantumGIS for geoprocessing, supplemented with the proprietary ESRI product ArcGIS. After introducing the used input data and the general processing approach, this paper presents a selection of geoprocessing routines for which Open Source GIS software was used. As an exemplary 'use case' for the conclusions from the consecutive statistical analysis, we summarize impacts of increased biogas production on agricultural land use change highlighting the trend in biogas maize cultivation and the conversion of permanent grassland to agricultural cropland.
 Okay, well, hello. We want to present our paper, a new GIF, or GIS, two books, Integrating Methodist Data for Lentius Analysis. And first of all, I want to give you a short overview. I will present the scientific background and our research objective and all used input data. After that, I will show the conceptual approach of our GIS two books with the general processing workflow and the used software product and the requirements for vector processing. And after that, my colleague Birgit Luckner will present the results of the GIS two books, the vector processing, then the core of our GIS two books with spatial overlay, the RASTA data processing, the additional data, and in the end, she will evaluate our two books and show some pros and cons of used software products. Okay, both of us are working at the Schumann Institute of Rural Studies at Brunswick, Germany. The research area is the use of resources and environmental and natural protection, and particularly, we analyze lentius change in agricultural areas of Germany, focusing on environmental impacts and effects of legal regulation. You can see three project examples. The first one is the current project, and we presented GIS two books, can be seen as a result of all these three projects. Yeah, the research objective or our research objective was to develop GIS two books, enabling the processing and analyzes of massive heterogeneous GIS data for statistical analysis, and therefore, the two books subsumes all necessary processing steps for data preparation and for integration of all input data sets into one combined data set. The data or there are some conceptual considerations. First, we decided to choose a vector polygon approach as the main database. We defined some necessary processing steps, and in the end, we, yeah, there are reviling requirements when it comes to software selection, and for example, the software has to fit in our existing infrastructure. The efficiency of the software product has to be considered, and interfaces has to be minimized, and in that context, it was very important to evaluate some open source software. On the slide, I want to show all used input data, and as you can see, there are different formats of the data. Most data is in a vector format, but there's also a russer format available or data in Excel charts. The extent of the data can be nationwide extent or only available for some of the federal lender, and there are different sources due to the federal system. As you can see, there's data for analyze of land use or natural environment, special conditions, and other data. The IICS, Integrated Administration and Control System is a very detailed data on field scale in a vector format, and it combines GIS data with land use and land use related information, but it's only available for areas for which subsidies have been paid, and outside the setting, we got a digital basic landscape model for Germany. On the slide, you can see the processing workflow and the used software product, because of the different data models, the processing, there are two parts, like the vector processing and the russer data processing. At the moment, the data is stored in a post-grasquare database with a post-GIS extension, because most of the data is available as vector data, we use a spatial overlay of all vector data at the main database. The russer data processing is done in graphs, and after all this processing is finished, we link the vector data to the russer data, and as you can see, the additional data down here, okay, that's not good. Okay, the additional data, like the situation of biogas plants in Germany is integrated into the database after geo-referencing, and all the results of all data processing we use for our statistical analyzes in SAS. As already mentioned, we use a spatial overlay of all vector data as the main data format, as the main database, and as a result, we got area sums calculated for the geometries for our statistical analyzes of land use change, and therefore the data has to meet several requirements. The datasets must be complete within system boundaries, so the boundaries of Germany or of the federal land, there should be no larger overlapping, the geometries should be valid, and the transformation of single polygons in multi-polygons is also integrated in our data preparation. Okay, now I will hand over. Thank you. So, now, now I would like to go to the results of our GIS toolbox, and the first slide is about the vector processing and the vector pre-processing. The first step is the plausibility check. We look at the data, the structure, the attribute data. I have a rough visual examination of the data for general understanding. Then we have to reproject the geometries for homogenizing of the coordinate system, and we repair all invalid geometries, and then we check for overlappings in our input data, and we do this with a self-intersection. And depending on the data we have, this step together with the elimination of the overlappings can be very time consuming. At the end, we convert all geometries into single polygons, said the wrong way, and create a unique ID for identification after the overlay process. This is a slide about our routine regarding the overlapping geometries. You see some examples for overlappings occurring in our data sets, and we have several criteria for classification of the overlappings for picking the instruments for the elimination. First is area and shape, which could be a parameter for the identification of sliver polygons. Next is the proportion of the intersecting area in relation to the original polygon area, which enables us to identify duplicate or almost duplicate. Then we have attribute data, which are relevant for the understanding if overlappings are intended, as in the case of zonal structures of a water protection area, or if they contain similar content and can be aggregated. As instruments for the elimination, we have delete for duplicates the clip for zonal structures or sliver polygons, and we can union geometries as we aggregate them. In the better case, we use an allocation table to relate the new geometries to the old ones. This slide is about our core process, the vector overlay operation. You see as an example two data sets, and what we want to do is to integrate them into one combined result, where all outlines of the polygons have to be preserved and the resulting geometries keep the information from the source polygons. We do this mainly with the overlay union function in Arqis, and we control this with this Python script because we have to cut our data into smaller chunks because we have this massive input data. We have to adjust our Python script for every federal land because of the differing input data for each land. In the end, we input the results into our PostgreSQL database and put all parts together and do an area calculation. This result, we get back into text files for SAS analyzers. Here is our raster data processing together with the raster overlay process. Since we used PostGIS 1.5, there wasn't any raster processing, and therefore we decided to use GRAS. We have to import all our data into GRAS and calculate slope and exposition maps for the elevation model. Yes, we tried to have a direct overlay of our vector data with the raster data, but it wasn't possible because of the massive data. We decided to use a point raster as the basis for the integration. What we do is we created a point raster of a 100 meter resolution with PostGIS. In PostGIS, we are doing an intersection of our vector overlay result and query all polygon information. In GRAS, we do the same query for the raster values and then we can join the data based on the point IDs. This is okay for most analyzers' questions, but we also have the ISCS data, which have a very high resolution. Together with the elevation model, with a resolution of 25 meters, we wanted a more precise approach. Therefore, we decided to use an alternative overlay operation. Because we could reduce our vector data to the ISCS vector data only, we were able to do univariate statistics in GRAS. Then we joined the statistics again to the vector overlay result based on the IDs of the ISCS geometries. This is our additional data. As Natasha already said, we have address data for biogas plants in Germany as an axle chart. We wanted to integrate the location of the biogas plants into our database because biogas plants are regarded as a potential driver for agricultural land use change. For the georeferencing, we used the database with georeferenced address data from the Federal Agency of Category 3 and Geodesy in Germany. After homogenizing formatting of our address strings, we had still problems with joining because of this field with the street address information. As you can see, there is a combination of strings and numeric characters together with delimiters and connectors. Sometimes there are only numbers or there is the street address inhabitant or only placeholders like this object without street. All information is not included in our georeferenced database. We were mostly only able to base our georeferencing on the combination of location and postcode. Only for 35% of the biogas plants, we could use the first characters of this street address field to have additional information for joining. In the end, we had a cluster of matching points for each biogas plant address. What we did, we took the center read of this cluster for the final point geometry. Now I want to come to the discussion part of the presentation. Our research object was to handle this heterogeneous massive geodata within an adequate time frame. We have to say that our approach is able to derive comprehensive GIS data sets for the statistical analyzers. We also have to state that this approach we are using is very time consuming and is regarding the data acquisition, the pre-processing and the processing part. We have to think about changes in our toolbox and we have a few options. We think about for future handling. The first one, the first one we would use the approach we are using now and using a fixed data set for a certain date and update only in very long cycles just to minimize the effort. The second option would be a step back. We would only prepare the input data sets to the state that bear ready for the overlay operation and the overlay itself would only be done for small data sets and tailored for a specific research question. The search option would be based on the point approach we used to the vector raster overlay and it would be very easy to add new data sets into our existing database then but we still have the drawback of losing information. In the end I would like to evaluate the open source software we used from our point of view and in general I will say that the software we have used was very capable of processing the massive geodata within an exact processing time for us acceptable means something between hours and several days and regarding first guess quail, we think it's a very great tool for the data storage of this massive data and each member of our team has access to every data set they need without having to exchange them. Then also the table manipulation processes are very effective and last but not least it has post GIS. Yeah post GIS is a very strong tool with many geofrocessing routines and also the community is very amazing. We still have found some drawbacks in post GIS for our approach and the main was the high accuracy of some functions which resulted in geometrical artifacts or topology exceptions. Yeah growth has shown itself as a very efficient in raster processing but for us the human export was a drawback and also we could say that if there are improvements in usability or documentation we think that the capabilities of grass could be better accessed. So on this that I would like to finish our presentation would be open for questions or discussion. Thank you. Yeah we try to use ArcGIS2 just to minimize the the number of software we had to use and but for our massive data it wasn't really working I have to say so grass was the only solution for us. At the opposite question which is why we're using ArcGIS for the unioning and intersection. We're using post GIS right? What? We're using post GIS. Yes. So why not just do it right in there? Yeah we tried. Was that working for you? No. Yeah we developed the PLPGS固al routine for doing the same thing as we do in ArcGIS but yeah the problem we have it's not able to work with more than two data sets at one time which ArcGIS is able to do and also it's much more slower I have to admit so probably with the topology the new topology at the on some probably there will be changes in the future but I can only speak for for the current. One more. No okay. Oh sorry. Well you've been you've been two options for the unioning. I was wondering how important SAS is to your workload and could you not union it in something like R which has very strong master vector union? Yes I don't know because the statistical analysis is not done by the people who do the GIS part and we have to deliver a project our colleagues are able to work with so SAS is the software everyone uses at our institute so.
 Okay, well, hello.  We want to present our paper, a new GIF, or GIS, two books, Integrating Methodist  Data for Lentius Analysis. And first of all, I want to give you a short overview.  I will present the scientific background and our research objective and all used input  data. After that, I will show the conceptual approach of our GIS two books with the general  processing workflow and the used software product and the requirements for vector processing.  And after that, my colleague Birgit Luckner will present the results of the GIS two books,  the vector processing, then the core of our GIS two books with spatial overlay, the RASTA  data processing, the additional data, and in the end, she will evaluate our two books  and show some pros and cons of used software products.  Okay, both of us are working at the Schumann Institute of Rural Studies at Brunswick, Germany.  The research area is the use of resources and environmental and natural protection, and  particularly, we analyze lentius change in agricultural areas of Germany, focusing on  environmental impacts and effects of legal regulation.  You can see three project examples. The first one is the current project, and we presented  GIS two books, can be seen as a result of all these three projects.  Yeah, the research objective or our research objective was to develop GIS two books,  enabling the processing and analyzes of massive heterogeneous GIS data for statistical analysis,  and therefore, the two books subsumes all necessary processing steps for data preparation  and for integration of all input data sets into one combined data set.  The data or there are some conceptual considerations. First, we decided to choose a vector polygon  approach as the main database. We defined some necessary processing steps, and in the end,  we, yeah, there are reviling requirements when it comes to software selection, and for example,  the software has to fit in our existing infrastructure. The efficiency of the software product has  to be considered, and interfaces has to be minimized, and in that context, it was very  important to evaluate some open source software. On the slide, I want to show all used input data,  and as you can see, there are different formats of the data. Most data is in a vector format,  but there's also a russer format available or data in Excel charts. The extent of the data can be  nationwide extent or only available for some of the federal lender, and there are different sources  due to the federal system. As you can see, there's data for analyze of land use or natural  environment, special conditions, and other data. The IICS, Integrated Administration and Control  System is a very detailed data on field scale in a vector format, and it combines GIS data with  land use and land use related information, but it's only available for areas for which subsidies  have been paid, and outside the setting, we got a digital basic landscape model for Germany.  On the slide, you can see the processing workflow and the used software product, because of the  different data models, the processing, there are two parts, like the vector processing and the russer  data processing. At the moment, the data is stored in a post-grasquare database with a  post-GIS extension, because most of the data is available as vector data, we use a spatial overlay  of all vector data at the main database. The russer data processing is done in graphs,  and after all this processing is finished, we link the vector data to the russer data,  and as you can see, the additional data down here, okay, that's not good.  Okay, the additional data, like the situation of biogas plants in Germany is integrated into  the database after geo-referencing, and all the results of all data processing we use for our  statistical analyzes in SAS. As already mentioned, we use a spatial overlay of all vector data as the  main data format, as the main database, and as a result, we got area sums calculated for the  geometries for our statistical analyzes of land use change, and therefore the data has to meet  several requirements. The datasets must be complete within system boundaries, so the  boundaries of Germany or of the federal land, there should be no larger overlapping, the  geometries should be valid, and the transformation of single polygons in multi-polygons is also  integrated in our data preparation. Okay, now I will hand over.  Thank you. So, now,  now I would like to go to the results of our GIS toolbox, and the first slide is about  the vector processing and the vector pre-processing. The first step is the plausibility check. We  look at the data, the structure, the attribute data. I have a rough visual examination of the data  for general understanding. Then we have to reproject the geometries for homogenizing  of the coordinate system, and we repair all invalid geometries, and then we check for  overlappings in our input data, and we do this with a self-intersection. And depending on the  data we have, this step together with the elimination of the overlappings can be very time consuming.  At the end, we convert all geometries into single polygons, said the wrong way,  and create a unique ID for identification after the overlay process.  This is a slide about our routine regarding the overlapping geometries. You see some examples  for overlappings occurring in our data sets, and we have several  criteria for classification of the overlappings for picking the instruments for the elimination.  First is area and shape, which could be a parameter for the identification of sliver polygons.  Next is the proportion of the intersecting area in relation to the original polygon area,  which enables us to identify duplicate or almost duplicate. Then we have attribute data,  which are relevant for the understanding if overlappings are intended, as in the case of  zonal structures of a water protection area, or if they contain similar content and can be aggregated.  As instruments for the elimination, we have delete for duplicates the clip for zonal structures  or sliver polygons, and we can union geometries as we aggregate them. In the better case, we use  an allocation table to relate the new geometries to the old ones.  This slide is about our core process, the vector overlay operation. You see as an example  two data sets, and what we want to do is to integrate them into one combined result,  where all outlines of the polygons have to be preserved and the resulting geometries  keep the information from the source polygons. We do this mainly with the overlay union function  in Arqis, and we control this with this Python script because we have to cut our data into  smaller chunks because we have this massive input data. We have to adjust our Python script  for every federal land because of the differing input data for each land. In the end, we input  the results into our PostgreSQL database and put all parts together and do an area calculation.  This result, we get back into text files for SAS analyzers.  Here is our raster data processing together with the raster overlay process.  Since we used PostGIS 1.5, there wasn't any raster processing,  and therefore we decided to use GRAS. We have to import all our data into GRAS and  calculate slope and exposition maps for the elevation model.  Yes, we tried to have a direct overlay of our vector data with the raster data, but it wasn't  possible because of the massive data. We decided to use a point raster as the basis for the  integration. What we do is we created a point raster of a 100 meter resolution with PostGIS.  In PostGIS, we are doing an intersection of our vector overlay result and query all  polygon information. In GRAS, we do the same query for the raster values and then we can  join the data based on the point IDs. This is okay for most analyzers' questions, but we also  have the ISCS data, which have a very high resolution. Together with the elevation model,  with a resolution of 25 meters, we wanted a more precise approach. Therefore, we decided to use an  alternative overlay operation. Because we could reduce our vector data to the ISCS vector data only,  we were able to do univariate statistics in GRAS. Then we joined the statistics again  to the vector overlay result based on the IDs of the ISCS geometries.  This is our additional data. As Natasha already said, we have address data for  biogas plants in Germany as an axle chart. We wanted to integrate the location of the biogas  plants into our database because biogas plants are regarded as a potential driver for agricultural  land use change. For the georeferencing, we used the database with georeferenced address data from  the Federal Agency of Category 3 and Geodesy in Germany. After homogenizing formatting of our  address strings, we had still problems with joining because of this field with the street address  information. As you can see, there is a combination of strings and numeric characters together with  delimiters and connectors. Sometimes there are only numbers or there is the street address  inhabitant or only placeholders like this object without street. All information is not included  in our georeferenced database. We were mostly only able to base our georeferencing on the  combination of location and postcode. Only for 35% of the biogas plants, we could use the first  characters of this street address field to have additional information for joining.  In the end, we had a cluster of matching points for each biogas plant address. What we did,  we took the center read of this cluster for the final point geometry.  Now I want to come to the discussion part of the presentation. Our research object was to  handle this heterogeneous massive geodata within an adequate time frame. We have to say that our  approach is able to derive comprehensive GIS data sets for the statistical analyzers. We also have  to state that this approach we are using is very time consuming and is regarding the data  acquisition, the pre-processing and the processing part. We have to think about changes in our  toolbox and we have a few options. We think about for future handling. The first one,  the first one we would use the approach we are using now and using a fixed data set for a certain  date and update only in very long cycles just to minimize the effort. The second option would be  a step back. We would only prepare the input data sets to the state that bear ready for the overlay  operation and the overlay itself would only be done for small data sets and tailored for a  specific research question. The search option would be based on the point approach we used to  the vector raster overlay and it would be very easy to add new data sets into our existing  database then but we still have the drawback of losing information. In the end I would like to  evaluate the open source software we used from our point of view and in general I will say that  the software we have used was very capable of processing the massive geodata within an exact  processing time for us acceptable means something between hours and several days and regarding  first guess quail, we think it's a very great tool for the data storage of this massive data  and each member of our team has access to every data set they need  without having to exchange them. Then also the table manipulation processes are very  effective and last but not least it has post GIS. Yeah post GIS is a very strong tool with many  geofrocessing routines and also the community is very amazing. We still have found some drawbacks  in post GIS for our approach and the main was the high accuracy of some functions  which resulted in geometrical artifacts or topology exceptions.  Yeah growth has shown itself as a very efficient in raster processing  but for us the human export was a drawback and also we could say that if there are improvements in  usability or documentation we think that the capabilities of grass could be better accessed.  So on this that I would like to finish our presentation would be open for questions or  discussion. Thank you.  Yeah we try to use ArcGIS2 just to minimize the the number of software we had to use  and but for our massive data it wasn't really working I have to say so grass was the only solution  for us. At the opposite question which is why we're using ArcGIS for the unioning and intersection.  We're using post GIS right? What? We're using post GIS. Yes. So why not just do it right in there?  Yeah we tried. Was that working for you? No. Yeah we developed the PLPGS固al routine for  doing the same thing as we do in ArcGIS but yeah the problem we have it's not  able to work with more than two data sets at one time which ArcGIS is able to do  and also it's much more slower I have to admit so probably with the topology the new topology  at the on some probably there will be changes in the future but I can only speak for for the current.  One more. No okay. Oh sorry.  Well you've been you've been two options for the unioning. I was wondering how important SAS is  to your workload and could you not union it in something like R which has very strong master vector union? Yes I don't know because the statistical analysis is not done by the  people who do the GIS part and we have to deliver a project our colleagues are able to work with so  SAS is the software everyone uses at our institute so.
A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_0_611.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_612_2015.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_2016_3155.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_3156_5279.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_5280_7163.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_7164_8735.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_8736_8783.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_8784_9383.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_9384_11243.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_11244_11303.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_11304_13199.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_13200_15479.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_15480_17435.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_17436_20723.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_20724_22163.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_22164_23567.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_23568_24119.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_24120_26855.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_26856_29915.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_29916_30071.png A_New_GIS_Toolbox_For_Integrating_Massive_Heterogeneous_GIS_Data_For_Land_Use_Change_Analysis__720p__2_30072_35014.png