Adopting OGC Standards in a Flood Alert System
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Martinez, Ivan
This presentation is about the adoption of the OGC - Open Geospatial Consortium standards in Sao Paulo Flood Alert System which was based on matrix coordinates and static maps.The Flood Alert System has more than 300 telemetric stations composed by rain gauges, water level sensors placed on rivers and reservoirs, water quality sensors, weather stations and a S-band weather radar reaching 240 kilometers of scanning range. The system offers Real Time support for a large metropolitan area and its Emergency Centers, Civil Defense groups, Government, Service companies and general public.We have integrated Geotools (for data conversion), Geoserver (services like WMS, WFS), DB2, OpenStreetMap, uDig, Quantum GIS and some other softwares in our architecture. This set of tools provides many possibilities to easily integrate our data with other systems and external data, like some Hydraulic and Hydrological models that return geospacial data with flooding area forecast and vulnerable buildings.Talking about the architecture, the adoption process, some of the issues, apllied solutions and further development.
 And Sao Paulo, just to contest to allies, Sao Paulo is one of the states of Brazil, the biggest, not the biggest but the most, with the biggest population of Brazil. And our work is related to mainly in the metropolitan area of Sao Paulo, that you can see that the population there, it's very huge, about 20 million people living in this area, so very populated and very, with a lot of problems with the flooding, which is our main concern in this area. Just to contest to allies, in which situation we use the GIS processing, I'd like to talk about Sao Paulo, the problems that we have during the summer. This is a typical day in the afternoon in Sao Paulo without rain. And this is a bad day in Sao Paulo, in the afternoon after a severe event of rain. So the same situation without rain, with rain. It doesn't happen every day, but sometimes it happens and it causes a lot of problems because imagine a city, Sao Paulo has 12 million people. So the traffic, normally it's a big problem during the harsh hours, but with this situation it becomes even worse. So Sao Paulo, we have some tools to deal with this situation. We use weather radar to do now casting and for some actions to be taken in the events. So we have, nowadays we have SBAND radar with a very nice accuracy. So this is the most important tool that we have to deal with this problem. So this is a typical information of radar. There's on the left hand side we have the scale, the hotter the color, the more intense the rain, and the lighter the color, the lower intense the rain. We also use some ground stations. We use to measure the level of the rivers and the intensity of the rain using some equipment whether, installing the weather stations. For doing this in metropolitan area we have about 300 stations. We use GPRS to transmit the data or some, in some cases, a satellite. You can see some charts here. The first one is the accumulated rain in time and the second one is the level of the river. This is good to, it would be good to realize that we have some levels, warning levels cataloged previously like attention alert, emergency and flooding. So in each case that's the water which is this level. Some actions must be taken by civil defense and governmental institutions. It's just a river cross section sample in the river in a regular situation. This is river in a hard situation near the flooding level. Our main customers are the governmental agencies like water and energy users and the civil defense, some civil construction and the traffic engineering also, they use this kind of information. São Paulo is here, is divided in many sub areas and we do this kind of monitoring for each sub area. So we have the color, I scale off color which shows that when it's yellow, it's in the tension and red, it's in alert. So at this point I will pass for my colleague Ivan to continue and to show how to use the, how we use the GIS tools. Hello, morning. In this map you can see that we get this map from GeoServer and we have some static information and some dynamic information. The states are feed to the GeoServer to change the color depending of the status of the area. Our infrastructure is based on virtual machines, locally virtual machines on VMWare and remote virtual machines on Amazon. We use Linux with basic center OS with varnish for getting performance and cache because we have a lot of traffic mainly in that site you saw in previous slide. That site we have one day with more than 2 million hits in one day and when the information is created online, it spends a lot of processing and we now use each processing gets cached by one minute. Then we have some delay on the update but just one minute, that's not too much but gives good performance with lower machines. We use Java with Apache Donkets 7 and GeoServer and our main database is the DB2. We have also Postgres and Postgees but for smaller things. Our current database for telemetric data has 260 million records, it's almost 47 gigabytes of data. The structure is based on varnishing in front of our servers, all of servers and then our request passes on varnish to get cache and gives good performance. All the services are virtual and they are redundant. In this slide you don't see, you only see the one box for each service but they are redundant. What we use about the GIS services and open protocols, we use WMS, WFX for local data and external data. We got some data from outside also. We create on GeoServer and GeoTools images for publishing directly and to use internally based on shape files and our products. We use also image maps generated on GeoServer to create hints, show hints about the places where the user puts the mouse over. Here we have an example about how we use GeoServer to simplify our work. We have a product that show a contour of the rain and it's the only one product generated by the radar. We have that as one shape on the GeoServer and we have some filters to show it as many different layers for the user. And also we use CSS and SLD for defining the structure, the display and for that product I will show the image that shows that we have a contour on the rain cells and a small point in the middle of the cells. That is the case that I show on the XML before. Here we have all the cells from the past. You can see on the right the check box on CTR passado, that one, and we index the slides that we are changing. The current selected now, we have the forecast and here we have all then together. We have this option that is very easy for the user to select and very easy for us to program because we just need to create an XML to configure our servers, our application. We generate using the GeoServer many formats and also some formats are get directly to the user and the user on the browser and some formats are processed again by our application. In mainly we provide PNG, GIF, JPG for the web browsers together with the image maps. And for use on Google Earth, we generate directly KML and PNG. Some images we do with transparency, some images without transparency to make easier depending on where it will be shown to decide what's better. And in GIS we use the two services, WMS, WFS. Here we have one of the changes on our system. We have the map on the left that shows old system that we have AutoCAD based map for the region. Every time we have to make a cut or change something, we have to do manually everything, check if the bounds are correct, everything would be made manually. We have a lot of problems with this, some errors, time consuming for updating. It's not good. Now we have some maps that are directly from shape files and some data comes directly from database that we can put together. Easy way if you need to change the bounds, the cuts, we just have to re-execute the query. No problem to update and it's faster, easy to decide what you want to over each layer then it's very efficient. Here we have another example that was very difficult to implement in our old system. Now we have some arrows to show the wind direction directly over the rain. The colors are easy to update. If you decide sometimes we need to change the color, just one line of configuration and we can change everything. Just for information, in this new system also we have option to click in these colors and it will blink the same cells in image that are with same color and you can say, oh where is 10 millimeter raining and that's arrow will blink. We have option to use external data like open street map. Sometimes the user wants to see at street level. We don't have to create in our system, we just connect to open street map to use this. It's written without rain. More technical information. I don't know if you are technical or not. We have some things that we have to do that take a lot of work to understand because documentation was not very good but we have implemented our own data store to plug in the gel tools and that store converts our data that comes from the radar that's proprietary from the company that made the radar but they provide the data format and we created the data store to integrate directly on the server. Currently we implemented using the abstract file data store but Jordy said that it's not a good thing because it will be deprecated and we started to migrate to content that data store. It's very easy to migrate. I have done during the workshop yesterday morning because we have all the difficulties was done before. For everyone that wants to make something like this I recommend to use these two links of the documentation that are the data store and also the links with tutorial have the sample how to create a CSV data store from the old method and new method. It's very easy to do this. For quick steps you have to implement a factory, extends the data store and you have to put in a file the name of your class. That's the quick steps to create your own data store. Creating the data store you have also to create the data source but it's on the tutorial you can see. We have two basic implementations. One that reads our database and provides the layers, definition of the layers that we use and also we have the file data store that reads the file and show the vector data. For the JailTools, JailServer to be presented. For doing this it's not easy to restart in JailServer. I recommend to use JMap for testing. You code in JailTools, show in Jframe and if everything works you probably will have no problem putting it to working on JailServer. You have just to put the jars in that directory. Data conversion. We have tried to use Raster and also we have tested with vectors and we decided to use vectors creating a square cell for each point from our radar data that becomes easier to style and show to the users. Then creates a lot of memory use but our data is just a square. We don't have much problems. We have one of things that are difficult that we have some products that comes from the radar that one single file have multiple types of products. Different shapes, different data. Then we created a virtual layer that we read the same file and provides a different schema for that data. We use the WGS84 that is EPSG426 and JailTools do the conversion when necessary. We don't have to mind about the conversion problem, projection problems. What we want to do is difficult to say we will do because sometimes some things changes and we have priorities. I expect to implement the SOS interface, the sensor observation interface and also implement the data store with time support correctly. I have a simple implementation but I think it's not correct. We have to rewrite it. Offer direct WMS services for the external users. Only our internal system users, our internal system uses the WMS and also we are internal users but not for the public because it needs too many machines, computing and we are checking what's the best way to provide this. But we want to provide this kind of service for users because the user will have a better interface can use other programs, local programs instead of just web browsers. Multidimensional, we have seen a workshop that I understand now that ImageMosaic can be used together also for vector and appear to be a good way to integrate with our system. And next time I will get the code, I will try to implement that in our system. I think it's an eye overview what we do and maybe you have some questions that can be used on the time we have here. No, when you have the code, you have for the data, you have basically two options. We use raster, that's just points and when I call vector is like the shapefiles. But vector is internal format. All shapefiles will be managed by vector. As I'm converting my matrix to internal, I use the vector like the format. You say how we go from this to that alerts. That point is manually done because we have many information and we have some problems when we try with some automatic information. Because sometimes we get a sensor that gets a problem. It goes very high, too fast and not real. Sometimes it's real. Then the people that works 24 by 7 in the emerging system check the information, get some of our internal alerts and really the sites. It goes yellow, it goes red. Based on some information that the system provides. But that the images go to public and it's updated one minute by one minute. But just when one person selects the information, it goes to public. We have many things that provide the information and one user decides this. We have the radar uses square points. We have created a routine that makes a simple conversion that defines the central point of the radar. And we go one kilometer, one kilometer and create the square grid. We have tried with different routines and find one that gets better results. We have plugins, grid models and decided by one. That's the difference. We have, let me show what is not really on the presentation, after presentation, just for reference. We have that square and we have that one that you see that's not really square in the same area. We can choose between the projections and decide what's better. We have decided for this square. Okay, we also, we don't use next red. Yes, we use hydrological model. And now we are using a swim model from EPA. And we assimilate data from weather radar and weather stations. And this model, it runs in real time every 10 minutes. It makes a brand to do a forecasting for this forecasting generates a flooding area for the next maximum two hours. So we use the images from weather radar, images, the current image, but also the now casting generated by weather radar. Which can give us one hour in advance with a good reliability. So this information is assimilated by our hydrological model. Okay, more questions? No? Time is over. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.
 And Sao Paulo, just to contest to allies, Sao Paulo is one of the states of Brazil,  the biggest, not the biggest but the most, with the biggest population of Brazil.  And our work is related to mainly in the metropolitan area of Sao Paulo, that you can see that the  population there, it's very huge, about 20 million people living in this area, so very  populated and very, with a lot of problems with the flooding, which is our main concern  in this area.  Just to contest to allies, in which situation we use the GIS processing, I'd like to talk  about Sao Paulo, the problems that we have during the summer.  This is a typical day in the afternoon in Sao Paulo without rain.  And this is a bad day in Sao Paulo, in the afternoon after a severe event of rain.  So the same situation without rain, with rain.  It doesn't happen every day, but sometimes it happens and it causes a lot of problems  because imagine a city, Sao Paulo has 12 million people.  So the traffic, normally it's a big problem during the harsh hours, but with this situation  it becomes even worse.  So Sao Paulo, we have some tools to deal with this situation.  We use weather radar to do now casting and for some actions to be taken in the events.  So we have, nowadays we have SBAND radar with a very nice accuracy.  So this is the most important tool that we have to deal with this problem.  So this is a typical information of radar.  There's on the left hand side we have the scale, the hotter the color, the more intense  the rain, and the lighter the color, the lower intense the rain.  We also use some ground stations.  We use to measure the level of the rivers and the intensity of the rain using some equipment  whether, installing the weather stations.  For doing this in metropolitan area we have about 300 stations.  We use GPRS to transmit the data or some, in some cases, a satellite.  You can see some charts here.  The first one is the accumulated rain in time and the second one is the level of the river.  This is good to, it would be good to realize that we have some levels, warning levels cataloged  previously like attention alert, emergency and flooding.  So in each case that's the water which is this level.  Some actions must be taken by civil defense and governmental institutions.  It's just a river cross section sample in the river in a regular situation.  This is river in a hard situation near the flooding level.  Our main customers are the governmental agencies like water and energy users and the civil  defense, some civil construction and the traffic engineering also, they use this kind of information.  São Paulo is here, is divided in many sub areas and we do this kind of monitoring for  each sub area.  So we have the color, I scale off color which shows that when it's yellow, it's in the  tension and red, it's in alert.  So at this point I will pass for my colleague Ivan to continue and to show how to use the,  how we use the GIS tools.  Hello, morning.  In this map you can see that we get this map from GeoServer and we have some static information  and some dynamic information.  The states are feed to the GeoServer to change the color depending of the status of the area.  Our infrastructure is based on virtual machines, locally virtual machines on VMWare and remote  virtual machines on Amazon.  We use Linux with basic center OS with varnish for getting performance and cache because  we have a lot of traffic mainly in that site you saw in previous slide.  That site we have one day with more than 2 million hits in one day and when the information  is created online, it spends a lot of processing and we now use each processing gets cached  by one minute.  Then we have some delay on the update but just one minute, that's not too much but gives  good performance with lower machines.  We use Java with Apache Donkets 7 and GeoServer and our main database is the DB2.  We have also Postgres and Postgees but for smaller things.  Our current database for telemetric data has 260 million records, it's almost 47 gigabytes  of data.  The structure is based on varnishing in front of our servers, all of servers and then our  request passes on varnish to get cache and gives good performance.  All the services are virtual and they are redundant.  In this slide you don't see, you only see the one box for each service but they are  redundant.  What we use about the GIS services and open protocols, we use WMS, WFX for local data  and external data.  We got some data from outside also.  We create on GeoServer and GeoTools images for publishing directly and to use internally  based on shape files and our products.  We use also image maps generated on GeoServer to create hints, show hints about the places  where the user puts the mouse over.  Here we have an example about how we use GeoServer to simplify our work.  We have a product that show a contour of the rain and it's the only one product generated  by the radar.  We have that as one shape on the GeoServer and we have some filters to show it as many  different layers for the user.  And also we use CSS and SLD for defining the structure, the display and for that product  I will show the image that shows that we have a contour on the rain cells and a small point  in the middle of the cells.  That is the case that I show on the XML before.  Here we have all the cells from the past.  You can see on the right the check box on CTR passado, that one, and we index the slides  that we are changing.  The current selected now, we have the forecast and here we have all then together.  We have this option that is very easy for the user to select and very easy for us to  program because we just need to create an XML to configure our servers, our application.  We generate using the GeoServer many formats and also some formats are get directly to  the user and the user on the browser and some formats are processed again by our application.  In mainly we provide PNG, GIF, JPG for the web browsers together with the image maps.  And for use on Google Earth, we generate directly KML and PNG.  Some images we do with transparency, some images without transparency to make easier  depending on where it will be shown to decide what's better.  And in GIS we use the two services, WMS, WFS.  Here we have one of the changes on our system.  We have the map on the left that shows old system that we have AutoCAD based map for  the region.  Every time we have to make a cut or change something, we have to do manually everything,  check if the bounds are correct, everything would be made manually.  We have a lot of problems with this, some errors, time consuming for updating.  It's not good.  Now we have some maps that are directly from shape files and some data comes directly from  database that we can put together.  Easy way if you need to change the bounds, the cuts, we just have to re-execute the query.  No problem to update and it's faster, easy to decide what you want to over each layer  then it's very efficient.  Here we have another example that was very difficult to implement in our old system.  Now we have some arrows to show the wind direction directly over the rain.  The colors are easy to update.  If you decide sometimes we need to change the color, just one line of configuration and  we can change everything.  Just for information, in this new system also we have option to click in these colors and  it will blink the same cells in image that are with same color and you can say, oh where  is 10 millimeter raining and that's arrow will blink.  We have option to use external data like open street map.  Sometimes the user wants to see at street level.  We don't have to create in our system, we just connect to open street map to use this.  It's written without rain.  More technical information.  I don't know if you are technical or not.  We have some things that we have to do that take a lot of work to understand because documentation  was not very good but we have implemented our own data store to plug in the gel tools  and that store converts our data that comes from the radar that's proprietary from the  company that made the radar but they provide the data format and we created the data store  to integrate directly on the server.  Currently we implemented using the abstract file data store but Jordy said that it's not  a good thing because it will be deprecated and we started to migrate to content that  data store.  It's very easy to migrate.  I have done during the workshop yesterday morning because we have all the difficulties  was done before.  For everyone that wants to make something like this I recommend to use these two links  of the documentation that are the data store and also the links with tutorial have the  sample how to create a CSV data store from the old method and new method.  It's very easy to do this.  For quick steps you have to implement a factory, extends the data store and you have to put  in a file the name of your class.  That's the quick steps to create your own data store.  Creating the data store you have also to create the data source but it's on the tutorial you  can see.  We have two basic implementations.  One that reads our database and provides the layers, definition of the layers that we  use and also we have the file data store that reads the file and show the vector data.  For the JailTools, JailServer to be presented.  For doing this it's not easy to restart in JailServer.  I recommend to use JMap for testing.  You code in JailTools, show in Jframe and if everything works you probably will have  no problem putting it to working on JailServer.  You have just to put the jars in that directory.  Data conversion.  We have tried to use Raster and also we have tested with vectors and we decided to use vectors  creating a square cell for each point from our radar data that becomes easier to style  and show to the users.  Then creates a lot of memory use but our data is just a square.  We don't have much problems.  We have one of things that are difficult that we have some products that comes from the  radar that one single file have multiple types of products.  Different shapes, different data.  Then we created a virtual layer that we read the same file and provides a different schema  for that data.  We use the WGS84 that is EPSG426 and JailTools do the conversion when necessary.  We don't have to mind about the conversion problem, projection problems.  What we want to do is difficult to say we will do because sometimes some things changes  and we have priorities.  I expect to implement the SOS interface, the sensor observation interface and also implement  the data store with time support correctly.  I have a simple implementation but I think it's not correct.  We have to rewrite it.  Offer direct WMS services for the external users.  Only our internal system users, our internal system uses the WMS and also we are internal  users but not for the public because it needs too many machines, computing and we are checking  what's the best way to provide this.  But we want to provide this kind of service for users because the user will have a better  interface can use other programs, local programs instead of just web browsers.  Multidimensional, we have seen a workshop that I understand now that ImageMosaic can be used  together also for vector and appear to be a good way to integrate with our system.  And next time I will get the code, I will try to implement that in our system.  I think it's an eye overview what we do and maybe you have some questions that can be  used on the time we have here.  No, when you have the code, you have for the data, you have basically two options.  We use raster, that's just points and when I call vector is like the shapefiles.  But vector is internal format.  All shapefiles will be managed by vector.  As I'm converting my matrix to internal, I use the vector like the format.  You say how we go from this to that alerts.  That point is manually done because we have many information and we have some problems  when we try with some automatic information.  Because sometimes we get a sensor that gets a problem.  It goes very high, too fast and not real.  Sometimes it's real.  Then the people that works 24 by 7 in the emerging system check the information, get  some of our internal alerts and really the sites.  It goes yellow, it goes red.  Based on some information that the system provides.  But that the images go to public and it's updated one minute by one minute.  But just when one person selects the information, it goes to public.  We have many things that provide the information and one user decides this.  We have the radar uses square points.  We have created a routine that makes a simple conversion that defines the central point  of the radar.  And we go one kilometer, one kilometer and create the square grid.  We have tried with different routines and find one that gets better results.  We have plugins, grid models and decided by one.  That's the difference.  We have, let me show what is not really on the presentation, after presentation, just  for reference.  We have that square and we have that one that you see that's not really square in the same  area.  We can choose between the projections and decide what's better.  We have decided for this square.  Okay, we also, we don't use next red.  Yes, we use hydrological model.  And now we are using a swim model from EPA.  And we assimilate data from weather radar and weather stations.  And this model, it runs in real time every 10 minutes.  It makes a brand to do a forecasting for this forecasting generates a flooding area for the  next maximum two hours.  So we use the images from weather radar, images, the current image, but also the now casting  generated by weather radar.  Which can give us one hour in advance with a good reliability.  So this information is assimilated by our hydrological model.  Okay, more questions?  No?  Time is over.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.
03_Ivan_Martinez_720p_0_647.png 03_Ivan_Martinez_720p_648_1919.png 03_Ivan_Martinez_720p_1920_3695.png 03_Ivan_Martinez_720p_3696_3971.png 03_Ivan_Martinez_720p_3972_4175.png 03_Ivan_Martinez_720p_4176_4631.png 03_Ivan_Martinez_720p_4632_4991.png 03_Ivan_Martinez_720p_4992_6035.png 03_Ivan_Martinez_720p_6036_6383.png 03_Ivan_Martinez_720p_6384_6875.png 03_Ivan_Martinez_720p_6876_8627.png 03_Ivan_Martinez_720p_8628_11363.png 03_Ivan_Martinez_720p_11364_12395.png 03_Ivan_Martinez_720p_12396_13499.png 03_Ivan_Martinez_720p_13500_15047.png 03_Ivan_Martinez_720p_15048_16427.png 03_Ivan_Martinez_720p_16428_18251.png 03_Ivan_Martinez_720p_18252_20051.png 03_Ivan_Martinez_720p_20052_23063.png 03_Ivan_Martinez_720p_23064_24335.png 03_Ivan_Martinez_720p_24336_30623.png 03_Ivan_Martinez_720p_30624_30719.png 03_Ivan_Martinez_720p_30720_30767.png 03_Ivan_Martinez_720p_30768_33635.png 03_Ivan_Martinez_720p_33636_33683.png 03_Ivan_Martinez_720p_33684_33731.png 03_Ivan_Martinez_720p_33732_33791.png 03_Ivan_Martinez_720p_33792_34475.png 03_Ivan_Martinez_720p_34476_34535.png 03_Ivan_Martinez_720p_34536_36563.png 03_Ivan_Martinez_720p_36564_36580.png