Tilez: serving seamless polygons in the browser with TopoJSON and Node.js
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Morandini, Luca
This talk will introduce the Tilez project, which provides aNode.js-based realisation of a Tile Map Service tiles in both GeoJSON andTopoJSON formats. This formats provide a seamless and highly performant usermapping experience in both OpenLayers and Leaflet.The key to fast display of vector geometries in Tilezz lies in the use oftiles, which leverage both local and server-side caching. Whilst linear features lend themselves easily to tiling, polygons have traditionally represented more of a challenge.Tilez provides further efficiencies by using TopoJSON as a transport formatbetween the server and the client. Tilez implements all these improvements to support web-based vector tiling, delivering good performance under heavy load through Node,js and CouchDB-based caching, and efficient transport through TopoJSON. This talk will cover Tilez and the practical aspects of its implementation together with use cases from the Australian Urban Research Infrastructure Network (AURIN - www.aurin.org.au).
 Oh now, more on vector tiles. No disadvantage as the MyPox ones. Actually, we aim to produce, to develop something which was pre-diagnostic in terms of client and pre-informance, and work well with our current infrastructure. By the way, I'm working at the University of Melbourne at the ORIM project, and me and my colleague Andrew Bromage developed this server called tiles. If you're on GitHub, so you go to GitHub, ORIM, tiles is there, with the Z at the end is there. There's been open source last week. So you can go and do whatever you like with it because the license is Apache 2. This is based on Node.js and uses both CouchDB and PostgreSQL. So, but pre- first of all, let's start with the issue we're facing. We faced. The ORIM project wanted to give our users a very rich user experience, a very good one. Which means that we wanted to have vector data, especially polygons, on the client. So, which meant having a large amount of data. So, and we, of course, we use the raster data only as a background, but for the, say, we wanted some high degree of interactivity. So, vector data were the only way to go. We wanted to have highlights, tool tips, that kind of stuff, or coroplets on the fly. We actually allow our user to upload their data, join with the geography data we have. So, polygons at the level of, say, state, county level, that kind of stuff, right? Statistical divisions, whatever, basically the data are joined. And then from, so the user uploads some data, those data are joined. And then on the client, the user can choose the visualization to apply. So, we were aiming to this level of interactivity. This is the level of freedom and the richness of user experience. So, of course, there were a few pitfalls along the way. So, basically, okay, of course we are, all our data that I will be showing are related to Australia. But basically, one of our requirements was to show a coroplet map like this, visualizing for the whole Australia, like two plus, two thousand plus polygons. And, as I said, vector features. So, basically, the user should have been able to change the color of the mapping, so change the intervals, change the number of classes, change the colors, highlight, click on something, adding something back, that kind of stuff, right? So, and of course, we couldn't allow all this information to be transferred to the browser. There was another requirement. For instance, we had a layer which was composed by more than 60,000 polygons, basically 90 megabytes of data. And the idea was to show all of them at state level because it wouldn't be, it's country level because it wouldn't be possible, but still to let the user interact with those kind of data. So, the, now, we all know what the rest of the types are. So, of course, they are useful because they, mainly because data can be cached in the browser and in the cache server. This is one big advantage of rest of the types. The other big advantage is that they can be pre-computed. So, this is for rest of the types. For vector types, it is, sorry, there is something missing here. Oh, it was supposed to be something, an image there. Damn it. This is upsetting. I tried to empower points and they clearly didn't work. I didn't check all of the images. All right, all right, all right, all right. Okay. So, basically, what's the issue with vector types? Is that especially with polygons, I mean, with line data, it's not like straight data, it's not really an issue to cut features into, into tiles. With polygon data, it's more of an issue because those polygons, you don't want to lead to user to see all those split polygons. So, you need a way to recombine them somehow so that the user can see a polygon all so it doesn't look at the individual tiles. So, what we aimed at was something on the client to reconstruct those polygons. Basically, the idea is that we have a lot of processing power on the client, so we can do a two-step thing. First, you load the vector types and then from the vector types, recompute the original polygons on the fly in the client because we have powerful clients. So, another issue that we were facing is that there's redundancy on data because if we, oh, by the way, we were talking about JSON data. So, we started with GeoJSON. The problem with GeoJSON is, of course, it's not the topological format. So, shared arcs between polygons are repeated, which means that data, say, between any better, between 50% or 70% larger, depending on your data, of course. So, we looked into GeoJSON as well. Damn it, really, without... Okay, so, okay, this is an example of split polygons, and of course, we don't want to have this effect. So, as you may see, Australia is a continent, and unfortunately, there's a lot of desert areas with a lot of... from a geographic point of view, it has a lot of small islands, for instance, which need to be visualized, but of course, are heavy in terms of data size. So, simplify... Another step that we had to take is to simplify geometries. Now, simplifying geometries, we use the Wigital's Douglas-Pekker algorithm, and we did this behind the scenes, of course, beforehand, using Postgrease. So, this is one of our examples. So, you may see that the lower zoom level, that polygon, that peninsula over there has been simplified, so it has fewer points. Entered over to GeoJSON. So, with TopoJSON, you don't have to repeat arcs which are already there, which are shared by two polygons. So, this was another advantage of proper processing. There are a few standards to servitize, and a few protocols, if you like. We chose the TMS one, which is the one which is basically Beakwood's now. So, a little bit more about the architecture. We use both CouchDB and Postgrease. Why? Postgrease is used to generate data the first time, and then those styles, the data, are put into CouchDB, which is used as a caching mechanism, as a cache server. Now, of course, it could be another level of cache server, but we didn't implement it so squid or whatever. We didn't use that, though. And, of course, there is a cache on the browser. So, basically, whenever there is a vector tile, first is the browser looks into its own internal cache. If it's not there, it has the that particular vector tile to the two tiles, which is the server. And tiles looks for it in CouchDB. If it is there, it serves directly to the browser. If it's not there, it calls Postgrease to generate that tile. Now, a little bit about configuring tiles. We needed a fair degree of flexibility because we wanted to serve all JSON and top of JSON. We wanted to have different layers because data are generalized. So, we store every, not every zoom level, but every set of zoom levels in a separate table. So, you could have an ungeneralized version of data in one table, and that's associated with the highest zoom level, say, between 15 and 20. You can have a more generalized, but not terapy generalized, dataset when storing a table to serve zoom levels between, say, 15 and 13 and so on. So, we need something flexible in order for tiles to retrieve data from different tables. So, we chose these, and of course, we could have many different layers. So, basically, there is a JSON to specify all the, how to produce those tiles, and in the property file. The property file is used to just give system information like where database, where the couch database is, user name, password, that kind of stuff. So, this is a fragment from the configuration file. So, basically, there are, of course, layers. One layer can be split in a different, in a range of different zoom levels. And basically, every set of, every range of zoom levels is independent. So, you can specify a different post-gis table. You can add an additional expression if you like. So, you can have the same table, but behaving in a different manner because it's some, say, some polygons or arcs or whatever are selected at a zoom level and not an other one. You can add, of course, more than one column, sorry, additional columns. So, other than the geometry and the ID, of course, the geometry and the ID must be there because you use the ID of polygons to reconstruct the polygons on the client. They probably specify the full projection and the location of the couch database, the kind of stuff. Now, the images are missing something. So, again, it was supposed to show an image, but it doesn't. Okay. Sorry about that. The way, in tiles as an API, of course, in addition to implementing the standard TMS protocol, which is, of course, is an HTTP GET verb, it uses delete and post. The post is used to populate precede a layer. So, you basically specify the name of the layer, the format could be J-O-J zone or Topo-J zone, and then a Minimax zoom level. So, via HTTP, you can just populate the cache in this way. It's prep-populated for you. With delete, of course, you can delete a layer. And if you just want to have the list of layers, the number of tiles, the weight in terms of the layer and the kind of stuff you just should get with the layers, without specifying the layer, pretty intuitive, I would say. So, another word about generating Topo-J zone. We wanted to generate both J-O-J zone and Topo-J zone. Now, Topo-J zone is implemented in Post-GIS and J-O-J zone as well. But while generating J-O-J zone in Post-GIS is simple, Topo-J zone, not much so. Because actually, you need to create a temporary table to store the data, even to generate one single Topo-J zone tile. So, it's not terribly easy. Moreover, it's terribly slow. It is low. It has a reason, because it's very dory. Very, very dory. And of course, Post-GIS is meant to be the link with massive amount of data. But these, of course, created a problem with us. So, what we did is to use the JavaScript, Topo-J zone's JavaScript library, true.js. And that was very fast. A generation of Topo-J, basically, it is a two-step process. We take J-O-J zone out of Post-GIS, then we transform into Topo-J zone, then we store in CouchDB. It was fast, but doesn't work with massive amount of data. It's okay if your data are not that massive, because it keeps everything in memory, this is why the reason why it's fast. Another thing is that it's not as dory as the Post-GIS topology, which means that you cannot be catching all the topological issues and errors. So, we had actually my colleague Andrew Bromage spend a good deal of time trying to work out all the topological errors in Post-GIS first. But then we use these, the Topo-J zone's library to generate those data. Sorry for the missing images. Okay, so for the clients, what we did is to develop a leaflet client, pre-stored forward for both J-O-J zone and Topo-J zone. Topo-J zone is very simple. For Topo-J zone, what we had to add is the just reassembling, the merging of the polygons in the client. We did it as well with OpenLayers 2, which was tricky and still under development, to be honest. The notable thing about this is if you want to merge the polygons, merge the single tiles in the original polygons on the client, there are some tricky things, because tiles are loaded synchronously, of course. And you cannot be sure that everything has been loaded. So basically we use some timing. There is an initial timing to start the merging process. And every time a new tile is loaded, there is a further delay to start a new merging process once hopefully all the tiles are being collected. So far, I mean, it is smooth, so I don't see any trouble with this approach. But to be honest, I haven't tried with this low connection. We tried with 512 kilobyte connection simulated one, rather smoothly. For lower connection speeds, I'm not quite sure. It could be that it's a little bit sketchy because probably there might be holes of tiles which are not loaded yet, and then once it's loaded, the merge process is re-triggered again. So it may not be as smooth as I would like to, but still I haven't tried that. So OK, with Leiflet it's simple enough. That's how to add the Torval JSON layer. Yeah, this is what I explained right now. These are the few references, and that's about it, I think. Questions?
 Oh now, more on vector tiles.  No disadvantage as the MyPox ones.  Actually, we aim to produce, to develop something which was pre-diagnostic in terms of client and pre-informance,  and work well with our current infrastructure.  By the way, I'm working at the University of Melbourne at the ORIM project, and me and my colleague Andrew Bromage developed this server called tiles.  If you're on GitHub, so you go to GitHub, ORIM, tiles is there, with the Z at the end is there.  There's been open source last week.  So you can go and do whatever you like with it because the license is Apache 2.  This is based on Node.js and uses both CouchDB and PostgreSQL.  So, but pre- first of all, let's start with the issue we're facing.  We faced.  The ORIM project wanted to give our users a very rich user experience, a very good one.  Which means that we wanted to have vector data, especially polygons, on the client.  So, which meant having a large amount of data.  So, and we, of course, we use the raster data only as a background, but for the, say, we wanted some high degree of interactivity.  So, vector data were the only way to go.  We wanted to have highlights, tool tips, that kind of stuff, or coroplets on the fly.  We actually allow our user to upload their data, join with the geography data we have.  So, polygons at the level of, say, state, county level, that kind of stuff, right?  Statistical divisions, whatever, basically the data are joined.  And then from, so the user uploads some data, those data are joined.  And then on the client, the user can choose the visualization to apply.  So, we were aiming to this level of interactivity.  This is the level of freedom and the richness of user experience.  So, of course, there were a few pitfalls along the way.  So, basically, okay, of course we are, all our data that I will be showing are related to Australia.  But basically, one of our requirements was to show a coroplet map like this, visualizing for the whole Australia, like two plus, two thousand plus polygons.  And, as I said, vector features.  So, basically, the user should have been able to change the color of the mapping, so change the intervals, change the number of classes, change the colors, highlight, click on something, adding something back, that kind of stuff, right?  So, and of course, we couldn't allow all this information to be transferred to the browser.  There was another requirement.  For instance, we had a layer which was composed by more than 60,000 polygons, basically 90 megabytes of data.  And the idea was to show all of them at state level because it wouldn't be, it's country level because it wouldn't be possible, but still to let the user interact with those kind of data.  So, the, now, we all know what the rest of the types are.  So, of course, they are useful because they, mainly because data can be cached in the browser and in the cache server.  This is one big advantage of rest of the types.  The other big advantage is that they can be pre-computed.  So, this is for rest of the types.  For vector types, it is, sorry, there is something missing here.  Oh, it was supposed to be something, an image there.  Damn it.  This is upsetting.  I tried to empower points and they clearly didn't work.  I didn't check all of the images.  All right, all right, all right, all right.  Okay.  So, basically, what's the issue with vector types?  Is that especially with polygons, I mean, with line data, it's not like straight data, it's not really an issue to cut features into, into tiles.  With polygon data, it's more of an issue because those polygons, you don't want to lead to user to see all those split polygons.  So, you need a way to recombine them somehow so that the user can see a polygon all so it doesn't look at the individual tiles.  So, what we aimed at was something on the client to reconstruct those polygons.  Basically, the idea is that we have a lot of processing power on the client, so we can do a two-step thing.  First, you load the vector types and then from the vector types, recompute the original polygons on the fly in the client because we have powerful clients.  So, another issue that we were facing is that there's redundancy on data because if we, oh, by the way, we were talking about JSON data.  So, we started with GeoJSON.  The problem with GeoJSON is, of course, it's not the topological format.  So, shared arcs between polygons are repeated, which means that data, say, between any better,  between 50% or 70% larger, depending on your data, of course.  So, we looked into GeoJSON as well.  Damn it, really, without...  Okay, so, okay, this is an example of split polygons, and of course, we don't want to have this effect.  So, as you may see, Australia is a continent, and unfortunately, there's a lot of desert areas with a lot of...  from a geographic point of view, it has a lot of small islands, for instance, which need to be visualized,  but of course, are heavy in terms of data size.  So, simplify...  Another step that we had to take is to simplify geometries.  Now, simplifying geometries, we use the Wigital's Douglas-Pekker algorithm,  and we did this behind the scenes, of course, beforehand, using Postgrease.  So, this is one of our examples.  So, you may see that the lower zoom level, that polygon, that peninsula over there has been simplified,  so it has fewer points.  Entered over to GeoJSON.  So, with TopoJSON, you don't have to repeat arcs which are already there, which are shared by two polygons.  So, this was another advantage of proper processing.  There are a few standards to servitize, and a few protocols, if you like.  We chose the TMS one, which is the one which is basically Beakwood's now.  So, a little bit more about the architecture.  We use both CouchDB and Postgrease.  Why?  Postgrease is used to generate data the first time, and then those styles, the data, are put into CouchDB,  which is used as a caching mechanism, as a cache server.  Now, of course, it could be another level of cache server, but we didn't implement it so squid or whatever.  We didn't use that, though.  And, of course, there is a cache on the browser.  So, basically, whenever there is a vector tile, first is the browser looks into its own internal cache.  If it's not there, it has the that particular vector tile to the two tiles, which is the server.  And tiles looks for it in CouchDB.  If it is there, it serves directly to the browser.  If it's not there, it calls Postgrease to generate that tile.  Now, a little bit about configuring tiles.  We needed a fair degree of flexibility because we wanted to serve all JSON and top of JSON.  We wanted to have different layers because data are generalized.  So, we store every, not every zoom level, but every set of zoom levels in a separate table.  So, you could have an ungeneralized version of data in one table, and that's associated with the highest zoom level, say, between 15 and 20.  You can have a more generalized, but not terapy generalized, dataset when storing a table to serve zoom levels between, say, 15 and 13 and so on.  So, we need something flexible in order for tiles to retrieve data from different tables.  So, we chose these, and of course, we could have many different layers.  So, basically, there is a JSON to specify all the, how to produce those tiles, and in the property file.  The property file is used to just give system information like where database, where the couch database is, user name, password, that kind of stuff.  So, this is a fragment from the configuration file.  So, basically, there are, of course, layers.  One layer can be split in a different, in a range of different zoom levels.  And basically, every set of, every range of zoom levels is independent.  So, you can specify a different post-gis table.  You can add an additional expression if you like.  So, you can have the same table, but behaving in a different manner because it's some, say, some polygons or arcs or whatever are selected at a zoom level and not an other one.  You can add, of course, more than one column, sorry, additional columns.  So, other than the geometry and the ID, of course, the geometry and the ID must be there because you use the ID of polygons to reconstruct the polygons on the client.  They probably specify the full projection and the location of the couch database, the kind of stuff.  Now, the images are missing something.  So, again, it was supposed to show an image, but it doesn't.  Okay. Sorry about that.  The way, in tiles as an API, of course, in addition to implementing the standard TMS protocol,  which is, of course, is an HTTP GET verb, it uses delete and post.  The post is used to populate precede a layer.  So, you basically specify the name of the layer, the format could be J-O-J zone or Topo-J zone, and then a Minimax zoom level.  So, via HTTP, you can just populate the cache in this way.  It's prep-populated for you.  With delete, of course, you can delete a layer.  And if you just want to have the list of layers, the number of tiles, the weight in terms of the layer and the kind of stuff you just should get with the layers,  without specifying the layer, pretty intuitive, I would say.  So, another word about generating Topo-J zone.  We wanted to generate both J-O-J zone and Topo-J zone.  Now, Topo-J zone is implemented in Post-GIS and J-O-J zone as well.  But while generating J-O-J zone in Post-GIS is simple, Topo-J zone, not much so.  Because actually, you need to create a temporary table to store the data, even to generate one single Topo-J zone tile.  So, it's not terribly easy.  Moreover, it's terribly slow.  It is low.  It has a reason, because it's very dory.  Very, very dory.  And of course, Post-GIS is meant to be the link with massive amount of data.  But these, of course, created a problem with us.  So, what we did is to use the JavaScript, Topo-J zone's JavaScript library, true.js.  And that was very fast.  A generation of Topo-J, basically, it is a two-step process.  We take J-O-J zone out of Post-GIS, then we transform into Topo-J zone, then we store in CouchDB.  It was fast, but doesn't work with massive amount of data.  It's okay if your data are not that massive, because it keeps everything in memory,  this is why the reason why it's fast.  Another thing is that it's not as dory as the Post-GIS topology,  which means that you cannot be catching all the topological issues and errors.  So, we had actually my colleague Andrew Bromage spend a good deal of time trying to work out all the topological errors in Post-GIS first.  But then we use these, the Topo-J zone's library to generate those data.  Sorry for the missing images.  Okay, so for the clients, what we did is to develop a leaflet client, pre-stored forward for both J-O-J zone and Topo-J zone.  Topo-J zone is very simple.  For Topo-J zone, what we had to add is the just reassembling, the merging of the polygons in the client.  We did it as well with OpenLayers 2, which was tricky and still under development, to be honest.  The notable thing about this is if you want to merge the polygons,  merge the single tiles in the original polygons on the client, there are some tricky things,  because tiles are loaded synchronously, of course.  And you cannot be sure that everything has been loaded.  So basically we use some timing.  There is an initial timing to start the merging process.  And every time a new tile is loaded, there is a further delay to start a new merging process once hopefully all the tiles are being collected.  So far, I mean, it is smooth, so I don't see any trouble with this approach.  But to be honest, I haven't tried with this low connection.  We tried with 512 kilobyte connection simulated one, rather smoothly.  For lower connection speeds, I'm not quite sure.  It could be that it's a little bit sketchy because probably there might be holes of tiles which are not loaded yet,  and then once it's loaded, the merge process is re-triggered again.  So it may not be as smooth as I would like to, but still I haven't tried that.  So OK, with Leiflet it's simple enough.  That's how to add the Torval JSON layer.  Yeah, this is what I explained right now.  These are the few references, and that's about it, I think.  Questions?
09_Luca_Morandini_720p_0_107.png 09_Luca_Morandini_720p_108_1487.png 09_Luca_Morandini_720p_1488_4079.png 09_Luca_Morandini_720p_4080_6083.png 09_Luca_Morandini_720p_6084_10463.png 09_Luca_Morandini_720p_10464_10751.png 09_Luca_Morandini_720p_10752_10883.png 09_Luca_Morandini_720p_10884_10931.png 09_Luca_Morandini_720p_10932_10979.png 09_Luca_Morandini_720p_10980_11243.png 09_Luca_Morandini_720p_11244_12131.png 09_Luca_Morandini_720p_12132_12755.png 09_Luca_Morandini_720p_12756_13571.png 09_Luca_Morandini_720p_13572_13799.png 09_Luca_Morandini_720p_13800_15419.png 09_Luca_Morandini_720p_15420_19919.png 09_Luca_Morandini_720p_19920_21803.png 09_Luca_Morandini_720p_21804_21875.png 09_Luca_Morandini_720p_21876_25151.png 09_Luca_Morandini_720p_25152_29576.png