There and Back Again: Lessons learned in transitioning from GeoServer to MapProxy
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Banting, James
Christian, Tom
The hurdles we encountered when transitioning a project from a single GeoServer instance to an autoscaling MapProxy system while trying to mimic existing functionality. The hurdles we encountered when transitioning a project from a single GeoServer instance to an autoscaling MapProxy system while trying to mimic existing functionality. Authors and Affiliations – James Banting, Sparkgeo Tom Christian, Sparkgeo Track – Use cases & applications Topic – Software/Project development Level – 2 - Basic. General basic knowledge is required. Language of the Presentation – English
 So here we have Tom James Banting and Tom Christian from SparkGEO. In the meantime you could try to share your screen. I think James will be sharing his screen. You can hear me. Okay. And let's see. Well, they will provide a talk called There and Back Again. Lessons learned in transitioning from GeoServer to Metproxy. Again Metproxy, that's good to hear. So it will be a dual presentation. And James and Tom, they are from SparkGEO, a very innovative company from British Columbia, Canada. And in short, SparkGEO helps customers to make sense of geospatial data and maps, providing analytics insights and development support. And James here is the Vice President of Research for SparkGEO. And he is a remote sensing scientist with a background in Geography. And Tom here is a full stack developer, of course also from SparkGEO with emphasis on front end web development. I'm a strong background in geospatial technologies. So in the meantime I see your screen is shared. And I'll give the floor to you both. And we'll see each other later. Okay. Excuse me, I'm not hearing you, James. Is that better? Yeah, that's much better. Yeah. Okay, thank you. Very good. Yeah. So I'm James and my colleague Tom is here presenting on Maproxy and GeoServer. We did a project pre-pandemic about GeoServer and Maproxy. So we'll get into that today. Before we do, I'd just like to acknowledge that today in Canada is the National Day for Truth and Reconciliation. This is where we kind of honour the families lost during the residential schools here in Canada. It's kind of fitting for Indigenous workshop stuff that we're doing at FosWorG. So I'd just like to acknowledge that before we get into it. Yeah, as I mentioned, we're doing GeoServer and Maproxy pre-pandemic. We kind of started with GeoServer. We had a project where we had to move a lot of customers from one managed service onto another managed service. The older product, it was pretty old and a lot of customers had built workflows around some of the nuances in this product. So we found out that we had to address some of these nuances with our GeoServer implementation, which didn't exactly suit our needs, so we had to do a little of massaging of GeoServer and other products. I think the other project was based on MapServer, so it was an OGC thing. They had not full OGC support, but pretty close. And then they also had kind of an Esri rest implementation as well. So it was a lot to move from one managed service to the other. One of the hurdles we encountered with bringing in GeoServer right off the bat was the nuances that the customers had. So there was a lack of understanding of, yes, the customers say they need WMS, but what exactly does that mean? So we had a plan to attack GeoServer and as all plans do, they go awry. So what we picked off right at the beginning was default vanilla GeoServer, and this satisfied probably 80% of the needs for us, so it was great. We could obviously do a lot of the performance tuning that is required on GeoServer. The docs on GeoServer are good for that. There's a lot on performance tuning, and it helps quite a bit. One of the shortcomings we did have, we tried to use the REST API quite a bit, and I'm not a Java developer, and I have to go down to Java to see what some of these REST calls are doing, and it's never fun for me as a remote sensing taskist to go into Java world. Individually these extensions played really well with GeoServer together. They didn't work so well. We had a lot of issues with auth coming through for GeoWebCache, a lot of stuff with Geofence and making sure that we can keep the auth for regional masking. Again, we had a plan, and these were the best ways that we could immediately address the issues in those plans, and GeoServer was tried and proved, and it was the best solution for us. So GeoServer, we struggled with getting it after a certain point. It becomes difficult to manage. Auth is always a challenge. GeoServer comes with basic auth out of the back, out of the bag. We used some key modules. We did some header stuff. Some of these clients, as I mentioned, were built their applications on an older tech, so basic auth. GeoServer insecure it is, the answer is very, was one of the requirements that we had to do. So we had to modify the auth stuff. Some of these people had token parameters in the query. Some had it in the path. It was kind of a fun game for one of our developers to go through and find out exactly how people are authenticating. We got to play with projections quite a bit. A lot of these customers were using datums that were kind of frustrating to work with, especially NAD27. In Canada, this was used quite a bit for oil and gas, and through my career, I've come across NAD27 a lot, and I've come to really hate NAD27 just because of the datum shifts that happen. Caching was an issue for us. We set up GeoWebCache, and GeoWebCache was working great for us. We didn't have the customization we needed to do, so we looked at something else. And then for WMS, this was one of the biggest hurdles. Some of our clients were using WMS 1.0, some were using 1.1, some were using 1.3. And the switch between 1.1 and 1.3 has this XY swapping positions a lot of the time. And this is GeoNerds versus MathNerds. But this was a big hurdle we had to overcome, is getting to identify the CRS for each one or the EPSG and make sure that we're doing swaps if it's WMS. Again, we had to modify the plan according to real-world situations. It was doable. It was just frustrating. So we had one GeoServer instance running up, and customers keep coming on. And GeoServer was doing fine. It was starting to handle the load. And then it's always a DevOps problem. We had to scale quite a bit. And as people have talked in other FosWordGees, and there's a talk, I believe this year on scaling FosWordGee in kind of a distributed environment, we needed to do something like that. And it was very frustrating. We had kind of separated our development process into auth and, pardon me, we had separated our development process. So we had auth, we had Geo, we had caching, and we were trying to keep them kind of separate and be able to spin up independently in a distributed environment. So DevOps was required to make GeoServer really sing or at least make the other parts compensate for GeoServer. Oh, my God, the little map proxy logos. Well, you get it in stereo. So we brought in map proxy. This was helped to, this was brought in to help a lot with the Geo web caching issues with some authentication going through there. It allowed us a whole ton of flexibility, as was previously mentioned. It's written in Python. So this is really good for us. We're predominantly a Python shop and it makes development a lot easier. We can bring in a whole bunch of different tools. One of the big things that Tom will get into a minute is the need to dynamically generate configuration scripts for map proxy. And one of the other things that Tom will get in is authentication around map proxy and how that kind of module works and how we address that. So we, again, had to modify the plan, this time we brought in newer, kind of well-tested software to augment older, well-tested, well, I shouldn't say that, older, well-developed software. So I kind of shoehorned Tom into letting me have an added 27 rant here. So datums are very important for us in the Geo world if we're pushing out data and someone wants a specific projection, specifically, let's say, oil and gas, not specifically. Let's say an oil and gas example, a customer wants a projection in a UTM coordinate, but their base data is made off NAD 27, North American datum from 1927. And as the earth does, things move. So people are still using that 27 as a datum to reference real-world objects. It's very frustrating as a Geo to have this. So that's the end of my rant, and I will transition over to Tom now. All right, thanks, James. So as James mentioned, I'm just going to dig into some of the details on some of our implementation, particularly, I'm going to start looking at auth here. And in this context, auth is both authentication and authorization. And this is something that we had to implement with a Maproxy work, because obviously Geo server comes with a fairly rich set of configuration options around auth. But with Maproxy, we were kind of a little bit more on our own, producing our own setup. So broadly speaking, our goals were to ensure that if a WMS or the gap capabilities request comes in, then the response will only include those layers that are authorized for the caller. And so we're not showing anyone data that they cannot access. And then secondly, that if someone issues a getMap request, that we're not returning any data for which they're not authorized if they manage to get a layer name from outside of the getCapabilities document. Or if they request a mix of data for which they are and are not authorized, then we filter out the data that they're not supposed to have. And a nice thing about the approach that we took with Maproxy is that we, caching is transparent from the rest of our system. So because Maproxy manages auth for us and Maproxy manages the cache, we don't have to worry about who has access to the cache, which I think was one of the issues that we ran up against with Geo Web Cache. So on my next slide, I've just got a basic architecture diagram. So I'll spend a minute or two talking through this. So on the top left, we've got GIS client. So let's think of this as QGIS or ArcGIS desktop or any number of proprietary applications. And so this is issuing, for example, a getMapRequest. And that request hits our REST API first. So this is a custom API that we developed using a fast API, an excellent Python API development framework. And so this is kind of our proxy. This is our layer that we stick in front of Maproxy to manage auth for us, among other things. So within that API, we first have an authentication check. So this is clearly just making sure that the caller is who they say they are. And then we get into the meat of it, which is the layer authorization check. So we have an authorization DB. And this is configured with layer names, user names, and layer grouping, so we can give users access to any combination of layers. So we go to the database and we say, tell me every layer that this caller is allowed to have access to. That comes back. And then we bundle that up into a JWT JSON web token. This is an encoded string that we send as an HTTP header when we make our HTTP request over to Maproxy. So Maproxy receives the request that's been forwarded on from our API. And it includes a JWT that sells Maproxy. This caller is allowed to see this list of layers. We tell it to Maproxy and Maproxy does its thing to determine whether or not the request has access to the layers that it's requested. Next we jump over to S3, which is image cache. Just see if we have the images already cached. If we don't, then we push that request up to our upstream WMS or WMTS services, which are either managed by us or managed by the customer that we're doing this work for. And so the whole kind of auth piece here is really communicated through that JSON web token that we passed to Maproxy. And that's kind of the logic of that is wrapped up in a filter that we register with Maproxy. When it starts up, that we have to write. On the bottom right, I've just got a small example of a contracted example of what that JWT looks like. It's pretty straightforward, especially if you're already familiar with JSON web tokens. We're just listing the layers. And then just in the bottom left, grayed out there because it's not completely relevant, I just wanted to add the context. We also have a UI that we built to allow the customer to configure who has access to which layers. So that's a fairly simple UI, but it just allows the admin to say user A has access to layers, one, two, three, user B has access to layer two. And that obviously feeds into the auth DB. So next up, I think I'm talking about caching. So yeah, as James mentioned, within JWT server, we were using a GEO web cache to manage our cache. But now that we're moving over to Maproxy, we're letting Maproxy do the bulk of the work and the cache itself is stored on AWS S3. In line with Maproxy's recommendations, we configure that with a particular directory structure because we are expecting to see a very large number of requests per second in some scenarios. And that's part of the advice to avoid issues with AWS. Maproxy also suggests that whatever CRS we expect most image requests to be made in, we cache in a grid suitable for that CRS because although Maproxy can do on the fly reprojections, so let's say we have a cache in EPSG 3857 and a request comes in in 4326, Maproxy will do that reprojection on the fly for us, but we obviously want to minimize the number of times that happens for the overhead and for any potential projection issues. As James mentioned as well, we had a lot of issues with NAD27 or just a lot of fun working around its peculiarities. So one of the things that we do is we store a completely separate cache for each layer in this based on NAD27 so that if we do have to do that on the fly reprojection to a NAD27 based projection, we can do it off the NAD27 cache. So we're not forcing that data shift. So we obviously increases the volume of data that we're caching, but it allows us to sidestep some of those issues during reprojection. Next up I'm talking about Maproxy configuration. So for anyone not familiar with Maproxy, the system is configured through YAML configuration files and the way that we approach that configuration is a separate YAML file for each of the services that we expose. I don't really have time to get into why we separated out to those three services in the upper right. But essentially we have to produce these YAML files and then when that YAML configuration file changes, Maproxy reads it in and reconfigures itself. We also produce a SEED configuration file because we have background SEED tasks that populate the cache for us for any image tiles that have not already been requested by user requests. So we have a process that just watches for configuration changes in our config database. When it detects that something has changed, we have a custom Python module that reads to the config DB, generates a whole round of new YAML files and then dumps those into the directory that Maproxy reads that configuration from. So on the next slide I just talk about some of the issues we encountered with that. Part of it is because we're using S3 for our cache, one of the behaviors that we found with Maproxy is when Maproxy is given a new YAML configuration file, it immediately reconfigures the services. But when you're using S3, what that means is the first getMap request that comes in after a reconfiguration, Maproxy essentially says, I don't know if I have access to these S3 buckets and directories. And so it goes through a process of firing off HTTP head requests to a large number of end points just to see where it has access to within S3. And I believe just from looking through the code that the intent here is to throw an error if an unauthorized response comes back. But assuming everything is good and even you do have access, this is just a very simple process. It's just a very large amount of traffic that doesn't serve an immediately useful purpose for us. But that first getMap request has to wait for all those requests to complete. We have a large number of layers, a large number of grids, and a large number of zoom levels that we cache to. So this can be thousands and thousands of HTTP head requests. So some of our production users can be waiting several minutes for that first map to come back. So that's one of the issues we ran into and we're kind of addressing that by requiring that we don't really reconfigure during core business hours. The YAML format that Configure that Map Proxy expects is a little bit finicky, very particular about the formatting of certain elements and indentations. So that was a little bit of a hurdle for us to get over to understand that. And then the improvements that we'd like to make if we were to revisit this or take on another project like this. I'd like to make that S3 check with the large number of HTTP head requests optional so we don't always have to hit the buckets in S3 with a large amount of traffic. But I'd also really like to see Map Proxy with a configuration API rather than relying on these YAML configuration files so that when our configuration changes, we don't have to regenerate these YAML files every time and have Map Proxy read them in. It'd be really nice if we could just hit Map Proxy directly and say, this is the configuration through an API. So I think that's everything I wanted to talk about with Map Proxy. Yeah, cool. Thanks, Tom. So kind of general lessons learned from this. GeoServer is still very powerful and still very hard. There's a lot of good companies and very good developers who know how to make this thing sing. Map Proxy is very easy to get up and running. It's relatively easy for devs to approach it. And it works well in a distributed system which we want to play with. The auth filter in any kind of middleware there is pretty easy. I don't want to speak for Tom developing it, but it was fairly easy to develop against. So GeoServer as a back end with Map Proxy sitting on front controlling all the access and everything is a win. We actually have some customers who are still using GeoServer in the application. One of the other lessons learned is when you're filtering for blog posts and you're doing a time filter, go back, GeoServer is old, go back way before the land, before time, and find out what little tricks has developers made to get around GeoServer's nuances. So look back in the research. There's a lot of stuff there. As I mentioned at the beginning, the old service was doing native registry rest. So we had this request come through. We actually ended up working with Esri to be able to support this. And there was some finicky stuff we got to chat with some of their developers on their image server team. So that was very good. But we were able to get native Esri support from OGC stuff. That's it. That's the presentation. I'm James Banting and my colleague Tom Christian who did kind of all the work on this project is here as well. These are our contact stuff. So thanks for coming to our presentation and we're available for any questions. Okay. Thanks James and Tom. Yeah, very interesting to see how you do with Integrate for instance, Maproxy. Let's see. We have one question and maybe questions come in while you're answering. I'll put the question on screen. It's a short question but probably you will know how to answer this one. Yeah. It's a reverse TMS structure. Sorry, I'm reading it also for the listeners. I can speak a little bit about that. It's essentially just, it's probably best described by the Maproxy documentation but it's essentially just a particular ordering of nested directories. So the way that Maproxy will by default structure is cache can lead to a high level of nesting directories within directories within directories. And although I don't recall all the details because this project was a little while back, there is some behavior in AWS's S3 storage service that if you have a large number of requests per second going through that heavily nested directory structure, it starts to get upset and I think you start to see some service performance issues. But yeah, the reverse structure is essentially just changes. So rather than, I think the reverse structure ultimately starts with the zoom level, the zoom level is the last directory, it goes Y, X, Z rather than a different ordering. And essentially it just changes the degree of nesting that you have within your cache. Yeah, so rather than storing the zoom level last within X, Y, Z, you have zoom level first to look through the zooms in that kind of thing. Okay, yeah, since we have some time and I happen to be also a Maproxy user for years, because at some point I didn't use this file storage anymore and I switched over to geo package, storing tiles in geo package. And that saved me quite some problems, especially when moving, let's say from a test to a production server, basically you have one file with a geo package and it's quite efficient. So why should, well, but of course if you have S3 you have these remote calls and then probably geo package doesn't work that well. Yeah, we, so maybe you can comment. Yeah, absolutely. We had, so we had some requirements for vector products that were being drawn as rasters, so we had to put up SLDs and some of these vector products were image footprints. So there's a lot of them, they cover wide areas. So geo package worked well for a bunch of those. We loaded those up into geo server as well at the beginning because we had to throw around vectors and everyone loves shapefiles, so we went with geo package. It worked out well for us in that regard, but for the size of work we need to do, dumping all our tiles and stuff in geo package wouldn't have worked for us. Yeah, and I have another thought on that as well. So my previous experience with caching large numbers of files is one of the best reasons to avoid, one of the best reasons to package a large number of files in something like geo package is to avoid some issues you can see on with, for example, a hard drive running out of IOs, like hitting its IOs limit where you simply have too many files and the hard drives addressing system can't handle it. And so that's where something like geo package or like the MB tiles format can be really useful in just collating those things into a single file. And as you mentioned, use the transferring of files between environments becomes a lot easier. I think for us because we're using S3, those kinds of hardware related problems just go out the window, you know, we don't care how many files we're storing on S3. And if storing those files individually means that we avoid the overhead of packaging and unpackaging in and out of the kind of collated format, then that saves us a tiny bit of performance every single time. So I think in a different environment, having such a large number of individual files could be a real problem, but just given how we're storing our data and how we're using S3, it just stops being our problem. My approach has quite some possibilities in that respect. I know the Dutch National Geographic Infrastructure, they even use the Couch TV or use the Couch TV back end also from my proxy. But I see in the meantime, we have, wow, I think the question came also up in the previous, so there's another question here. Do you know if Maproxy can cache Maproxy vector files? So I was curious about the vector aspect of Maproxy and going through old emails from Maproxy group. It doesn't explicitly do vector tiles, but my understanding is that the hookups for it, for your own vector implementation are there. I don't think it caches vector tiles so often out of the box. Yeah, it came up also in the previous talk and of course this is open source software and it's also a matter of funding. And yeah, I happen to be also in the Project Steering Committee for Maproxy and yeah, what you'd rather see is this type of contribution and yeah, it's a long running issue. Next question is, can Maproxy be run in a container and I guess a Docker container? Yeah, actually, so we were looking at the Netherlands, Dutch, I think it was an environmental agency and they, PDock, PDOK is the group. Yeah, that's very familiar to me. They've done a great job of packaging up Maproxy into a Docker container with some hooks if you're deploying it in Kubernetes or if you're just running it locally. So have a look at PDOK on GitHub. They have a lot of great resources that they push out there. Yeah, so I followed up that from PDOK and have even bundled Maproxy with MapServer in single image because Maproxy can call MapServer as a library. So you don't even need back end WMS. We actually did that too. Tom was writing some of this in Maproxy. So we have one minute, sorry. I'm getting over-antigestic here. Let's see, we have probably time for one last question. Yeah, I have to move switch between environments here. That's maybe a long question.
 So here we have Tom James Banting and Tom Christian from SparkGEO.  In the meantime you could try to share your screen.  I think James will be sharing his screen.  You can hear me.  Okay.  And let's see.  Well, they will provide a talk called There and Back Again.  Lessons learned in transitioning from GeoServer to Metproxy.  Again Metproxy, that's good to hear.  So it will be a dual presentation.  And James and Tom, they are from SparkGEO, a very innovative company from British Columbia,  Canada.  And in short, SparkGEO helps customers to make sense of geospatial data and maps, providing  analytics insights and development support.  And James here is the Vice President of Research for SparkGEO.  And he is a remote sensing scientist with a background in Geography.  And Tom here is a full stack developer, of course also from SparkGEO with emphasis on  front end web development.  I'm a strong background in geospatial technologies.  So in the meantime I see your screen is shared.  And I'll give the floor to you both.  And we'll see each other later.  Okay.  Excuse me, I'm not hearing you, James.  Is that better?  Yeah, that's much better.  Yeah.  Okay, thank you.  Very good.  Yeah.  So I'm James and my colleague Tom is here presenting on Maproxy and GeoServer.  We did a project pre-pandemic about GeoServer and Maproxy.  So we'll get into that today.  Before we do, I'd just like to acknowledge that today in Canada is the National Day for  Truth and Reconciliation.  This is where we kind of honour the families lost during the residential schools here in  Canada.  It's kind of fitting for Indigenous workshop stuff that we're doing at FosWorG.  So I'd just like to acknowledge that before we get into it.  Yeah, as I mentioned, we're doing GeoServer and Maproxy pre-pandemic.  We kind of started with GeoServer.  We had a project where we had to move a lot of customers from one managed service onto  another managed service.  The older product, it was pretty old and a lot of customers had built workflows around  some of the nuances in this product.  So we found out that we had to address some of these nuances with our GeoServer implementation,  which didn't exactly suit our needs, so we had to do a little of massaging of GeoServer  and other products.  I think the other project was based on MapServer, so it was an OGC thing.  They had not full OGC support, but pretty close.  And then they also had kind of an Esri rest implementation as well.  So it was a lot to move from one managed service to the other.  One of the hurdles we encountered with bringing in GeoServer right off the bat was the nuances  that the customers had.  So there was a lack of understanding of, yes, the customers say they need WMS, but what  exactly does that mean?  So we had a plan to attack GeoServer and as all plans do, they go awry.  So what we picked off right at the beginning was default vanilla GeoServer, and this satisfied  probably 80% of the needs for us, so it was great.  We could obviously do a lot of the performance tuning that is required on GeoServer.  The docs on GeoServer are good for that.  There's a lot on performance tuning, and it helps quite a bit.  One of the shortcomings we did have, we tried to use the REST API quite a bit, and I'm not  a Java developer, and I have to go down to Java to see what some of these REST calls  are doing, and it's never fun for me as a remote sensing taskist to go into Java world.  Individually these extensions played really well with GeoServer together.  They didn't work so well.  We had a lot of issues with auth coming through for GeoWebCache, a lot of stuff with Geofence  and making sure that we can keep the auth for regional masking.  Again, we had a plan, and these were the best ways that we could immediately address the  issues in those plans, and GeoServer was tried and proved, and it was the best solution for  us.  So GeoServer, we struggled with getting it after a certain point.  It becomes difficult to manage.  Auth is always a challenge.  GeoServer comes with basic auth out of the back, out of the bag.  We used some key modules.  We did some header stuff.  Some of these clients, as I mentioned, were built their applications on an older tech,  so basic auth.  GeoServer insecure it is, the answer is very, was one of the requirements that we had to  do.  So we had to modify the auth stuff.  Some of these people had token parameters in the query.  Some had it in the path.  It was kind of a fun game for one of our developers to go through and find out exactly how people  are authenticating.  We got to play with projections quite a bit.  A lot of these customers were using datums that were kind of frustrating to work with,  especially NAD27.  In Canada, this was used quite a bit for oil and gas, and through my career, I've come  across NAD27 a lot, and I've come to really hate NAD27 just because of the datum shifts  that happen.  Caching was an issue for us.  We set up GeoWebCache, and GeoWebCache was working great for us.  We didn't have the customization we needed to do, so we looked at something else.  And then for WMS, this was one of the biggest hurdles.  Some of our clients were using WMS 1.0, some were using 1.1, some were using 1.3.  And the switch between 1.1 and 1.3 has this XY swapping positions a lot of the time.  And this is GeoNerds versus MathNerds.  But this was a big hurdle we had to overcome, is getting to identify the CRS for each one  or the EPSG and make sure that we're doing swaps if it's WMS.  Again, we had to modify the plan according to real-world situations.  It was doable.  It was just frustrating.  So we had one GeoServer instance running up, and customers keep coming on.  And GeoServer was doing fine.  It was starting to handle the load.  And then it's always a DevOps problem.  We had to scale quite a bit.  And as people have talked in other FosWordGees, and there's a talk, I believe this year on  scaling FosWordGee in kind of a distributed environment, we needed to do something like  that.  And it was very frustrating.  We had kind of separated our development process into auth and, pardon me, we had separated  our development process.  So we had auth, we had Geo, we had caching, and we were trying to keep them kind of separate  and be able to spin up independently in a distributed environment.  So DevOps was required to make GeoServer really sing or at least make the other parts compensate  for GeoServer.  Oh, my God, the little map proxy logos.  Well, you get it in stereo.  So we brought in map proxy.  This was helped to, this was brought in to help a lot with the Geo web caching issues  with some authentication going through there.  It allowed us a whole ton of flexibility, as was previously mentioned.  It's written in Python.  So this is really good for us.  We're predominantly a Python shop and it makes development a lot easier.  We can bring in a whole bunch of different tools.  One of the big things that Tom will get into a minute is the need to dynamically generate  configuration scripts for map proxy.  And one of the other things that Tom will get in is authentication around map proxy  and how that kind of module works and how we address that.  So we, again, had to modify the plan, this time we brought in newer, kind of well-tested  software to augment older, well-tested, well, I shouldn't say that, older, well-developed  software.  So I kind of shoehorned Tom into letting me have an added 27 rant here.  So datums are very important for us in the Geo world if we're pushing out data and someone  wants a specific projection, specifically, let's say, oil and gas, not specifically.  Let's say an oil and gas example, a customer wants a projection in a UTM coordinate, but  their base data is made off NAD 27, North American datum from 1927.  And as the earth does, things move.  So people are still using that 27 as a datum to reference real-world objects.  It's very frustrating as a Geo to have this.  So that's the end of my rant, and I will transition over to Tom now.  All right, thanks, James.  So as James mentioned, I'm just going to dig into some of the details on some of our  implementation, particularly, I'm going to start looking at auth here.  And in this context, auth is both authentication and authorization.  And this is something that we had to implement with a Maproxy work, because obviously Geo  server comes with a fairly rich set of configuration options around auth.  But with Maproxy, we were kind of a little bit more on our own, producing our own setup.  So broadly speaking, our goals were to ensure that if a WMS or the gap capabilities request  comes in, then the response will only include those layers that are authorized for the caller.  And so we're not showing anyone data that they cannot access.  And then secondly, that if someone issues a getMap request, that we're not returning  any data for which they're not authorized if they manage to get a layer name from outside  of the getCapabilities document.  Or if they request a mix of data for which they are and are not authorized, then we filter  out the data that they're not supposed to have.  And a nice thing about the approach that we took with Maproxy is that we, caching is  transparent from the rest of our system.  So because Maproxy manages auth for us and Maproxy manages the cache, we don't have  to worry about who has access to the cache, which I think was one of the issues that we  ran up against with Geo Web Cache.  So on my next slide, I've just got a basic architecture diagram.  So I'll spend a minute or two talking through this.  So on the top left, we've got GIS client.  So let's think of this as QGIS or ArcGIS desktop or any number of proprietary applications.  And so this is issuing, for example, a getMapRequest.  And that request hits our REST API first.  So this is a custom API that we developed using a fast API, an excellent Python API development  framework.  And so this is kind of our proxy.  This is our layer that we stick in front of Maproxy to manage auth for us, among other  things.  So within that API, we first have an authentication check.  So this is clearly just making sure that the caller is who they say they are.  And then we get into the meat of it, which is the layer authorization check.  So we have an authorization DB.  And this is configured with layer names, user names, and layer grouping, so we can give  users access to any combination of layers.  So we go to the database and we say, tell me every layer that this caller is allowed  to have access to.  That comes back.  And then we bundle that up into a JWT JSON web token.  This is an encoded string that we send as an HTTP header when we make our HTTP request  over to Maproxy.  So Maproxy receives the request that's been forwarded on from our API.  And it includes a JWT that sells Maproxy.  This caller is allowed to see this list of layers.  We tell it to Maproxy and Maproxy does its thing to determine whether or not the request  has access to the layers that it's requested.  Next we jump over to S3, which is image cache.  Just see if we have the images already cached.  If we don't, then we push that request up to our upstream WMS or WMTS services, which  are either managed by us or managed by the customer that we're doing this work for.  And so the whole kind of auth piece here is really communicated through that JSON web  token that we passed to Maproxy.  And that's kind of the logic of that is wrapped up in a filter that we register with Maproxy.  When it starts up, that we have to write.  On the bottom right, I've just got a small example of a contracted example of what that  JWT looks like.  It's pretty straightforward, especially if you're already familiar with JSON web tokens.  We're just listing the layers.  And then just in the bottom left, grayed out there because it's not completely relevant,  I just wanted to add the context.  We also have a UI that we built to allow the customer to configure who has access to which  layers.  So that's a fairly simple UI, but it just allows the admin to say user A has access  to layers, one, two, three, user B has access to layer two.  And that obviously feeds into the auth DB.  So next up, I think I'm talking about caching.  So yeah, as James mentioned, within JWT server, we were using a GEO web cache to manage our  cache.  But now that we're moving over to Maproxy, we're letting Maproxy do the bulk of the  work and the cache itself is stored on AWS S3.  In line with Maproxy's recommendations, we configure that with a particular directory  structure because we are expecting to see a very large number of requests per second  in some scenarios.  And that's part of the advice to avoid issues with AWS.  Maproxy also suggests that whatever CRS we expect most image requests to be made in,  we cache in a grid suitable for that CRS because although Maproxy can do on the fly  reprojections, so let's say we have a cache in EPSG 3857 and a request comes in in 4326,  Maproxy will do that reprojection on the fly for us, but we obviously want to minimize  the number of times that happens for the overhead and for any potential projection issues.  As James mentioned as well, we had a lot of issues with NAD27 or just a lot of fun working  around its peculiarities.  So one of the things that we do is we store a completely separate cache for each layer  in this based on NAD27 so that if we do have to do that on the fly reprojection to a NAD27  based projection, we can do it off the NAD27 cache.  So we're not forcing that data shift.  So we obviously increases the volume of data that we're caching, but it allows us to sidestep  some of those issues during reprojection.  Next up I'm talking about Maproxy configuration.  So for anyone not familiar with Maproxy, the system is configured through YAML configuration  files and the way that we approach that configuration is a separate YAML file for each of the services  that we expose.  I don't really have time to get into why we separated out to those three services in  the upper right.  But essentially we have to produce these YAML files and then when that YAML configuration  file changes, Maproxy reads it in and reconfigures itself.  We also produce a SEED configuration file because we have background SEED tasks that  populate the cache for us for any image tiles that have not already been requested by user  requests.  So we have a process that just watches for configuration changes in our config database.  When it detects that something has changed, we have a custom Python module that reads  to the config DB, generates a whole round of new YAML files and then dumps those into  the directory that Maproxy reads that configuration from.  So on the next slide I just talk about some of the issues we encountered with that.  Part of it is because we're using S3 for our cache, one of the behaviors that we found  with Maproxy is when Maproxy is given a new YAML configuration file, it immediately reconfigures  the services.  But when you're using S3, what that means is the first getMap request that comes in after  a reconfiguration, Maproxy essentially says, I don't know if I have access to these S3  buckets and directories.  And so it goes through a process of firing off HTTP head requests to a large number of  end points just to see where it has access to within S3.  And I believe just from looking through the code that the intent here is to throw an error  if an unauthorized response comes back.  But assuming everything is good and even you do have access, this is just a very simple  process.  It's just a very large amount of traffic that doesn't serve an immediately useful purpose  for us.  But that first getMap request has to wait for all those requests to complete.  We have a large number of layers, a large number of grids, and a large number of zoom  levels that we cache to.  So this can be thousands and thousands of HTTP head requests.  So some of our production users can be waiting several minutes for that first map to come  back.  So that's one of the issues we ran into and we're kind of addressing that by requiring  that we don't really reconfigure during core business hours.  The YAML format that Configure that Map Proxy expects is a little bit finicky, very particular  about the formatting of certain elements and indentations.  So that was a little bit of a hurdle for us to get over to understand that.  And then the improvements that we'd like to make if we were to revisit this or take on  another project like this.  I'd like to make that S3 check with the large number of HTTP head requests optional so we  don't always have to hit the buckets in S3 with a large amount of traffic.  But I'd also really like to see Map Proxy with a configuration API rather than relying  on these YAML configuration files so that when our configuration changes, we don't have  to regenerate these YAML files every time and have Map Proxy read them in.  It'd be really nice if we could just hit Map Proxy directly and say, this is the configuration  through an API.  So I think that's everything I wanted to talk about with Map Proxy.  Yeah, cool.  Thanks, Tom.  So kind of general lessons learned from this.  GeoServer is still very powerful and still very hard.  There's a lot of good companies and very good developers who know how to make this thing  sing.  Map Proxy is very easy to get up and running.  It's relatively easy for devs to approach it.  And it works well in a distributed system which we want to play with.  The auth filter in any kind of middleware there is pretty easy.  I don't want to speak for Tom developing it, but it was fairly easy to develop against.  So GeoServer as a back end with Map Proxy sitting on front controlling all the access  and everything is a win.  We actually have some customers who are still using GeoServer in the application.  One of the other lessons learned is when you're filtering for blog posts and you're doing  a time filter, go back, GeoServer is old, go back way before the land, before time,  and find out what little tricks has developers made to get around GeoServer's nuances.  So look back in the research.  There's a lot of stuff there.  As I mentioned at the beginning, the old service was doing native registry rest.  So we had this request come through.  We actually ended up working with Esri to be able to support this.  And there was some finicky stuff we got to chat with some of their developers on their  image server team.  So that was very good.  But we were able to get native Esri support from OGC stuff.  That's it.  That's the presentation.  I'm James Banting and my colleague Tom Christian who did kind of all the work on this project  is here as well.  These are our contact stuff.  So thanks for coming to our presentation and we're available for any questions.  Okay.  Thanks James and Tom.  Yeah, very interesting to see how you do with Integrate for instance, Maproxy.  Let's see.  We have one question and maybe questions come in while you're answering.  I'll put the question on screen.  It's a short question but probably you will know how to answer this one.  Yeah.  It's a reverse TMS structure.  Sorry, I'm reading it also for the listeners.  I can speak a little bit about that.  It's essentially just, it's probably best described by the Maproxy documentation but  it's essentially just a particular ordering of nested directories.  So the way that Maproxy will by default structure is cache can lead to a high level of nesting  directories within directories within directories.  And although I don't recall all the details because this project was a little while back,  there is some behavior in AWS's S3 storage service that if you have a large number of  requests per second going through that heavily nested directory structure, it starts to get  upset and I think you start to see some service performance issues.  But yeah, the reverse structure is essentially just changes.  So rather than, I think the reverse structure ultimately starts with the zoom level, the  zoom level is the last directory, it goes Y, X, Z rather than a different ordering.  And essentially it just changes the degree of nesting that you have within your cache.  Yeah, so rather than storing the zoom level last within X, Y, Z, you have zoom level first  to look through the zooms in that kind of thing.  Okay, yeah, since we have some time and I happen to be also a Maproxy user for years,  because at some point I didn't use this file storage anymore and I switched over to geo  package, storing tiles in geo package.  And that saved me quite some problems, especially when moving, let's say from a test to a production  server, basically you have one file with a geo package and it's quite efficient.  So why should, well, but of course if you have S3 you have these remote calls and then  probably geo package doesn't work that well.  Yeah, we, so maybe you can comment.  Yeah, absolutely.  We had, so we had some requirements for vector products that were being drawn as rasters,  so we had to put up SLDs and some of these vector products were image footprints.  So there's a lot of them, they cover wide areas.  So geo package worked well for a bunch of those.  We loaded those up into geo server as well at the beginning because we had to throw around  vectors and everyone loves shapefiles, so we went with geo package.  It worked out well for us in that regard, but for the size of work we need to do, dumping  all our tiles and stuff in geo package wouldn't have worked for us.  Yeah, and I have another thought on that as well.  So my previous experience with caching large numbers of files is one of the best reasons  to avoid, one of the best reasons to package a large number of files in something like  geo package is to avoid some issues you can see on with, for example, a hard drive running  out of IOs, like hitting its IOs limit where you simply have too many files and the hard  drives addressing system can't handle it.  And so that's where something like geo package or like the MB tiles format can be really  useful in just collating those things into a single file.  And as you mentioned, use the transferring of files between environments becomes a lot  easier.  I think for us because we're using S3, those kinds of hardware related problems just go  out the window, you know, we don't care how many files we're storing on S3.  And if storing those files individually means that we avoid the overhead of packaging and  unpackaging in and out of the kind of collated format, then that saves us a tiny bit of performance  every single time.  So I think in a different environment, having such a large number of individual files could  be a real problem, but just given how we're storing our data and how we're using S3, it  just stops being our problem.  My approach has quite some possibilities in that respect.  I know the Dutch National Geographic Infrastructure, they even use the Couch TV or use the Couch  TV back end also from my proxy.  But I see in the meantime, we have, wow, I think the question came also up in the previous,  so there's another question here.  Do you know if Maproxy can cache Maproxy vector files?  So I was curious about the vector aspect of Maproxy and going through old emails from  Maproxy group.  It doesn't explicitly do vector tiles, but my understanding is that the hookups for it,  for your own vector implementation are there.  I don't think it caches vector tiles so often out of the box.  Yeah, it came up also in the previous talk and of course this is open source software  and it's also a matter of funding.  And yeah, I happen to be also in the Project Steering Committee for Maproxy and yeah, what  you'd rather see is this type of contribution and yeah, it's a long running issue.  Next question is, can Maproxy be run in a container and I guess a Docker container?  Yeah, actually, so we were looking at the Netherlands, Dutch, I think it was an environmental  agency and they, PDock, PDOK is the group.  Yeah, that's very familiar to me.  They've done a great job of packaging up Maproxy into a Docker container with some hooks if  you're deploying it in Kubernetes or if you're just running it locally.  So have a look at PDOK on GitHub.  They have a lot of great resources that they push out there.  Yeah, so I followed up that from PDOK and have even bundled Maproxy with MapServer  in single image because Maproxy can call MapServer as a library.  So you don't even need back end WMS.  We actually did that too.  Tom was writing some of this in Maproxy.  So we have one minute, sorry.  I'm getting over-antigestic here.  Let's see, we have probably time for one last question.  Yeah, I have to move switch between environments here.  That's maybe a long question.
FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_0_359.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_360_455.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_456_539.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_540_1211.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_1212_2651.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_2652_2903.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_2904_3335.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_3336_4715.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_4716_6779.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_6780_8651.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_8652_12023.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_12024_13475.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_13476_15191.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_15192_15731.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_15732_16355.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_16356_16427.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_16428_18659.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_18660_21047.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_21048_22847.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_22848_25295.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_25296_27011.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_27012_30359.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_30360_31235.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_31236_32291.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_32292_32951.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_32952_34091.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_34092_41303.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_41304_41435.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_41436_41603.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_41604_42971.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_42972_44111.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_44112_44411.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_44412_44603.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_44604_44747.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_44748_45215.png FOSS4G_2021_-_There_and_Back_Again__Lessons_learne_2FNPQoy3VgQ_45216_45562.png