BeyondTracks: an OpenStreetMap case study
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Harvey, Andrew
BeyondTracks: an OpenStreetMap case study
 Yeah, I'm Andrew Harvey. I'm here to talk about a side project that I worked on just kind of on the weekends that I was interested in. It's called Beyond Tracks. It's a website, beyondtracks.com. In the next like two weeks or so, we're going to launch a new version of it. It's just been so busy with this conference. I haven't had a chance to do that yet. But, so I joined OpenStream app in 2010. And unlike many other maps, it actually showed a lot of outdoor walking tracks. I looked at other maps and either they were a topo map from 20 years ago, or it didn't have the detail, or it was another map that just didn't have the content there. But OpenStream app had a lot of these trails, which I thought was pretty cool. And the other thing is, I could actually go and edit it. So if I noticed that stuff was missing, I could go and edit that and fix it up, add in new tracks. So that was also really good. And you can download all the data. So you can use it for other purposes, apart from just kind of looking at the map that's on OpenStreetMap.org. So I kind of had all those criteria met. And I was really interested in how can I take OpenStreetMap data and the information that many people contribute to it. So not just where paths are, but what the surface of that is like and all kinds of other information about that that isn't really exposed on this map. So how can I make that more useful? So I thought, OK, I'll take, instead of using the OpenStreetMap map, I'll take the data and start showing it in a different way. So showing information about how long walks are, what the climb is, photos from it, information about the surface, about if this trail is accessible or not. And I'll go into some of the ways that we use OpenStreetMap to determine accessibility of walks. So this is an example of the new one that's going to launch. It's just got all these different tracks from OpenStreetMap, but laid out in a way that people can search for them easier, as opposed to being map-based. There's also a map on here too, as well. So it's all built on top of OpenStreetMap data. So OpenStreetMap stores paths. So who's used OpenStreetMap? Who's edited OpenStreetMap? OK. Yeah, so it stores paths. But although there's a route hiking relation, it's only really good for well-defined and well-marked out routes. A lot of the times things are just kind of, they're not defined as here's a route. It's just, this is a good way to go. You kind of walk along this track and then along this one. So that part, I couldn't really upload that to OpenStreetMap, because it's just my opinion of why I think it's a good route to take for a walk. So I built a database of walks on top of the OpenStreetMap paths and data. And so I've open-licensed it. It's all on GitLab or OpenSource. And it's just in the.osm format. So I actually use.osm, the desktop OpenStreetMap editor, to maintain all that. Pulling tracks from OpenStreetMap and then using them as the source. So you can actually go in and look at that GitLab repo if you actually want to get into the data. So yeah, at the moment I'm actually just storing the whole routes, because it's quick and easy. But it does mean that if things get updated in OpenStreetMap, I've got to go and update it here. So I'm going to go ahead and improve that. So here's the data. This is what the website looks like. And I'm going to go through how I transform that data into something that is more accessible to just kind of everybody, regardless of whether they're like a really technical mapping person or not. So I use GitLab. They've got really good tools to automate this process so that I can automatically run these updates of when the OpenStreetMap data gets updated. So it's set up to run as a nightly job. And so I pull in these.OSM walk files, the Australian OpenStreetMap extract, the terrain model of the whole of Australia, and a whole bunch of photos from Flickr. And put that all into PostgreSQL with PostGIS extensions. There's a whole bunch of crunching that goes on there. And the output says GeoJSON files and some Mapbox Vector tiles. And that's the output that the web page uses. So there's a whole bunch of tools that are really handy for that, like OGR to OGR, Rasta to PGSQL. So it actually loads in a PostGIS Rasta type. And some other tools to actually load in the Australian OpenStreetMap data into PostGIS. So from all of that crunching, we can actually generate this elevation profile from that root. And that's all done. You can see there's a link here. I can share these slides to how that actually gets crunched in PostGIS. We also generate these little spark graphs, which is just the elevation profile. So you can see here it's quite a simple Postgres function. And other things like calculating the ascent and descent, the climb. This is a PostGIS function. This kind of goes through every step and calculates the difference. The other thing is map matching. So there's a lot of features that are relevant to people doing a bush walk or a hike, things like where there's toilets, where there's lookouts, all kinds of things. And all of that is mapped in OpenStreetMap or can be mapped. So we pull out those features that are kind of along that track or very close to the track and match it back to the track. So we know what features from OpenStreetMap are on each walk. So we pull out the SAC scale. That's a tag that you can add to features in OpenStreetMap, which says, is this like a kind of a basic walk that people with not much experience can do? Or is this like a really demanding track where you kind of need to use your hands to climb over things? So it's a very broad tag, but it is useful. There's also a tag called trail visibility. It's probably not the best tag because it's very hard to kind of separate some of these levels apart. There's on the Wiki page, it kind of describes what the different levels are. But as a broad sense, there is like a difference between excellent and horrible. So it is helpful when OpenStreetMap has this. And I personally try to add this every time I add a track. But anyone else who I see other people adding it, and it is really helpful. But also things like ladders. So in OpenStreetMap, you can tag a ladder. Some people, they might get a bit scared of this. So we want to be able to know where that is from OpenStreetMap and show on the on the on tracks that for this walk, yeah, there's a ladder and it's it's like 10 meters higher. So just warning people in advance. Things like safety ropes, you can tag them. And so these are all things that I guess like big mapping providers probably didn't think of. They're not it's not there. It's not what they would typically map. So that's why I like OpenStreetMap because we can map things that are interesting to to whatever kind of interest you have. Rung so to kind of climb up a rock. Surface information. So we want to know if it's sand, if it's paved or if it's unpaved because say that affects how accessible the walk is, how long, how slow is going to be to walk along it? Like it's a lot slower walking in sand than on concrete. So yeah, it affects the walk difficulty and time estimate. So we use all these postures functions to take features from OpenStreetMap, match them back to the actual path and then work out how far along the walk. That is so yeah, we can determine how far along the features are. So to know say for drinking water or toilets, are they so far apart affects how we factor in the accessibility of a walk. And also ordering photos. So we have all these photos they can automatically be ordered based on where they are along the walk. And there's a whole bunch of other libraries that we pull in, useful information. So I mean this walk has a location and from that location we can, there's good open source libraries to calculate sunrise, sunset times, weather forecasts based on the, so we use dark sky where you can pass in lat long coordinates and it will give the forecast based on that position as opposed to kind of the nearest weather station. We pull in some open data feeds. So in New South Wales they publish park alerts, so we pull that in as a data feed. And also bushfire alerts. So I know for New South Wales ACT they have CCB, open licensed, open data that we pull in and show. So where that fire is a certain distance away from the walk would actually show that alert on the page to say, hey there's an active bushfire near this walk. So you can see here is an example of here's the walk and we just pop it up there so people are aware. People have that information. It brings, it's really bringing together a lot of information sources and putting them together. And just quickly, so we, when you share like a URL on Twitter it will generate this nice card and we have like this short description here and all that is generated automatically from the data. In this case it says it's short so we just classify, determine how long the walk is to save it so short and medium or long. We say where it is. We say what kinds of things you'd likely see. So that's not manually entered. That's if we find there's waterfalls or swimming on this walk from the open street map data then we will populate that. There's other things we say like if it's near the coastline, so in open stream map you can enter the coastline then we'll say it's a coastal walk. If it's a long way from roads or populated settlements we say it's a remote walk. So all of that is just automatically generated to form this walk description as opposed to manually entering it. And yeah like we've implemented search. So open stream map has a lot of good detailed level of data that other providers don't necessarily have and that was kind of one of my initial concerns or initial reasons of doing this. So that hey we have all this great data in open stream that's entered but if I search it on Google Maps it's not going to show up. So it's great that we're able to use that. So I encourage you to go to the site, check it out. Yeah as I mentioned we're launching a new version next two weeks or so. I don't know if we have time for questions. Okay, okay perfect timing. Yeah, yeah. Does OSM already support things like does this trail have trail markers? Yeah so the tag, the trail visibility, the definition of excellent means that there's a lot of either guideposts to tell you like turn here or there's just markers along the track so you can kind of map it with that trail visibility tag. And whether the track is maintained? Yeah so that's a hard one because like if it's really overgrown typically the trail visibility you wouldn't score excellent. It's not really, that's why that tag, yeah it's not the best tag but you can kind of enter that kind of detail with it. Yeah, David? Thanks for talking Andrew. Have you used or like my whole app story if you're writing something like Street to complete to get some of the information? Yeah well I think Street complete is great because people who are going out and walking it'll just prompt them with hey there's a road here and we don't have surface information what is it? So you like you can use that even when you're going outside it's not just for roads. I know that, some of those people don't want to like help in Street complete. Okay. Yeah, yeah so. It's not something completely different. Yeah like the Street complete is a mobile app for everyone else that lets you, it's basically it gives you, it says hey this street that you're walking on we don't know what surface it is. Please select from these options. Is it paved? Is it unpaved? Is it sand? So you can very easily just go out and map and contribute to OpenStreetMap. And so it's an open source tool and it's good because anyone can submit challenges so for some of the things like trail visibility they're probably not on Street complete but I do think it's saying that we should try to do. Yeah. Yep, yeah so it's quite easy for everyone to contribute that. Yeah. I have a question. So in Victoria I think Australia there's guidelines on like what different types of track quality and distance and how hard they are. Yeah, yeah that's like a strain standard. Yeah, yeah it's like grade one to five. They actually know something about trail visibility and marked distance as well that's like logical. Yeah, yeah so I guess I've modeled it after OpenStreetMap which tries to be kind of work, it works globally and it can be hard to devise things that work globally. We haven't like for the on tracks we haven't adopted that as strain standard. I do think there are some challenges with it and in the way they classify walks because like I think they say if it's wheelchair accessible it's grade one but you could have, yeah so I won't go into all the details but it's not something we have done but yeah it's a good option. How many tracks do you go to in your data space? About 380. But the thing is they're mostly ones that I'm familiar with. The repository for the walks is open source and as I mentioned like a lot of the time it's just a personal opinion of yeah take this track and then go this track so we can't necessarily enter that into OpenStreetMap so hopefully if there are particular walks that you know of you can submit them in. Couldn't they just give a name? Yeah, yeah there could be the name but sometimes it might be take this track which is called like such and such name and then there's another segment which has a different name in another segment and it's about joining them together.
 Yeah, I'm Andrew Harvey.  I'm here to talk about a side project that I worked on just kind of on the weekends that  I was interested in.  It's called Beyond Tracks.  It's a website, beyondtracks.com.  In the next like two weeks or so, we're going to launch a new version of it.  It's just been so busy with this conference.  I haven't had a chance to do that yet.  But, so I joined OpenStream app in 2010.  And unlike many other maps, it actually showed a lot of outdoor walking tracks.  I looked at other maps and either they were a topo map from 20 years ago, or it didn't  have the detail, or it was another map that just didn't have the content there.  But OpenStream app had a lot of these trails, which I thought was pretty cool.  And the other thing is, I could actually go and edit it.  So if I noticed that stuff was missing, I could go and edit that and fix it up, add  in new tracks.  So that was also really good.  And you can download all the data.  So you can use it for other purposes, apart from just kind of looking at the map that's  on OpenStreetMap.org.  So I kind of had all those criteria met.  And I was really interested in how can I take OpenStreetMap data and the information that  many people contribute to it.  So not just where paths are, but what the surface of that is like and all kinds of other  information about that that isn't really exposed on this map.  So how can I make that more useful?  So I thought, OK, I'll take, instead of using the OpenStreetMap map, I'll take the data  and start showing it in a different way.  So showing information about how long walks are, what the climb is, photos from it, information  about the surface, about if this trail is accessible or not.  And I'll go into some of the ways that we use OpenStreetMap to determine accessibility  of walks.  So this is an example of the new one that's going to launch.  It's just got all these different tracks from OpenStreetMap, but laid out in a way that  people can search for them easier, as opposed to being map-based.  There's also a map on here too, as well.  So it's all built on top of OpenStreetMap data.  So OpenStreetMap stores paths.  So who's used OpenStreetMap?  Who's edited OpenStreetMap?  OK.  Yeah, so it stores paths.  But although there's a route hiking relation, it's only really good for well-defined and  well-marked out routes.  A lot of the times things are just kind of, they're not defined as here's a route.  It's just, this is a good way to go.  You kind of walk along this track and then along this one.  So that part, I couldn't really upload that to OpenStreetMap, because it's just my opinion  of why I think it's a good route to take for a walk.  So I built a database of walks on top of the OpenStreetMap paths and data.  And so I've open-licensed it.  It's all on GitLab or OpenSource.  And it's just in the.osm format.  So I actually use.osm, the desktop OpenStreetMap editor, to maintain all that.  Pulling tracks from OpenStreetMap and then using them as the source.  So you can actually go in and look at that GitLab repo if you actually want to get into  the data.  So yeah, at the moment I'm actually just storing the whole routes, because it's quick  and easy.  But it does mean that if things get updated in OpenStreetMap, I've got to go and update  it here.  So I'm going to go ahead and improve that.  So here's the data.  This is what the website looks like.  And I'm going to go through how I transform that data into something that is more accessible  to just kind of everybody, regardless of whether they're like a really technical mapping person  or not.  So I use GitLab.  They've got really good tools to automate this process so that I can automatically run  these updates of when the OpenStreetMap data gets updated.  So it's set up to run as a nightly job.  And so I pull in these.OSM walk files, the Australian OpenStreetMap extract, the terrain  model of the whole of Australia, and a whole bunch of photos from Flickr.  And put that all into PostgreSQL with PostGIS extensions.  There's a whole bunch of crunching that goes on there.  And the output says GeoJSON files and some Mapbox Vector tiles.  And that's the output that the web page uses.  So there's a whole bunch of tools that are really handy for that, like OGR to OGR, Rasta  to PGSQL.  So it actually loads in a PostGIS Rasta type.  And some other tools to actually load in the Australian OpenStreetMap data into PostGIS.  So from all of that crunching, we can actually generate this elevation profile from that  root.  And that's all done.  You can see there's a link here.  I can share these slides to how that actually gets crunched in PostGIS.  We also generate these little spark graphs, which is just the elevation profile.  So you can see here it's quite a simple Postgres function.  And other things like calculating the ascent and descent, the climb.  This is a PostGIS function.  This kind of goes through every step and calculates the difference.  The other thing is map matching.  So there's a lot of features that are relevant to people doing a bush walk or a hike, things  like where there's toilets, where there's lookouts, all kinds of things.  And all of that is mapped in OpenStreetMap or can be mapped.  So we pull out those features that are kind of along that track or very close to the track  and match it back to the track.  So we know what features from OpenStreetMap are on each walk.  So we pull out the SAC scale.  That's a tag that you can add to features in OpenStreetMap, which says, is this like  a kind of a basic walk that people with not much experience can do?  Or is this like a really demanding track where you kind of need to use your hands to climb  over things?  So it's a very broad tag, but it is useful.  There's also a tag called trail visibility.  It's probably not the best tag because it's very hard to kind of separate some of these  levels apart.  There's on the Wiki page, it kind of describes what the different levels are.  But as a broad sense, there is like a difference between excellent and horrible.  So it is helpful when OpenStreetMap has this.  And I personally try to add this every time I add a track.  But anyone else who I see other people adding it, and it is really helpful.  But also things like ladders.  So in OpenStreetMap, you can tag a ladder.  Some people, they might get a bit scared of this.  So we want to be able to know where that is from OpenStreetMap and show on the on the  on tracks that for this walk, yeah, there's a ladder and it's it's like 10 meters higher.  So just warning people in advance.  Things like safety ropes, you can tag them.  And so these are all things that I guess like big mapping providers probably didn't think  of.  They're not it's not there.  It's not what they would typically map.  So that's why I like OpenStreetMap because we can map things that are interesting to  to whatever kind of interest you have.  Rung so to kind of climb up a rock.  Surface information.  So we want to know if it's sand, if it's paved or if it's unpaved because say that affects  how accessible the walk is, how long, how slow is going to be to walk along it?  Like it's a lot slower walking in sand than on concrete.  So yeah, it affects the walk difficulty and time estimate.  So we use all these postures functions to take features from OpenStreetMap, match them  back to the actual path and then work out how far along the walk.  That is so yeah, we can determine how far along the features are.  So to know say for drinking water or toilets, are they so far apart affects how we factor  in the accessibility of a walk.  And also ordering photos.  So we have all these photos they can automatically be ordered based on where they are along the  walk.  And there's a whole bunch of other libraries that we pull in, useful information.  So I mean this walk has a location and from that location we can, there's good open source  libraries to calculate sunrise, sunset times, weather forecasts based on the, so we use  dark sky where you can pass in lat long coordinates and it will give the forecast based on that  position as opposed to kind of the nearest weather station.  We pull in some open data feeds.  So in New South Wales they publish park alerts, so we pull that in as a data feed.  And also bushfire alerts.  So I know for New South Wales ACT they have CCB, open licensed, open data that we pull  in and show.  So where that fire is a certain distance away from the walk would actually show that alert  on the page to say, hey there's an active bushfire near this walk.  So you can see here is an example of here's the walk and we just pop it up there so people  are aware.  People have that information.  It brings, it's really bringing together a lot of information sources and putting them  together.  And just quickly, so we, when you share like a URL on Twitter it will generate this nice  card and we have like this short description here and all that is generated automatically  from the data.  In this case it says it's short so we just classify, determine how long the walk is to  save it so short and medium or long.  We say where it is.  We say what kinds of things you'd likely see.  So that's not manually entered.  That's if we find there's waterfalls or swimming on this walk from the open street map data  then we will populate that.  There's other things we say like if it's near the coastline, so in open stream map you can  enter the coastline then we'll say it's a coastal walk.  If it's a long way from roads or populated settlements we say it's a remote walk.  So all of that is just automatically generated to form this walk description as opposed to  manually entering it.  And yeah like we've implemented search.  So open stream map has a lot of good detailed level of data that other providers don't necessarily  have and that was kind of one of my initial concerns or initial reasons of doing this.  So that hey we have all this great data in open stream that's entered but if I search  it on Google Maps it's not going to show up.  So it's great that we're able to use that.  So I encourage you to go to the site, check it out.  Yeah as I mentioned we're launching a new version next two weeks or so.  I don't know if we have time for questions.  Okay, okay perfect timing.  Yeah, yeah.  Does OSM already support things like does this trail have trail markers?  Yeah so the tag, the trail visibility, the definition of excellent means that there's  a lot of either guideposts to tell you like turn here or there's just markers along the  track so you can kind of map it with that trail visibility tag.  And whether the track is maintained?  Yeah so that's a hard one because like if it's really overgrown typically the trail visibility  you wouldn't score excellent.  It's not really, that's why that tag, yeah it's not the best tag but you can kind of  enter that kind of detail with it.  Yeah, David?  Thanks for talking Andrew.  Have you used or like my whole app story if you're writing something like Street to  complete to get some of the information?  Yeah well I think Street complete is great because people who are going out and walking  it'll just prompt them with hey there's a road here and we don't have surface information  what is it?  So you like you can use that even when you're going outside it's not just for roads.  I know that, some of those people don't want to like help in Street complete.  Okay.  Yeah, yeah so.  It's not something completely different.  Yeah like the Street complete is a mobile app for everyone else that lets you, it's basically  it gives you, it says hey this street that you're walking on we don't know what surface  it is.  Please select from these options.  Is it paved?  Is it unpaved?  Is it sand?  So you can very easily just go out and map and contribute to OpenStreetMap.  And so it's an open source tool and it's good because anyone can submit challenges so for  some of the things like trail visibility they're probably not on Street complete but I do think  it's saying that we should try to do.  Yeah.  Yep, yeah so it's quite easy for everyone to contribute that.  Yeah.  I have a question.  So in Victoria I think Australia there's guidelines on like what different types of  track quality and distance and how hard they are.  Yeah, yeah that's like a strain standard.  Yeah, yeah it's like grade one to five.  They actually know something about trail visibility and marked distance as well that's like logical.  Yeah, yeah so I guess I've modeled it after OpenStreetMap which tries to be kind of work,  it works globally and it can be hard to devise things that work globally.  We haven't like for the on tracks we haven't adopted that as strain standard.  I do think there are some challenges with it and in the way they classify walks because  like I think they say if it's wheelchair accessible it's grade one but you could have, yeah so  I won't go into all the details but it's not something we have done but yeah it's a good  option.  How many tracks do you go to in your data space?  About 380.  But the thing is they're mostly ones that I'm familiar with.  The repository for the walks is open source and as I mentioned like a lot of the time  it's just a personal opinion of yeah take this track and then go this track so we can't  necessarily enter that into OpenStreetMap so hopefully if there are particular walks  that you know of you can submit them in.  Couldn't they just give a name?  Yeah, yeah there could be the name but sometimes it might be take this track which is called  like such and such name and then there's another segment which has a different name in another  segment and it's about joining them together.
Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_0_59.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_60_491.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_492_1883.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_1884_3527.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_3528_4871.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_4872_5339.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_5340_6275.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_6276_7223.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_7224_7295.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_7296_8423.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_8424_11279.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_11280_12959.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_12960_13163.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_13164_16367.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_16368_22403.png Andrew_Harvey__BeyondTracks__an_OpenStreetMap_case_study_Sxi3PmDu6kg_22404_22823.png