Metadata Management for Spatial Data Infrastructures
CC Attribution - NonCommercial - ShareAlike 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Durante, Kim
This presentation will focus on creating geospatial metadata for spatial data infrastructures. The growing emphasis on data management practices in recent years has underscored the need for well-structured metadata to support the preservation and reuse of digital geographic information. Despite its value, creation of geospatial metadata is widely recognized as a complex and labor-intensive process, often creating a barrier to effective identification and evaluation of digital datasets. We will discuss our set of best practices for describing a variety of spatial content types using the ISO Series of Geographic Metadata Standards. We will share a series of Python and XSLT routines, which automate the creation of ISO-compliant metadata for geospatial datasets, web services, and feature catalogs. These auto-generation tools are designed to work directly with XML documents, making them suitable for use within any XML-aware cataloging platform. Our goals are to make metadata creation simpler for data providers, and to increase standardization across organizations in order to increase the potential for metadata sharing and data synchronization among the geospatial community.
 My name is Kim Duranti and I'm the metadata librarian for Geospatial and Scientific Data. I work at Stanford University. Probably unlike or maybe unlike a lot of you guys are what you do since I work in a library or university setting. We're kind of new to GIS maybe compared to some of what you guys are doing maybe over the last decade. There's been extreme growth in the amount of people coming in asking for instruction in GIS and people are now starting to use GIS and their scholarly research and so we've had really a lot of growth. Some of the metadata things I'm going to talk about templates or scripts and XSLT. I obviously don't have time to demonstrate that today so I'll point you to where those documents are at the end of this presentation. It's a little bit about GIS at Stanford. I'm actually physically located in our Earth Sciences Library on campus and this is where a majority of the cartographic research happens. One of my pictures is sliced off there. Sorry. The library work at is primarily responsible for purchasing or downloading most of the GIS data that we then distribute to our users. We also have a physical space. We have a lot of people coming in for geospatial instruction. This is called the Stanford Geospatial Center and then we also have a teaching lab. Like I said, I think we taught maybe like a thousand classes last year and so this has really grown in terms of the kinds of people that are coming in and wanting to use GIS. We are also the maps library for campus. We have everything from general topographic maps and like road maps to rare and historic maps. I don't know if you guys are familiar with David Rumsey. He's a historic map collector. He's actually donated his map collection to Stanford so we'll be opening up a David Rumsey map center in the middle of 2016 and now we have people coming in and they're starting to geo-reference these maps and now they're creating geospatial data out of all of this. So now it's sort of ours to deal with. So at the center of all of this, we've got people coming in asking for data, we're distributing data. It was clear after about ten years of doing this that we needed a better way to not only gather resources in but then a more efficient way to serve those resources out. Historically in the past, people would actually physically have to come to the library to check out like a DVD or something or they would have to email one of us or contact one of us and then we'd have to essentially find data for them. I don't know how much you know about libraries but people don't really come into the desk and ask for help anymore. They Google things. So we needed to kind of move into that space. Outside of geo, libraries deal a lot with digital files. Think about scanned books or digitized photographs, things like that. There's a lot of digital files but they really don't have that much in common otherwise and that is why for a number of years Stanford has been involved in what's called the Hydra Project and this is an open source community-based project. There's a number of institutions involved as well as radio stations or news organizations and it's pretty much anybody that has digital assets that they need to manage and more specifically preserve and uniquely identify within a repository environment. As I said, this is completely open source. It's built using the Ruby language, the Fedora repository, Apache Solar Index as well as the Blacklight Discovery platform. And Blacklight, I don't know if you guys have heard of this, it was developed as an alternative to a lot of library catalog software out there that's commercially purchased. So this gives us, our library catalog for instance is built using Blacklight. We have several other interfaces that are also built off of Blacklight where we can sort of scope customizable interfaces for discovery of different types of content. Given all of this and the fact that this was already existing, so I should say one more thing, everything that comes in and out of the repository is assigned a persistent URL as well as a persistent URL page, meaning that if you go to this URL down there, we're guaranteeing you that at least at that identifier level that those data will always be associated with that universally unique alphanumeric string. Right now if you're accessing it through HTTP, you can get at it through that protocol but within our repository every piece of data is identified by this unique identifier. And so this is more than just a landing page for data. There's some descriptive information here but it also allows you to cite that data. So if someone had written an article and they use this data, they could say, hey, I reference this. If someone then takes these data and uses it for another project, it's going to be siteable and always available at this particular URL. These metadata right here is stored in a descriptive standard called mods, which I'm not sure anyone has ever heard of that. This is a digital library standard. It's really quite common and everything that flows in and out of our repository gets what's called a mods record. So given the fact that we already had this repository in place and we already have these Perl services for data, it made sense for us to adapt the existing black light platform to accommodate geospatial resources. And this led to the development of what we call geoblacklight. We have branded that software in the front end locally at Stanford as a product called Earthworks. So you can go to earthworks.stanford.edu and see that in action. I wanted to give you a live demonstration but I don't think that's a very good idea. So feel free to browse around. One of the benefits of this catalog is it actually aggregates metadata from several institutions. They're mostly universities right now. One that has metadata we can take it in, we'll distill it down and we'll make it searchable in geoblacklight. You can also do a map search and then we employ faceted searches in this interface as well. Just to give you an example of one of the resources that somebody might find in here, this is sort of a descriptive page for the metadata. Really basic stuff. You can go view the Perl page for this item if you want to at the Perl. You could click on one of the data points there and start to view the attribute values that are associated with that. You can download the shape file and do other sort of library-ish things like bookmark it, email it to yourself, that kind of thing. Those were all pulled from the existing blacklight software. So it was really just a plug-in that we had to adapt to get this to happen. To get to this point, we've really had to, from a metadata standpoint anyway, move from storing ISO metadata. We then have to move that to the mods which I just mentioned to support the Perl functionality and then finally we distill that down into what we call the geoblacklight schema. That is a set of JSON solar documents that make it available for someone to search and find on the web. So it's pretty much just like a Dublin Core standard with some geo-fields tacked onto it. It's basically just for end-user discovery of geospatial data within a web interface. To make the metadata for these is extremely hard, I should say. It kind of looks easy and maybe the examples I'll put up here make it look somewhat easy, but it's actually kind of a pain. When we usually begin creating metadata for this, we start from a collection level and that's because when we buy things, it used to maybe come on a DVD or layers are served out together thematically usually. So we register what's called a collection object into the repository. We create some metadata for it in that mod standard that I mentioned earlier and then we register it according to a rights policy. So this could be something like we carry a lot of restricted data because we purchased them. It could be public domain data. We have those as well as a variance of creative comments licenses that we've set up. Once we register that into the repository, it gets one of those unique identifiers. The layer level workflow is a little more complex and it can certainly vary depending on the type of metadata that you're dealing with. Often you have to kind of handle one layer at a time. We're trying to speed this up as much as possible. The first thing you have to do before you even think about putting a layer into a repository for preservation purposes is to check the data properties. Projections. I can't really say enough about how missing or incorrect projections cause infinite problems downstream. It's probably the biggest issue we've had in terms of trying to expose something on the web. Everyone's nodding here, so that's great. I think I have to really encourage you if you take anything away from this. If you guys are creating data, please use a standard projection. Anyway, once we have the data properties, things like file names are also kind of weird, bad characters and spaces in file names or rarely we'll get objects where the data are extending beyond the side of the box. We maybe have to clip it down to the data frame. Once we've got the data properties in place, we can then actually register that layer according to the collection that we've already created. We can then assign a rights policy to it, which is usually the same as the collection, but sometimes layers differ in terms of their rights capability. You may want to change that. Then there's two scenarios that everything can fall into. If you're dealing with items that have no metadata, usually start out with some kind of XML template. It's very controlled. You set up your fields. You can import that in. That generally works pretty well. We also get a lot of metadata that already has all the stuff in it. That's not something that we want to just erase and rewrite. That process, we use an XSLT to normalize the metadata. Then we both steps move to a localized editing. This is where you would have to go in. This is what nobody does. This is where you have to go in and actually edit fields that are specific to that layer, things like just describing what it's about, defining the attributes, maybe a general title that's preferably not the file name. The next thing we do is then auto-generate a bunch of metadata using Python. This is usually scripting in all these identifiers we write. This is not information that you want to be typing in by hand. Then finally, once that's done, we deposit that into a staging area. From there, it's picked up by a series of robotic workflows, one of which will package it up for a session into the repository. The other one will package it up for delivery to the user upon request. I said a little bit about identifiers. I just want to talk about those a little more because I think they're great. It's really probably the single most important piece of metadata that you could have in a record. Everything that we call it a digital repository unique identifier or a druid. At the layer level, you've got this alphanumeric string. You've got its URL. We also create metadata file identifiers according to this. There's a newer convention with assigning file identifiers where you use your string at the end and then everything is in that Stanford namespace. That number is guaranteed to be unique within the edu.stanford.pearl. Actually, URLs can appear several times within a record. It's nice to have a scripting effect where you can go write those in where you don't have to type a URL into several fields. Collection level metadata, same thing. You've got a druid. Then that fills in two fields for the ISO 19139. The collective title as well as the identifier for the parent metadata. Now, the fun stuff. This is probably pretty easy, but for the layer I showed earlier about geothermal power, when we start with a collection of these, we had a whole set of data that dealt with renewable energy. Right away, there's a bunch of metadata that we can put into a template that we don't have to enter by hand. Between the identifiers and this, we've already got a pretty good record shaping up in terms of creating a nice ISO record. I would say creating XML templates as opposed to normalizing something with XSLT is really a lot easier. You have much more control over what you're putting into your system. This is probably a pretty straightforward example of how you would make a template. I didn't want to put XML up here, but all these fields go into every single record. Make a template. You could either apply it if you're doing geo-network or if you're using a system that imports it into the XML, either way it works. Localized editing, again, this is a pretty straightforward example, but when you're making titles for things, you want to have sort of a what, where, when, specifically the when. I notice a lot temporal extents are left out of data. If you have something that says average annual rainfall, but there's no dates, that doesn't mean anything. What is it? Where is it? What does the data cover as a very simple pattern for titles? Again, description is really straightforward here, but just what is the data? What does it represent? And then keywords generally differ. We're slightly constrained as a library because we have to follow these hard and fast rules about controlled vocabulary. We pull keywords from the Library of Congress, or we pull names for researchers from either something called VIF or the Library of Congress. I think we rely a lot more on controlled names than maybe some other institutions. So when we pull in data, we have to make sure that those correspond to an existing vocabulary. And again, temporal extents, I can't say enough about those. Updating existing metadata, like I said, is really can be kind of tough, this varies significantly by collection. The reason why we've chosen to do things this way, specifically when you get data that have very complex lineage information associated with them, or if someone's filled out attribute definitions, we don't want to erase that. That's something that I wouldn't know just by looking at the data. And so in order to preserve that, we run an XSLT on those metadata, and it essentially works sort of like a template in reverse where it applies all those normalized elements to the XML. To do this, takes a little bit more time because you have to survey all the XML that you're looking at before you can really write an XSLT that works cleanly with your metadata. And then you have to go in just like you would with the other process and edit things locally. There's things like title, abstract, keywords, and so forth. And then our final step, we've gotten a lot more used to using Python to script as many fields as possible. This is because more and more as URIs are appearing in metadata, no one wants to be copying and pasting or typing in these types of information. So we have some scripts in Python that insert all of these into the record without us having to touch them. So after looking at this, there's really only a few fields that you guys have to go in and enter in order to have a really nice record, and everything else hopefully will be automated for you. You can also use Python to add things like credit statements if you're supplying those by pulling variables out of the records like authors and titles and dates. If you want to add file names in like a web services link, you can also use something like that for this as opposed to going in and typing the name of the file into the linkage section. Let's see. I'll feature catalog metadata. So this was one of the probably the most difficult things to deal with, I think, when you're looking at data because you can open up an attribute table, you have no idea what it means. Labels like city and country, I think we can all pretty much infer what those mean or take a really good guess, but with things like demographic data or household information, all that kind of stuff, we get these code lists like this, and we have no idea what that means. The data is pretty much essentially useless at this point, people doing some kind of analysis. So we have enforced people who are giving these giant code books for their attribute tables. We are now requesting that in a CSV. When we do the same thing, we've created Python routines that script the entire ISO 19110 record from a CSV file because this is most often how we receive those metadata. I don't think anyone's ever gone into a form and actually filled out the definitions for their attribute labels, but CSV, I think, is an easier way for researchers or people providing data to understand a code book. And so this has worked pretty well for us. So after we've created our metadata, it kicks off into this robotic workflow that separates the data and the metadata. For the metadata piece on the right-hand side, everything starts out, like I said, with the ISO 19139. We crosswalk that into a mod's record to support the Perl functionality. We do a little thing that generates some URIs in the place names in the attributes, and then it's finished. It kicks off a discovery workflow. There's some data wrangling that has to happen here, and that sort of relates to the data properties issue I said before, where the robot checks to make sure that all the data properties are valid. After that, they are joined together, and they are kicked off into a discovery workflow where they can then be made available for people in the front end. So after we have our mods metadata, we generate metadata in the GeoBlackLite schema in JSON. We load that GeoJSON into a solar index, and we pump it out to the Earthworks catalog, and this is where people can go and search for the data. We then export the data to GitHub, and I'll show you that in just a second. And that workflow is kicked off. And this might actually be my last slide. I'm probably under time here, but I can talk about this. When I first started having to make metadata for a bunch of resources, I was just googling things all the time, like googling file names or emailing this guy at Harvard and asking him to email me XML, and it just wasn't working. And so we have started to deposit all of our metadata on GitHub in something called Open GeoMetadata. I think there's maybe about 10 institutions here right now. You can store metadata in any format you want. Just put it up there. And then what we do is we gather these from all the other institutions, and we pull them into our Earthworks catalog that way. What's nice about this is that I no longer have to write someone to ask them if there's a record. I can search for the title on Google, and it leads me right to this. Or I can search for it right in here. And so I think there's about 25,000 or 30,000 records in here right now. And for somebody who has to make metadata all day, this has been a huge help because I can't tell you how much time it's saved. And so we're starting to see more people sign on to this. There's also some tools that are built around it. So if you want to, the tool Geo Combine will actually pull, help you pull in and use style sheets to transform your metadata and bring it in. I think that's actually about it. If you want more information about Hydra, Geo Blacklight, OpenGeo Metadata. As I mentioned, we've got some templates in the XSLT and the Python I talked about. It's all up on GitHub. I don't want to demonstrate that live. And then if you want to email me or follow me on Twitter, and I want to just say thanks to all my contributors because this is a pretty large project. Any questions? Yeah? So when you get things like metadata coming in, when you have maybe one piece has lat longs and one piece has meters maybe, do you ever have to project anything between the different types of things you get in? Yeah, what happens is we take, because it's a preservation environment, we take the data in whatever, even if it's custom, we take it with whatever projection it's come in. And then everything is transformed into 4326 for the web front end. So that's why if there's a standard projection, I think the GDAL tool can read those. But when it's this custom stuff, that's when we see problems downstream. Yeah, that's caught my ear. What is a standard projection for you? A standard projection is right now anything that's registered in the EPSG. Okay, and another question would be, maybe I didn't understand it correctly, but have you considered, you know, I come from the GIS world, from Europe, so we have inspired. And inspire initiative is much about metadata and catalog services. Have you somehow evaluated this option as well? We have, and we actually looked at Geo Network, I think it's a great product. I think because we had already had all this technology existing, that it just makes sense for us to adapt it that way. If there's upgrades or changes, it's part of our technology stack. If we are running something like Inspire and all those other tools, it requires almost a completely separate set of personnel sometimes to manage and upgrades happen at different times. Now, if Hydra is updated or Fedora or Blacklight, Earthworks gets, you know, it's part of the package and so we don't have these external applications running that change. That was, that's just sort of what made sense for us. From the metadata standpoint, though, I have to say I don't think it matters because if we follow the standard, we can share it with whomever and put it into whatever system we like. I have one last question. You were speaking about georeferencing or the existing old data, datasets. Have you heard about the old maps online project, which is kind of crowd georeferencing? Yeah. Is it the New York Public Library? I doubt so. Okay. You know, it's not. Okay. There are some, some we've looked at and some others we've had people have questions. We've seen it happen, but the issue I think with some of those georeferencers and I don't know about old maps online is there's actually no file created. It's just data. So we're not getting a geotiff in receipt. I know the British Museum, I don't want to call that, I'm not sure exactly, but someone from the British Museum had a bunch of geodata, but there were no files to be extracted. It was just data at that point. And so we need to have some in some format. Is that it? Yes.
 My name is Kim Duranti and I'm the metadata librarian for Geospatial and Scientific Data.  I work at Stanford University.  Probably unlike or maybe unlike a lot of you guys are what you do since I work in a library  or university setting.  We're kind of new to GIS maybe compared to some of what you guys are doing maybe over  the last decade.  There's been extreme growth in the amount of people coming in asking for instruction in  GIS and people are now starting to use GIS and their scholarly research and so we've  had really a lot of growth.  Some of the metadata things I'm going to talk about templates or scripts and XSLT.  I obviously don't have time to demonstrate that today so I'll point you to where those  documents are at the end of this presentation.  It's a little bit about GIS at Stanford.  I'm actually physically located in our Earth Sciences Library on campus and this is where  a majority of the cartographic research happens.  One of my pictures is sliced off there.  Sorry.  The library work at is primarily responsible for purchasing or downloading most of the  GIS data that we then distribute to our users.  We also have a physical space.  We have a lot of people coming in for geospatial instruction.  This is called the Stanford Geospatial Center and then we also have a teaching lab.  Like I said, I think we taught maybe like a thousand classes last year and so this has  really grown in terms of the kinds of people that are coming in and wanting to use GIS.  We are also the maps library for campus.  We have everything from general topographic maps and like road maps to rare and historic  maps.  I don't know if you guys are familiar with David Rumsey.  He's a historic map collector.  He's actually donated his map collection to Stanford so we'll be opening up a David  Rumsey map center in the middle of 2016 and now we have people coming in and they're starting  to geo-reference these maps and now they're creating geospatial data out of all of this.  So now it's sort of ours to deal with.  So at the center of all of this, we've got people coming in asking for data, we're distributing  data.  It was clear after about ten years of doing this that we needed a better way to not only  gather resources in but then a more efficient way to serve those resources out.  Historically in the past, people would actually physically have to come to the library to  check out like a DVD or something or they would have to email one of us or contact one  of us and then we'd have to essentially find data for them.  I don't know how much you know about libraries but people don't really come into the desk  and ask for help anymore.  They Google things.  So we needed to kind of move into that space.  Outside of geo, libraries deal a lot with digital files.  Think about scanned books or digitized photographs, things like that.  There's a lot of digital files but they really don't have that much in common otherwise and  that is why for a number of years Stanford has been involved in what's called the Hydra  Project and this is an open source community-based project.  There's a number of institutions involved as well as radio stations or news organizations  and it's pretty much anybody that has digital assets that they need to manage and more specifically  preserve and uniquely identify within a repository environment.  As I said, this is completely open source.  It's built using the Ruby language, the Fedora repository, Apache Solar Index as well as  the Blacklight Discovery platform.  And Blacklight, I don't know if you guys have heard of this, it was developed as an alternative  to a lot of library catalog software out there that's commercially purchased.  So this gives us, our library catalog for instance is built using Blacklight.  We have several other interfaces that are also built off of Blacklight where we can  sort of scope customizable interfaces for discovery of different types of content.  Given all of this and the fact that this was already existing, so I should say one more  thing, everything that comes in and out of the repository is assigned a persistent URL  as well as a persistent URL page, meaning that if you go to this URL down there, we're  guaranteeing you that at least at that identifier level that those data will always be associated  with that universally unique alphanumeric string.  Right now if you're accessing it through HTTP, you can get at it through that protocol but  within our repository every piece of data is identified by this unique identifier.  And so this is more than just a landing page for data.  There's some descriptive information here but it also allows you to cite that data.  So if someone had written an article and they use this data, they could say, hey, I reference  this.  If someone then takes these data and uses it for another project, it's going to be  siteable and always available at this particular URL.  These metadata right here is stored in a descriptive standard called mods, which I'm not sure anyone  has ever heard of that.  This is a digital library standard.  It's really quite common and everything that flows in and out of our repository gets what's  called a mods record.  So given the fact that we already had this repository in place and we already have these  Perl services for data, it made sense for us to adapt the existing black light platform  to accommodate geospatial resources.  And this led to the development of what we call geoblacklight.  We have branded that software in the front end locally at Stanford as a product called  Earthworks.  So you can go to earthworks.stanford.edu and see that in action.  I wanted to give you a live demonstration but I don't think that's a very good idea.  So feel free to browse around.  One of the benefits of this catalog is it actually aggregates metadata from several  institutions.  They're mostly universities right now.  One that has metadata we can take it in, we'll distill it down and we'll make it searchable  in geoblacklight.  You can also do a map search and then we employ faceted searches in this interface as well.  Just to give you an example of one of the resources that somebody might find in here,  this is sort of a descriptive page for the metadata.  Really basic stuff.  You can go view the Perl page for this item if you want to at the Perl.  You could click on one of the data points there and start to view the attribute values  that are associated with that.  You can download the shape file and do other sort of library-ish things like bookmark it,  email it to yourself, that kind of thing.  Those were all pulled from the existing blacklight software.  So it was really just a plug-in that we had to adapt to get this to happen.  To get to this point, we've really had to, from a metadata standpoint anyway, move from  storing ISO metadata.  We then have to move that to the mods which I just mentioned to support the Perl functionality  and then finally we distill that down into what we call the geoblacklight schema.  That is a set of JSON solar documents that make it available for someone to search and  find on the web.  So it's pretty much just like a Dublin Core standard with some geo-fields tacked onto  it.  It's basically just for end-user discovery of geospatial data within a web interface.  To make the metadata for these is extremely hard, I should say.  It kind of looks easy and maybe the examples I'll put up here make it look somewhat easy,  but it's actually kind of a pain.  When we usually begin creating metadata for this, we start from a collection level and  that's because when we buy things, it used to maybe come on a DVD or layers are served  out together thematically usually.  So we register what's called a collection object into the repository.  We create some metadata for it in that mod standard that I mentioned earlier and then  we register it according to a rights policy.  So this could be something like we carry a lot of restricted data because we purchased  them.  It could be public domain data.  We have those as well as a variance of creative comments licenses that we've set up.  Once we register that into the repository, it gets one of those unique identifiers.  The layer level workflow is a little more complex and it can certainly vary depending  on the type of metadata that you're dealing with.  Often you have to kind of handle one layer at a time.  We're trying to speed this up as much as possible.  The first thing you have to do before you even think about putting a layer into a repository  for preservation purposes is to check the data properties.  Projections.  I can't really say enough about how missing or incorrect projections cause infinite problems  downstream.  It's probably the biggest issue we've had in terms of trying to expose something on  the web.  Everyone's nodding here, so that's great.  I think I have to really encourage you if you take anything away from this.  If you guys are creating data, please use a standard projection.  Anyway, once we have the data properties, things like file names are also kind of weird,  bad characters and spaces in file names or rarely we'll get objects where the data are  extending beyond the side of the box.  We maybe have to clip it down to the data frame.  Once we've got the data properties in place, we can then actually register that layer according  to the collection that we've already created.  We can then assign a rights policy to it, which is usually the same as the collection,  but sometimes layers differ in terms of their rights capability.  You may want to change that.  Then there's two scenarios that everything can fall into.  If you're dealing with items that have no metadata, usually start out with some kind  of XML template.  It's very controlled.  You set up your fields.  You can import that in.  That generally works pretty well.  We also get a lot of metadata that already has all the stuff in it.  That's not something that we want to just erase and rewrite.  That process, we use an XSLT to normalize the metadata.  Then we both steps move to a localized editing.  This is where you would have to go in.  This is what nobody does.  This is where you have to go in and actually edit fields that are specific to that layer,  things like just describing what it's about, defining the attributes, maybe a general title  that's preferably not the file name.  The next thing we do is then auto-generate a bunch of metadata using Python.  This is usually scripting in all these identifiers we write.  This is not information that you want to be typing in by hand.  Then finally, once that's done, we deposit that into a staging area.  From there, it's picked up by a series of robotic workflows, one of which will package  it up for a session into the repository.  The other one will package it up for delivery to the user upon request.  I said a little bit about identifiers.  I just want to talk about those a little more because I think they're great.  It's really probably the single most important piece of metadata that you could have in a  record.  Everything that we call it a digital repository unique identifier or a druid.  At the layer level, you've got this alphanumeric string.  You've got its URL.  We also create metadata file identifiers according to this.  There's a newer convention with assigning file identifiers where you use your string  at the end and then everything is in that Stanford namespace.  That number is guaranteed to be unique within the edu.stanford.pearl.  Actually, URLs can appear several times within a record.  It's nice to have a scripting effect where you can go write those in where you don't  have to type a URL into several fields.  Collection level metadata, same thing.  You've got a druid.  Then that fills in two fields for the ISO 19139.  The collective title as well as the identifier for the parent metadata.  Now, the fun stuff.  This is probably pretty easy, but for the layer I showed earlier about geothermal power, when  we start with a collection of these, we had a whole set of data that dealt with renewable  energy.  Right away, there's a bunch of metadata that we can put into a template that we don't have  to enter by hand.  Between the identifiers and this, we've already got a pretty good record shaping up in terms  of creating a nice ISO record.  I would say creating XML templates as opposed to normalizing something with XSLT is really  a lot easier.  You have much more control over what you're putting into your system.  This is probably a pretty straightforward example of how you would make a template.  I didn't want to put XML up here, but all these fields go into every single record.  Make a template.  You could either apply it if you're doing geo-network or if you're using a system that  imports it into the XML, either way it works.  Localized editing, again, this is a pretty straightforward example, but when you're making  titles for things, you want to have sort of a what, where, when, specifically the when.  I notice a lot temporal extents are left out of data.  If you have something that says average annual rainfall, but there's no dates, that doesn't  mean anything.  What is it?  Where is it?  What does the data cover as a very simple pattern for titles?  Again, description is really straightforward here, but just what is the data?  What does it represent?  And then keywords generally differ.  We're slightly constrained as a library because we have to follow these hard and fast rules  about controlled vocabulary.  We pull keywords from the Library of Congress, or we pull names for researchers from either  something called VIF or the Library of Congress.  I think we rely a lot more on controlled names than maybe some other institutions.  So when we pull in data, we have to make sure that those correspond to an existing vocabulary.  And again, temporal extents, I can't say enough about those.  Updating existing metadata, like I said, is really can be kind of tough, this varies  significantly by collection.  The reason why we've chosen to do things this way, specifically when you get data that  have very complex lineage information associated with them, or if someone's filled out attribute  definitions, we don't want to erase that.  That's something that I wouldn't know just by looking at the data.  And so in order to preserve that, we run an XSLT on those metadata, and it essentially  works sort of like a template in reverse where it applies all those normalized elements to  the XML.  To do this, takes a little bit more time because you have to survey all the XML that you're  looking at before you can really write an XSLT that works cleanly with your metadata.  And then you have to go in just like you would with the other process and edit things locally.  There's things like title, abstract, keywords, and so forth.  And then our final step, we've gotten a lot more used to using Python to script as many  fields as possible.  This is because more and more as URIs are appearing in metadata, no one wants to be copying  and pasting or typing in these types of information.  So we have some scripts in Python that insert all of these into the record without us having  to touch them.  So after looking at this, there's really only a few fields that you guys have to go in and  enter in order to have a really nice record, and everything else hopefully will be automated  for you.  You can also use Python to add things like credit statements if you're supplying those  by pulling variables out of the records like authors and titles and dates.  If you want to add file names in like a web services link, you can also use something  like that for this as opposed to going in and typing the name of the file into the linkage  section.  Let's see.  I'll feature catalog metadata.  So this was one of the probably the most difficult things to deal with, I think, when you're  looking at data because you can open up an attribute table, you have no idea what it  means.  Labels like city and country, I think we can all pretty much infer what those mean or  take a really good guess, but with things like demographic data or household information,  all that kind of stuff, we get these code lists like this, and we have no idea what  that means.  The data is pretty much essentially useless at this point, people doing some kind of  analysis.  So we have enforced people who are giving these giant code books for their attribute  tables.  We are now requesting that in a CSV.  When we do the same thing, we've created Python routines that script the entire ISO 19110  record from a CSV file because this is most often how we receive those metadata.  I don't think anyone's ever gone into a form and actually filled out the definitions for  their attribute labels, but CSV, I think, is an easier way for researchers or people  providing data to understand a code book.  And so this has worked pretty well for us.  So after we've created our metadata, it kicks off into this robotic workflow that separates  the data and the metadata.  For the metadata piece on the right-hand side, everything starts out, like I said, with the  ISO 19139.  We crosswalk that into a mod's record to support the Perl functionality.  We do a little thing that generates some URIs in the place names in the attributes, and  then it's finished.  It kicks off a discovery workflow.  There's some data wrangling that has to happen here, and that sort of relates to the data  properties issue I said before, where the robot checks to make sure that all the data  properties are valid.  After that, they are joined together, and they are kicked off into a discovery workflow  where they can then be made available for people in the front end.  So after we have our mods metadata, we generate metadata in the GeoBlackLite schema in JSON.  We load that GeoJSON into a solar index, and we pump it out to the Earthworks catalog,  and this is where people can go and search for the data.  We then export the data to GitHub, and I'll show you that in just a second.  And that workflow is kicked off.  And this might actually be my last slide.  I'm probably under time here, but I can talk about this.  When I first started having to make metadata for a bunch of resources, I was just googling  things all the time, like googling file names or emailing this guy at Harvard and asking  him to email me XML, and it just wasn't working.  And so we have started to deposit all of our metadata on GitHub in something called Open  GeoMetadata.  I think there's maybe about 10 institutions here right now.  You can store metadata in any format you want.  Just put it up there.  And then what we do is we gather these from all the other institutions, and we pull them  into our Earthworks catalog that way.  What's nice about this is that I no longer have to write someone to ask them if there's  a record.  I can search for the title on Google, and it leads me right to this.  Or I can search for it right in here.  And so I think there's about 25,000 or 30,000 records in here right now.  And for somebody who has to make metadata all day, this has been a huge help because  I can't tell you how much time it's saved.  And so we're starting to see more people sign on to this.  There's also some tools that are built around it.  So if you want to, the tool Geo Combine will actually pull, help you pull in and use style  sheets to transform your metadata and bring it in.  I think that's actually about it.  If you want more information about Hydra, Geo Blacklight, OpenGeo Metadata.  As I mentioned, we've got some templates in the XSLT and the Python I talked about.  It's all up on GitHub.  I don't want to demonstrate that live.  And then if you want to email me or follow me on Twitter, and I want to just say thanks  to all my contributors because this is a pretty large project.  Any questions?  Yeah?  So when you get things like metadata coming in, when you have maybe one piece has lat  longs and one piece has meters maybe, do you ever have to project anything between the  different types of things you get in?  Yeah, what happens is we take, because it's a preservation environment, we take the data  in whatever, even if it's custom, we take it with whatever projection it's come in.  And then everything is transformed into 4326 for the web front end.  So that's why if there's a standard projection, I think the GDAL tool can read those.  But when it's this custom stuff, that's when we see problems downstream.  Yeah, that's caught my ear.  What is a standard projection for you?  A standard projection is right now anything that's registered in the EPSG.  Okay, and another question would be, maybe I didn't understand it correctly, but have  you considered, you know, I come from the GIS world, from Europe, so we have inspired.  And inspire initiative is much about metadata and catalog services.  Have you somehow evaluated this option as well?  We have, and we actually looked at Geo Network, I think it's a great product.  I think because we had already had all this technology existing, that it just makes sense  for us to adapt it that way.  If there's upgrades or changes, it's part of our technology stack.  If we are running something like Inspire and all those other tools, it requires almost  a completely separate set of personnel sometimes to manage and upgrades happen at different  times.  Now, if Hydra is updated or Fedora or Blacklight, Earthworks gets, you know, it's part of the  package and so we don't have these external applications running that change.  That was, that's just sort of what made sense for us.  From the metadata standpoint, though, I have to say I don't think it matters because if  we follow the standard, we can share it with whomever and put it into whatever system we  like.  I have one last question.  You were speaking about georeferencing or the existing old data, datasets.  Have you heard about the old maps online project, which is kind of crowd georeferencing?  Yeah.  Is it the New York Public Library?  I doubt so.  Okay.  You know, it's not.  Okay.  There are some, some we've looked at and some others we've had people have questions.  We've seen it happen, but the issue I think with some of those georeferencers and I don't  know about old maps online is there's actually no file created.  It's just data.  So we're not getting a geotiff in receipt.  I know the British Museum, I don't want to call that, I'm not sure exactly, but someone  from the British Museum had a bunch of geodata, but there were no files to be extracted.  It was just data at that point.  And so we need to have some in some format.  Is that it?  Yes.
PT3-16_0_95.png PT3-16_96_167.png PT3-16_168_755.png PT3-16_756_923.png PT3-16_924_971.png PT3-16_972_1067.png PT3-16_1068_1451.png PT3-16_1452_3599.png PT3-16_3600_5555.png PT3-16_5556_6419.png PT3-16_6420_7307.png PT3-16_7308_8687.png PT3-16_8688_10367.png PT3-16_10368_11687.png PT3-16_11688_15431.png PT3-16_15432_16835.png PT3-16_16836_18167.png PT3-16_18168_19799.png PT3-16_19800_21107.png PT3-16_21108_22391.png PT3-16_22392_23243.png PT3-16_23244_24179.png PT3-16_24180_25187.png PT3-16_25188_25931.png PT3-16_25932_27851.png PT3-16_27852_28655.png PT3-16_28656_28775.png PT3-16_28776_29495.png PT3-16_29496_29543.png PT3-16_29544_30155.png PT3-16_30156_30239.png PT3-16_30240_30323.png PT3-16_30324_30371.png PT3-16_30372_32939.png PT3-16_32940_33047.png PT3-16_33048_33107.png PT3-16_33108_33154.png