The Keyword Aggregator
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Benn, David
A recording of a presentation from David Benn (CSIRO) for the September 2017 AVSIG meeting. David discusses the Keyword Aggregator Web Service, a tool for managing digital objects’ keywords.
 Before we actually go through the slides, I just want to give you a little bit of background. So as Simon mentioned, in Sorrow Scientific Computing and the applications team now, I was in the data processing team at the time that Simon and I were working on this project. So basically the Keyword Aggregator really is in summary a web service, a web application and web service that allows search, keyword search across managed vocabularies, but also vocabularies, the vocabularies can include user defined terms and I'll demonstrate how that works and discuss the motivations. So apart from the application and the web service, what we've done is to provide a demonstration widget, jQuery based widget that you can use to pretty much as is, but it's really just intended to demonstrate how you might go about using the web service. So this work was actually originally carried out in the context of any research project and as Simon said, NICCAR is really the brains behind this. I'm more or less the code monkey who happens to understand enough about the domain to be able to work in it and as Simon said, he and also Jonathan, you provided some invaluable input at the beginning of the project in particular. So a lot of the work that was done towards this was leading up to the ModSIM 2015 conference and there was a hiatus of about a year and then in the last half year we've started to do some more work in the context of the data management capability enhancement program with the view to potentially looking at the use of this system in the Data Access portal, but that's to be determined yet. It's really just to try and understand whether we could get to the point where it was mature enough to think about using it in that sort of context and I'm going to provide a motivation for the creation of this thing in the first place in the context of the Data Access portal. There is a data collection, a DAP collection which I'll point to that you can go to to actually get the code. It's a slightly earlier version than where we are up to now and needs to be updated, but there's also a bunch of links to publications in relation to the Keywood aggregator and just general technical information about it. So with that I'll just launch into the slides and there's only a handful of them. What I want to do then after having motivated while we even want to do this is to then give a short demonstration of the aggregator and hopefully that will lead to some questions and conversation. Okay, so we're all familiar with the idea of needing keywords of some sort in publications whether we're talking about data publications or software publications or journal papers, conference papers, nothing new there. Thinking about DAP in particular, you will come across a couple of different sorts of keywords. You can think of the fields of research as essentially a control vocabulary, so this is from the ANZSRC, Fields of Research. David are you showing a full screen because I'm just stuck on one slide with the title publication keywords. Is that what you are trying to show? Do you currently see one that says about this data collection and then fields of research and keywords? No, seeing the Janolan CAFES one and it's not moving up. Now it's moved to publication keywords, now it's moved on down. Okay, that's interesting. Alright, well is that readable for everyone? Because I was in full screen mode because if it is I'll just stick with this. Is that alright? Yep, okay. So, yeah, as I was saying what we see here are two different sorts of keywords, the fields of research and free text entry keywords basically. So in terms of how you would select those within the DAP, you have essentially quite controlled drop-down boxes that let you select particular fields of research and that's pretty much the end of it. There's no sort of wiggle room. Contrast that with free entry keywords, that's pretty much as it appears. You type in the text, you separate it with semicolons and that's essentially how it appears in the final collection. So one's very controlled and one is completely freeform essentially. What would be nice, now actually what happens if I try to, if I show this can you guys see any animation? No, David. Okay, fine, thank you. I'll just leave it like that then. So if you think about those keywords that I just showed you, it would be nice instead of just typing in 3D to get some sort of guidance as to some possible keywords that might relate to 3D concepts. Same thing for LIDAR and SLAM and mapping and caves and whatnot. And so what this is intended to show, and the animation shows it a bit more clearly, but what this is intended to show is just the idea that if you type 3D you might get a list of things that might make you think about something more specific to enter rather than just 3D, likewise with mapping and SLAM and LIDAR and whatnot. But the question mark there above, Janowan Caves and Zebedee, is just to point out that they're always going to be, no matter what vast set of vocabularies or keywords, resources you might have, there's always going to be this situation where there will be keywords that just don't fit or that aren't catered for. So how do you deal with that basically? This is where we come across the idea of folksonomies. And so the idea there is that you might enter the word Zebedee and then essentially get no results and be prompted for description so that you can then add that or include that in future searches essentially. So this is just showing from left to right and then down and across to back to the left again how that might happen. And that likewise for Janowan Caves there might not be anything specific for that so you can enter that. But the question then becomes well if you enter those sorts of folksonomy based vocabulary keywords, if someone searches for them in the future should they appear at the top of the list, should they appear at the bottom of the list? And I guess the idea is that since they're not based upon a formal vocabulary that a group has decided or agreed upon that they should be last in the list of search results. So in summary what we've done is as I said created this web application web service that allows search across vocabularies including a folksonomy. And these are essentially isolated in named graphs, the idea of named graphs and the idea is that you may will have conflicts between them but you need to still be able to allow for the possibility of all these vocabularies living together happily. And then as I said we exposed it for a demonstration with it, widget. I'll also show you what's happening behind in terms of a regular web service response that you would get in terms of JSON. And then just briefly the architecture is just to say we've got a web service that uses particular implementation frameworks and we have a database for storing certain statistics and then a Spark LendPoint that talks to an RDF based triple store which has scos vocabularies. Alright so before I move on to a demonstration any questions or should I just go straight to the demonstration? Hi David, Nick here, I've just appeared. Hi Nick, how are you? That's good, you can deal with any Nali questions. Go to the demonstration, thank you David and then it becomes real for everyone of course. Very good. Okay so this is the web service, sorry this is the keyword aggregator top level page. Now I'm showing here instance of a containerized instance that's based on Docker. This was one of the things that we did in the work earlier this year. We talked about it a couple of years ago but didn't actually end up doing that. So I'm happy to talk about that if there's any interest. So what you see at the top level here is the ability to get to a widget, the widget that I mentioned, some information about the vocabularies that are in this particular instance of the keyword aggregator and I'll link to an example web service API. Cool. So what I want to do is talk about the widget first because that's really I guess the simplest way of seeing what happens here. That was an interesting noise. Okay so if I type fish for example I'll get a list and the number of results that you get back from a search is configurable. And yeah so essentially we can say alright well I've got to choose one of these and then if I start trying to use some of the examples that I had in the, okay there's nothing another vocabularies has that right now. So mapping for example. But decide that. David you're still showing the PowerPoint. Really? I think you might be showing your wrong screen David. Yeah I am indeed sorry I haven't used too much. Okay so I need you to just find that. Thanks for that. Actually I'm just going to share the screen. It might be easier. That's what I should have done. Okay can you see the web page now? Just waiting. No still seeing your PowerPoint control panel. Okay I'm just going to stop and reshare again. That's now we're seeing a web page. Okay sorry about that guys. So yes so again just very brief ago back so this is the top level page. I'm going to focus initially on the widget. And so if I type in a term here such as map similar to sort of to the sort of example we saw in the slides. I might decide that digital soil mapping is the one that I want. And then looking at things like LiDAR. So again the initial it may still make sense to just say no I just want to talk about LiDAR in general. But someone seeing a set of results of this might determine that actually is better to have something more specific for the keyword. SLAM was another one we were looking at. And this is good because you know the person trying out the keyword here might decide that actually the audience, the people who may be looking at the data collection might be better served by having not an acronym but the acronym expanded and so on. So we can keep doing this and obviously we get to the point where we start looking at caves and okay there was in particular the Genolan caves was one of the keywords that was entered by the depositors. And you wouldn't expect really to see a specific keyword in general vocabulary for that. So we now see the ad link here. So if I click the ad link I can enter a description such as New South Wales cave system or something like that. And then that will be added to the set of vocabularies in an isolated vocabulary which corresponds to this folksonomy idea. So then if I was to later search for that and type cave this time I get the other two that we saw before but the specific Genolan caves keyword as well. But as I said it appears at the bottom because we're essentially assigning this a lower weighting. And that's something that I haven't specifically talked about yet and I can answer questions about the kind of search that's being done here but it's essentially a weighting based upon where the keyword appears in the vocabulary and also what vocabulary. So we can accept that. And yeah I think that's really the main thing to show. I also just want to show the web service briefly and then make some closing remarks. You need to move along David? Yes. Yes. Okay. All right. Okay. So then briefly just returning to the top level. This is just to show what adjacent response for a web service based search would actually look like. And I guess again we can examine this in more detail in discussion if you like but there's a rich amount of information in here that we're not even showing in any way except for the text based values in the widget. So the final thing just to say is that there is as I mentioned a DAP collection and you can just if you just go to the Data Access Portal on top of the vocabulary or keyword aggregator you'll come across it. There are links out to the bucket repository where the code is and some publications relating to the work as well as a presentation that actually has the slides that you saw before but substantially more based on the ModSim 2015 talk. That's it.
 Before we actually go through the slides, I just want to give you a little bit of background.  So as Simon mentioned, in Sorrow Scientific Computing and the applications team now,  I was in the data processing team at the time that Simon and I were working on this project.  So basically the Keyword Aggregator really is in summary a web service, a web application  and web service that allows search, keyword search across managed vocabularies, but also  vocabularies, the vocabularies can include user defined terms and I'll demonstrate how that works  and discuss the motivations. So apart from the application and the web service, what we've done  is to provide a demonstration widget, jQuery based widget that you can use to pretty much as is,  but it's really just intended to demonstrate how you might go about using the web service.  So this work was actually originally carried out in the context of any research project and  as Simon said, NICCAR is really the brains behind this. I'm more or less the code monkey  who happens to understand enough about the domain to be able to work in it and as Simon said,  he and also Jonathan, you provided some invaluable input at the beginning of the project in particular.  So a lot of the work that was done towards this was leading up to the ModSIM 2015 conference  and there was a hiatus of about a year and then in the last half year we've started to do some  more work in the context of the data management capability enhancement program with the view  to potentially looking at the use of this system in the Data Access portal, but that's to be  determined yet. It's really just to try and understand whether we could get to the point  where it was mature enough to think about using it in that sort of context and I'm going to  provide a motivation for the creation of this thing in the first place in the context of the  Data Access portal. There is a data collection, a DAP collection which I'll point to that you can  go to to actually get the code. It's a slightly earlier version than where we are up to now  and needs to be updated, but there's also a bunch of links to publications in relation to the  Keywood aggregator and just general technical information about it. So with that I'll just  launch into the slides and there's only a handful of them. What I want to do then after having  motivated while we even want to do this is to then give a short demonstration of the aggregator  and hopefully that will lead to some questions and conversation. Okay, so we're all familiar with  the idea of needing keywords of some sort in publications whether we're talking about data  publications or software publications or journal papers, conference papers, nothing new there.  Thinking about DAP in particular, you will come across a couple of different sorts of  keywords. You can think of the fields of research as essentially a control vocabulary, so this is  from the ANZSRC, Fields of Research. David are you showing a full screen because I'm just stuck  on one slide with the title publication keywords. Is that what you are trying to show?  Do you currently see one that says about this data collection and then fields of research and  keywords? No, seeing the Janolan CAFES one and it's not moving up. Now it's moved to publication  keywords, now it's moved on down. Okay, that's interesting. Alright, well is that readable  for everyone? Because I was in full screen mode because if it is I'll just stick with this.  Is that alright? Yep, okay. So, yeah, as I was saying what we see here are two different sorts  of keywords, the fields of research and free text entry keywords basically. So in terms of  how you would select those within the DAP, you have essentially quite controlled drop-down boxes  that let you select particular fields of research and that's pretty much the end of it. There's no  sort of wiggle room. Contrast that with free entry keywords, that's pretty much as it appears.  You type in the text, you separate it with semicolons and that's essentially how it appears in the  final collection. So one's very controlled and one is completely freeform essentially. What would  be nice, now actually what happens if I try to, if I show this can you guys see any animation?  No, David. Okay, fine, thank you. I'll just leave it like that then. So if you think about those  keywords that I just showed you, it would be nice instead of just typing in 3D to get some sort of  guidance as to some possible keywords that might relate to 3D concepts. Same thing for LIDAR and  SLAM and mapping and caves and whatnot. And so what this is intended to show, and the animation  shows it a bit more clearly, but what this is intended to show is just the idea that if you  type 3D you might get a list of things that might make you think about something more specific to  enter rather than just 3D, likewise with mapping and SLAM and LIDAR and whatnot. But the question  mark there above, Janowan Caves and Zebedee, is just to point out that they're always going to be,  no matter what vast set of vocabularies or keywords, resources you might have,  there's always going to be this situation where there will be keywords that just don't fit or  that aren't catered for. So how do you deal with that basically? This is where we come across the  idea of folksonomies. And so the idea there is that you might enter the word Zebedee and then  essentially get no results and be prompted for description so that you can then add that or  include that in future searches essentially. So this is just showing from left to right and then  down and across to back to the left again how that might happen. And that likewise for Janowan Caves  there might not be anything specific for that so you can enter that. But the question then becomes  well if you enter those sorts of folksonomy based vocabulary keywords, if someone searches for  them in the future should they appear at the top of the list, should they appear at the bottom of  the list? And I guess the idea is that since they're not based upon a formal vocabulary that a group  has decided or agreed upon that they should be last in the list of search results. So in summary  what we've done is as I said created this web application web service that allows search  across vocabularies including a folksonomy. And these are essentially isolated in named graphs,  the idea of named graphs and the idea is that you may will have conflicts between them but you need  to still be able to allow for the possibility of all these vocabularies living together happily.  And then as I said we exposed it for a demonstration with it, widget. I'll also show you what's  happening behind in terms of a regular web service response that you would get in terms of JSON.  And then just briefly the architecture is just to say we've got a web service that uses particular  implementation frameworks and we have a database for storing certain statistics and then a  Spark LendPoint that talks to an RDF based triple store which has scos vocabularies.  Alright so before I move on to a demonstration any questions or should I just go straight to the  demonstration? Hi David, Nick here, I've just appeared.  Hi Nick, how are you? That's good, you can deal with any Nali questions.  Go to the demonstration, thank you David and then it becomes real for everyone of course.  Very good. Okay so this is the web service, sorry this is the keyword aggregator top level page.  Now I'm showing here instance of a containerized instance that's based on Docker.  This was one of the things that we did in the work earlier this year.  We talked about it a couple of years ago but didn't actually end up doing that.  So I'm happy to talk about that if there's any interest.  So what you see at the top level here is the ability to get to a widget,  the widget that I mentioned, some information about the vocabularies that are in this particular  instance of the keyword aggregator and I'll link to an example web service API.  Cool. So what I want to do is talk about the widget  first because that's really I guess the simplest way of seeing what happens here.  That was an interesting noise. Okay so if I type fish for example I'll get a list and  the number of results that you get back from a search is configurable.  And yeah so essentially we can say alright well I've got to choose one of these and then  if I start trying to use some of the examples that I had in the, okay there's nothing  another vocabularies has that right now.  So mapping for example.  But decide that.  David you're still showing the PowerPoint.  Really?  I think you might be showing your wrong screen David.  Yeah I am indeed sorry I haven't used too much.  Okay so I need you to just find that.  Thanks for that.  Actually I'm just going to share the screen.  It might be easier.  That's what I should have done.  Okay can you see the web page now?  Just waiting.  No still seeing your PowerPoint control panel.  Okay I'm just going to stop and reshare again.  That's now we're seeing a web page.  Okay sorry about that guys.  So yes so again just very brief ago back so this is the top level page.  I'm going to focus initially on the widget.  And so if I type in a term here such as map similar to sort of to the sort of example  we saw in the slides.  I might decide that digital soil mapping is the one that I want.  And then looking at things like LiDAR.  So again the initial it may still make sense to just say no I just want to talk about LiDAR  in general.  But someone seeing a set of results of this might determine that actually is better to  have something more specific for the keyword.  SLAM was another one we were looking at.  And this is good because you know the person trying out the keyword here might decide that  actually the audience, the people who may be looking at the data collection might be  better served by having not an acronym but the acronym expanded and so on.  So we can keep doing this and obviously we get to the point where we start looking at  caves and okay there was in particular the Genolan caves was one of the keywords that  was entered by the depositors.  And you wouldn't expect really to see a specific keyword in general vocabulary for that.  So we now see the ad link here.  So if I click the ad link I can enter a description such as New South Wales cave system or something  like that.  And then that will be added to the set of vocabularies in an isolated vocabulary which  corresponds to this folksonomy idea.  So then if I was to later search for that and type cave this time I get the other two  that we saw before but the specific Genolan caves keyword as well.  But as I said it appears at the bottom because we're essentially assigning this a lower weighting.  And that's something that I haven't specifically talked about yet and I can answer questions  about the kind of search that's being done here but it's essentially a weighting based  upon where the keyword appears in the vocabulary and also what vocabulary.  So we can accept that.  And yeah I think that's really the main thing to show.  I also just want to show the web service briefly and then make some closing remarks.  You need to move along David?  Yes.  Yes.  Okay.  All right.  Okay.  So then briefly just returning to the top level.  This is just to show what adjacent response for a web service based search would actually  look like.  And I guess again we can examine this in more detail in discussion if you like but there's  a rich amount of information in here that we're not even showing in any way except for  the text based values in the widget.  So the final thing just to say is that there is as I mentioned a DAP collection and you  can just if you just go to the Data Access Portal on top of the vocabulary or keyword  aggregator you'll come across it.  There are links out to the bucket repository where the code is and some publications relating  to the work as well as a presentation that actually has the slides that you saw before  but substantially more based on the ModSim 2015 talk.  That's it.
The_Keyword_Aggregator__qZFdaxIcp0_0_6287.png The_Keyword_Aggregator__qZFdaxIcp0_6288_6395.png The_Keyword_Aggregator__qZFdaxIcp0_6396_7331.png The_Keyword_Aggregator__qZFdaxIcp0_7332_7763.png The_Keyword_Aggregator__qZFdaxIcp0_7764_8459.png The_Keyword_Aggregator__qZFdaxIcp0_8460_11411.png The_Keyword_Aggregator__qZFdaxIcp0_11412_13031.png The_Keyword_Aggregator__qZFdaxIcp0_13032_18791.png The_Keyword_Aggregator__qZFdaxIcp0_18792_18875.png The_Keyword_Aggregator__qZFdaxIcp0_18876_19271.png The_Keyword_Aggregator__qZFdaxIcp0_19272_19343.png The_Keyword_Aggregator__qZFdaxIcp0_19344_19559.png The_Keyword_Aggregator__qZFdaxIcp0_19560_24167.png The_Keyword_Aggregator__qZFdaxIcp0_24168_24227.png The_Keyword_Aggregator__qZFdaxIcp0_24228_24323.png The_Keyword_Aggregator__qZFdaxIcp0_24324_24527.png The_Keyword_Aggregator__qZFdaxIcp0_24528_24635.png The_Keyword_Aggregator__qZFdaxIcp0_24636_24815.png The_Keyword_Aggregator__qZFdaxIcp0_24816_25151.png The_Keyword_Aggregator__qZFdaxIcp0_25152_25199.png The_Keyword_Aggregator__qZFdaxIcp0_25200_25271.png The_Keyword_Aggregator__qZFdaxIcp0_25272_25499.png The_Keyword_Aggregator__qZFdaxIcp0_25500_26047.png