Big Spatial Data seminar Part 1: Analythics in the Age of Big Data
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Ramachandran, Rahul
Big Spatial Data seminar - University of Alabama - July 25th 2013. Rahul discusses their NASA ACCESS funded online tool: Curated Data Albums, an online tool for gathering and presenting relevant data and information from distributed sources. These Data Albums are complied collections of information related to events with links to relevant data files (granular) from different instruments
 Dr Rahul Ramachandran is deputy editor for Earth Science Informatics and a principal research scientist at the Information Technology and Systems Centre at the University of Alabama in Huntsville. Rahul has pioneered the concept of data prospecting, which sits between data mining and data discovery. He is also chair of the newly formed Research Data Alliance Working Group on Big Data Analytics. He graduated with a PhD in 2002 from the Atmospheric Science Department at the University of Alabama in Huntsville and was selected for the 2010 Presidential Early Career Award for Scientists and Engineers. In this presentation, he speaks about his work on data discovery and big complex data in the domain of atmospheric science. Thank you for inviting me to give the start. So here's the outline art. I'll go a few slides to introduce who am I, what I do with the research, talk a little bit more about the research lab where I work. A lab actually runs a data archive. I think very much similar demands. So there may be things of interest there. And then I'll talk about two ongoing data projects that might be of interest to people here. The first one is actually looking at data and information aggregation. The other one is Big Data looking at event analytics. I have a rather eclectic educational background that's kind of all over the place. I guess I'm still an engineer. That's what I did as an undergrad. But I have degrees in atmospheric science and computer science. A little bit about my center. So we are a research lab. We have a lot of work in data mining and other stuff. We obviously work in the area of informatics. We do a fair amount of organic stuff. We build games that are used by the US military for training. So a lot of our funding comes from all these three letter agencies in the US. So we also run one of the 12 NASA data archives. So it's basically a fully operational data archive. We do operational data and just processing, archiving, distribution. And our primary data sets are lightning data. We also hold a lot of the ground validation free campaign data that we get at NASA once for instrument validation. And then we have the microwave data products. That's actually an important part of for climate science. You all know this. This is the data lifecycle. What has changed in the recent years is that data is no longer treated as a second class citizen. It's treated as a first class object. Importance of data is now being slowly realized. There's a code that data is now the currency in science. What we try to do is to figure out what are the processes and efficiencies that are there in the data lifecycle. How can we make scientific process be faster, better, more productive? So what's happening is as technology is evolving, some of the process components may be getting obsolete. So we have to start looking at new solutions that can be used. And the other thing that's happening in the US, and it may be happening there too, is you are getting new policy requirements that you have to deal with. Like Anna said, it now requires a data management plan. NASA has a data preservation requirement that you have to have and all the metadata details so that you can have. You have enough contextual understanding to years down the road that you can use the data and the other thing that's coming down our horizon is the whole notion of reproducibility and executable paper. That's the gold standard that everyone's going through with science, especially the aspects of science that have major policy implications. You have to make sure that you can reproduce those results. So the area that I work in is third science informatics. And that's basically looking at how do you apply systematic technology approaches to the entire aspects of the data lifecycle, not just the knowledge extraction and the decisions of the part, but the whole acquisition process, how do you gather information, all that part. But then the important thing is providing very customized solutions to the stakeholders and not giving tools that they cannot utilize. Now I'm going to transition. This is pretty much a background on what I do. And I'm going to talk about two ongoing projects. This presentation, these projects are at a fairly high level. So this is a slide that's required now to give a definition of big data. The first one is Gartner's definition of big data that we know is where you have the whole notion of velocity volume, to be able to understand the velocity that you have, real time aspects to the data. And the notion of the variety that there's different kinds of data that you're getting that will have different kinds of quality information, format types that you have to handle. I really don't like Gartner's definition of big data. That's a different perspective, I guess. So this is Jim Frew from University of Santa Barbara. His definition I like, I think it's more from a data center perspective. You can't really move it. It's like an organ. If you want to play, you have to go to the organ, the organ comes here. So that's the big data. You can't move it. If you want to use it, you have to go where it is. So implications to data center throughout the world is that now you have to start looking at systems that can actually do analysis on data. So the two projects that I'm presenting here, the first one is actually focusing on this whole notion of variety. How do you now actually have so much information and distributed data that you can get on the web at different locations, different sources. How can you automate aggregation around events of interest? The second case is actually looking at more analytics, looking at event analytics. So here's the first project. This is a NASA funded project. It's called curated data outcomes for science case studies. The concept here is that our data album is basically a compiled collection of information around an event of interest. This compiled collection includes not just the data files that you want to use for studying that event, but also has links to services, tools, news reports, videos, anything that gives you full contextual understanding that's useful for studying that event around the world. The motivation here is to allow an end user to basically customize the data album for their particular study. Each user may have a different view of what they want to study that event for. So the motivation behind building a tool like this is that it is an atmospheric science. One of the most common research that is done is case study analysis and planetary studies focused on a significant director, a major flooding event, a major hurricane through knowledge research is done on understanding how that event occurred. So to do that, what we need is a wide variety of data and information from all the distributed locations that are there. So for example, NASA has all the different DACs. Each DAC is holding one kind of data set. You are an individual researcher, then you have to figure out where to go and get the data based on the metadata that's there provided. The other thing is science is also becoming very interdisciplinary. You may have users who may not be experts in a particular data set, or they may not know the exact vocabulary or metadata term to use to find, to do the search right. So how do you support users like that? So the whole gathering of data and information around this whole notion of events is actually very tedious and time consuming. So the challenge is to build a tool that can do this gathering of information in an automated manner. But the gathering part is actually easy part. The hard part is how do you figure out what is relevant in terms of what's out there. So that's the challenge here. Once you're gathering stuff, figuring out what to filter and what to keep. The other part of it is the metadata tends to be fairly boring. How do you present this collated information in a manner that is actually more useful and intuitive? We can present metadata in a really dry manner. So can we do something that is different here? That is the challenge in the sense of building this tool. So the science driver here is for hurricane science. It's probably the easiest event to start with because it's a major event. There's lots of information about it. There's information about tracks and how does the hurricane actually progress. So the goal was to use this as the first science driver and build catalogs on all the hurricane events for last 20 years. This is not just on the data but it's also on the information that is not required. It's the background information. What was the damage caused? How many deaths were there? And all of these things require parsing through webpages or PDF files of storm reports. This is the conceptual architecture. You have these different resources on the left hand side that are coming from all these agencies in the US. You may have things that are crowdsourced like videos and pictures on YouTube that you can get. So the goal is to aggregate this, figure out what is relevant and then put it in a structured form and present it to a end user so that they can actually utilize this information. So this is the architecture for the tool. There is an engine that drives the different brokers that talk to the different stupid resources and then puts everything together in a no security database because some of the data gets so large. Effortively querying it is an issue. And there is a service layer and then at the top you have a presentation layer where you provide a user to do some interactive analytics, there is interactive visualization and a past visual search part. The piece I'm going to talk about is the piece that's new is this whole ontology-based relevancy ranking capability and then obviously I'll show you a little bit about the tool. So the ranking service, the ontology is ranked as a service and actually designed as a general service that can be customized to many different applications and it combines this ontology-based score and a traditional statistical score as based on these two papers by Boromol and the score-ranked paper. Now I'm going to go into the details of the algorithm but there are two components here and the first one is an ontology component where we have an application ontology. In this case we have an ontology for a hurricane. For all the concepts in the ontology we calculate the weights, basically the linkages between the concepts. So the more connected a concept is, the higher weight it has and then we calculate an activation value. So not everything in the ontology, not all the concepts in the ontology are important. There are certain ones that are much higher of a higher value. But those are the key ones, that's where the search starts. So those concepts get a much higher activation values compared to the other ones. And then we use the very standard DFID model for statistical calculations where you do a term frequency calculation for a word and then you do an inverse-document frequency calculation for a word. So the relevancy score where we calculate is that we look at a document, look at all the metadata and we match against the concepts in the ontology and then we calculate a score based on the ontology and we calculate a score based on the DFID model. And then that is given a relevancy score. So that's how we can do some relevancy filtering on the information that we are gathering. Question from Andrew Chauvre. So I can see how that's going to work for documents. I can see how it's going to work for some of the non-document resources that you're showing in the bottom layer of your conceptual architecture. Done like what? That's on YouTube for instance. So what we do on YouTube is we do a query expansion. The ontology runs the query expansion. If you're searching for say, a hurricane Sandy, right? So we do just a hurricane Sandy on YouTube. You may get people who call Sandy with their... Yeah, right. So we can use the ontology to automate the query expansion. So we add more detail itself so that does the relevancy filtering. So we did the... We looked at how well this algorithm works. Clearly the algorithm can be improved part of it. This is the rest of the work. There are known things we know we can improve it. Not in terms of the algorithm itself. So we compared the algorithm against truth data, which is against our data center collections for hurricane. So we manually selected 35 data collections and we compared it against the top 35 that were returned by the hurricane. So we get an accuracy of about 80% the precision and recall is about 60. So ideally you want high precision and high recall but normally that never happens. So for search purposes, we have tuned it to be at a high recall and a much lower frequency. The goal here is to make sure that everything that is important is as part of the search process. So if you should demo this page. So this is the information that's been aggregated for the different hurricanes. You have three different views. We still have people who like things in a tabular form. So we insisted on having a list view. There's a different view of like a bubble. You can see the different particular year. It's a different storm. And the size actually, the color is the category of the storm and the size is the number of it's information that's been collected for the particular storm. The sunburst is the same thing but you now have a way of doing a visual pass at some time. So you can actually can drill down to a particular year. You can see the storms based on different categories and again the angle of the size is the storm duration in this case. And if you select a particular storm, this is all the aggregated information that we are getting. So this is from Wikipedia. You don't have this information, the tabular information. All this is coming by passing PDF reports. Things like this that are important for the users. Like how well do they do the forecast for this particular storm list is their record. So this is using Brokers to introduce particular kinds of reports. And passing the PDF is based on a rule based on the slide rule that it passes itself. And then you have for the particular storm itself, now you have the actual datasets. So these are the different data collections that you have. And you have all the granules that you can actually get a list of that are on a server that you can actually have or study that with this storm. You can search based on keywords or instruments. And the user can change the threshold that are relevant to the current threshold. So they think the threshold is too high or too low and they are not getting enough results and they can change the threshold. Oops. Sorry about that. And you can select your individual, somehow it's not refreshing. You select your individual data collection when you get to those parts that you can utilize to study. So this is a different way of gathering distributed information. Here's how to get the information. you you you you
 Dr Rahul Ramachandran is deputy editor for Earth Science Informatics and a principal  research scientist at the Information Technology and Systems Centre at the University of Alabama  in Huntsville.  Rahul has pioneered the concept of data prospecting, which sits between data mining and data discovery.  He is also chair of the newly formed Research Data Alliance Working Group on Big Data Analytics.  He graduated with a PhD in 2002 from the Atmospheric Science Department at the University of Alabama  in Huntsville and was selected for the 2010 Presidential Early Career Award for Scientists  and Engineers.  In this presentation, he speaks about his work on data discovery and big complex data  in the domain of atmospheric science.  Thank you for inviting me to give the start.  So here's the outline art.  I'll go a few slides to introduce who am I, what I do with the research, talk a little  bit more about the research lab where I work.  A lab actually runs a data archive.  I think very much similar demands.  So there may be things of interest there.  And then I'll talk about two ongoing data projects that might be of interest to people  here.  The first one is actually looking at data and information aggregation.  The other one is Big Data looking at event analytics.  I have a rather eclectic educational background that's kind of all over the place.  I guess I'm still an engineer.  That's what I did as an undergrad.  But I have degrees in atmospheric science and computer science.  A little bit about my center.  So we are a research lab.  We have a lot of work in data mining and other stuff.  We obviously work in the area of informatics.  We do a fair amount of organic stuff.  We build games that are used by the US military for training.  So a lot of our funding comes from all these three letter agencies in the US.  So we also run one of the 12 NASA data archives.  So it's basically a fully operational data archive.  We do operational data and just processing, archiving, distribution.  And our primary data sets are lightning data.  We also hold a lot of the ground validation free campaign data that we get at NASA once  for instrument validation.  And then we have the microwave data products.  That's actually an important part of for climate science.  You all know this.  This is the data lifecycle.  What has changed in the recent years is that data is no longer treated as a second class  citizen.  It's treated as a first class object.  Importance of data is now being slowly realized.  There's a code that data is now the currency in science.  What we try to do is to figure out what are the processes and efficiencies that are there  in the data lifecycle.  How can we make scientific process be faster, better, more productive?  So what's happening is as technology is evolving, some of the process components may be getting  obsolete.  So we have to start looking at new solutions that can be used.  And the other thing that's happening in the US, and it may be happening there too, is  you are getting new policy requirements that you have to deal with.  Like Anna said, it now requires a data management plan.  NASA has a data preservation requirement that you have to have and all the metadata details  so that you can have.  You have enough contextual understanding to years down the road that you can use the data  and the other thing that's coming down our horizon is the whole notion of reproducibility  and executable paper.  That's the gold standard that everyone's going through with science, especially the aspects  of science that have major policy implications.  You have to make sure that you can reproduce those results.  So the area that I work in is third science informatics.  And that's basically looking at how do you apply systematic technology approaches to  the entire aspects of the data lifecycle, not just the knowledge extraction and the  decisions of the part, but the whole acquisition process, how do you gather information, all  that part.  But then the important thing is providing very customized solutions to the stakeholders  and not giving tools that they cannot utilize.  Now I'm going to transition.  This is pretty much a background on what I do.  And I'm going to talk about two ongoing projects.  This presentation, these projects are at a fairly high level.  So this is a slide that's required now to give a definition of big data.  The first one is Gartner's definition of big data that we know is where you have the  whole notion of velocity volume, to be able to understand the velocity that you have,  real time aspects to the data.  And the notion of the variety that there's different kinds of data that you're getting  that will have different kinds of quality information, format types that you have to  handle.  I really don't like Gartner's definition of big data.  That's a different perspective, I guess.  So this is Jim Frew from University of Santa Barbara.  His definition I like, I think it's more from a data center perspective.  You can't really move it.  It's like an organ.  If you want to play, you have to go to the organ, the organ comes here.  So that's the big data.  You can't move it.  If you want to use it, you have to go where it is.  So implications to data center throughout the world is that now you have to start looking  at systems that can actually do analysis on data.  So the two projects that I'm presenting here, the first one is actually focusing on this  whole notion of variety.  How do you now actually have so much information and distributed data that you can get on the  web at different locations, different sources.  How can you automate aggregation around events of interest?  The second case is actually looking at more analytics, looking at event analytics.  So here's the first project.  This is a NASA funded project.  It's called curated data outcomes for science case studies.  The concept here is that our data album is basically a compiled collection of information  around an event of interest.  This compiled collection includes not just the data files that you want to use for studying  that event, but also has links to services, tools, news reports, videos, anything that  gives you full contextual understanding that's useful for studying that event around the  world.  The motivation here is to allow an end user to basically customize the data album for  their particular study.  Each user may have a different view of what they want to study that event for.  So the motivation behind building a tool like this is that it is an atmospheric science.  One of the most common research that is done is case study analysis and planetary studies  focused on a significant director, a major flooding event, a major hurricane through  knowledge research is done on understanding how that event occurred.  So to do that, what we need is a wide variety of data and information from all the distributed  locations that are there.  So for example, NASA has all the different DACs.  Each DAC is holding one kind of data set.  You are an individual researcher, then you have to figure out where to go and get the  data based on the metadata that's there provided.  The other thing is science is also becoming very interdisciplinary.  You may have users who may not be experts in a particular data set, or they may not  know the exact vocabulary or metadata term to use to find, to do the search right.  So how do you support users like that?  So the whole gathering of data and information around this whole notion of events is actually  very tedious and time consuming.  So the challenge is to build a tool that can do this gathering of information in an automated  manner.  But the gathering part is actually easy part.  The hard part is how do you figure out what is relevant in terms of what's out there.  So that's the challenge here.  Once you're gathering stuff, figuring out what to filter and what to keep.  The other part of it is the metadata tends to be fairly boring.  How do you present this collated information in a manner that is actually more useful and  intuitive?  We can present metadata in a really dry manner.  So can we do something that is different here?  That is the challenge in the sense of building this tool.  So the science driver here is for hurricane science.  It's probably the easiest event to start with because it's a major event.  There's lots of information about it.  There's information about tracks and how does the hurricane actually progress.  So the goal was to use this as the first science driver and build catalogs on all the hurricane  events for last 20 years.  This is not just on the data but it's also on the information that is not required.  It's the background information.  What was the damage caused?  How many deaths were there?  And all of these things require parsing through webpages or PDF files of storm reports.  This is the conceptual architecture.  You have these different resources on the left hand side that are coming from all these  agencies in the US.  You may have things that are crowdsourced like videos and pictures on YouTube that you  can get.  So the goal is to aggregate this, figure out what is relevant and then put it in a structured  form and present it to a end user so that they can actually utilize this information.  So this is the architecture for the tool.  There is an engine that drives the different brokers that talk to the different stupid resources  and then puts everything together in a no security database because some of the data  gets so large.  Effortively querying it is an issue.  And there is a service layer and then at the top you have a presentation layer where you  provide a user to do some interactive analytics, there is interactive visualization and a past  visual search part.  The piece I'm going to talk about is the piece that's new is this whole ontology-based  relevancy ranking capability and then obviously I'll show you a little bit about the tool.  So the ranking service, the ontology is ranked as a service and actually designed as a general  service that can be customized to many different applications and it combines this ontology-based  score and a traditional statistical score as based on these two papers by Boromol and  the score-ranked paper.  Now I'm going to go into the details of the algorithm but there are two components here  and the first one is an ontology component where we have an application ontology.  In this case we have an ontology for a hurricane.  For all the concepts in the ontology we calculate the weights, basically the linkages between  the concepts.  So the more connected a concept is, the higher weight it has and then we calculate an activation  value.  So not everything in the ontology, not all the concepts in the ontology are important.  There are certain ones that are much higher of a higher value.  But those are the key ones, that's where the search starts.  So those concepts get a much higher activation values compared to the other ones.  And then we use the very standard DFID model for statistical calculations where you do  a term frequency calculation for a word and then you do an inverse-document frequency  calculation for a word.  So the relevancy score where we calculate is that we look at a document, look at all  the metadata and we match against the concepts in the ontology and then we calculate a score  based on the ontology and we calculate a score based on the DFID model.  And then that is given a relevancy score.  So that's how we can do some relevancy filtering on the information that we are gathering.  Question from Andrew Chauvre.  So I can see how that's going to work for documents.  I can see how it's going to work for some of the non-document resources that you're showing  in the bottom layer of your conceptual architecture.  Done like what?  That's on YouTube for instance.  So what we do on YouTube is we do a query expansion.  The ontology runs the query expansion.  If you're searching for say, a hurricane Sandy, right?  So we do just a hurricane Sandy on YouTube.  You may get people who call Sandy with their...  Yeah, right.  So we can use the ontology to automate the query expansion.  So we add more detail itself so that does the relevancy filtering.  So we did the...  We looked at how well this algorithm works.  Clearly the algorithm can be improved part of it.  This is the rest of the work.  There are known things we know we can improve it.  Not in terms of the algorithm itself.  So we compared the algorithm against truth data, which is against our data center collections  for hurricane.  So we manually selected 35 data collections and we compared it against the top 35 that  were returned by the hurricane.  So we get an accuracy of about 80% the precision and recall is about 60.  So ideally you want high precision and high recall but normally that never happens.  So for search purposes, we have tuned it to be at a high recall and a much lower frequency.  The goal here is to make sure that everything that is important is as part of the search  process.  So if you should demo this page.  So this is the information that's been aggregated for the different hurricanes.  You have three different views.  We still have people who like things in a tabular form.  So we insisted on having a list view.  There's a different view of like a bubble.  You can see the different particular year.  It's a different storm.  And the size actually, the color is the category of the storm and the size is the number of  it's information that's been collected for the particular storm.  The sunburst is the same thing but you now have a way of doing a visual pass at some  time.  So you can actually can drill down to a particular year.  You can see the storms based on different categories and again the angle of the size  is the storm duration in this case.  And if you select a particular storm, this is all the aggregated information that we  are getting.  So this is from Wikipedia.  You don't have this information, the tabular information.  All this is coming by passing PDF reports.  Things like this that are important for the users.  Like how well do they do the forecast for this particular storm list is their record.  So this is using Brokers to introduce particular kinds of reports.  And passing the PDF is based on a rule based on the slide rule that it passes itself.  And then you have for the particular storm itself, now you have the actual datasets.  So these are the different data collections that you have.  And you have all the granules that you can actually get a list of that are on a server  that you can actually have or study that with this storm.  You can search based on keywords or instruments.  And the user can change the threshold that are relevant to the current threshold.  So they think the threshold is too high or too low and they are not getting enough results  and they can change the threshold.  Oops.  Sorry about that.  And you can select your individual, somehow it's not refreshing.  You select your individual data collection when you get to those parts that you can utilize  to study.  So this is a different way of gathering distributed information.  Here's how to get the information.  you  you  you  you
Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_0_1247.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_1248_2219.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_2220_4151.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_4152_11183.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_11184_15323.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_15324_16007.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_16008_19799.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_19800_22415.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_22416_22943.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_22944_23831.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_23832_23939.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_23940_24095.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_24096_24155.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_24156_24479.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_24480_24551.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_24552_24815.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_24816_24863.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_24864_24935.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_24936_25283.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_25284_25451.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_25452_25991.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_25992_26927.png Big_Spatial_Data_seminar_Part_1_-_Rahul_Ramachandran_KYdHY5wZCUc_1_26928_30624.png