Earth Analytics Science and Innovation Hub
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Woodcock, Rob
Dr Robert Woodcock is a Research Scientist and Consultant in Earth Data and Analytics infrastructures with CSIRO Minerals and CSIRO Space and Astronomy. Data and its conversion to useful, decision supporting information is a universal need for business, community, and Government across a broad range of areas from improving business operations through application in decision support at a national and international levels in earth resource, urban and environmental management. As our ability to gather data has grown exponentially, so has our need to manage the flow of information within and between organisations and to make it useful. Dr Woodcock and his teams have address the issues of information flow across organisational boundaries and have developed innovative information systems adopted into national infrastructure and achieved lasting large scale economic impact on multiple organisations. Dr Woodcock's portfolio of activities involves national spatial data infrastructure and CSIRO Earth Analytics Science Innovation Cloud platform. His team’s portfolio has seen national application in Australian Industry and Government including the AuScope national geoscience information infrastructure. Internationally, he is a member of the Committee on Earth Observation Satellites (CEOS) and Chair for the CEOS Working Group on Information Systems and Services. Robert has worked for almost two decades in the field of visualisation, spatial information systems and analytics and its application to Earth Science with a focus on ensuring research innovation leads to business innovation.
 Hi, I'm Dr. Robert Woodcock and it's great to be with you today at the Open Data Cube Conference. I'd like to give you a bit of an overview of the impact the Open Data Cube has had at the CSIRO and with our many clients and collaborators in our work in the Earth observation. We'll begin by having a look at the Earth Analytics at CSIRO and the diversity of use cases and technologies that are involved and also our Earth Analytics Science Innovation Hub and how we're using that and the Open Data Cube within it to really accelerate how we take our research outcomes and make them available to industry and government globally. Then we'll look specifically at the contributions that we make to the Open Data Cube community and we hope you're enjoying some of those and consider some of those in your future work. A brief overview of the easy data pipelines and also a look at really just how this is actually playing out for us with both our research and our public good deployments and also our commercial enterprise deployments and engagement we have using the Open Data Cube. So Earth observation at CSIRO is a big area. We have a few hundred researchers working in Earth observation or related fields. We not only do satellite data from the public data sets that you see globally like Sentinel and Landsat but we also make very heavy use of high resolution data acquired from satellites and also from airborne and UAV type data sets. We combine that also with field work. So we have sensors like a Hydrospector which is used for in situ sensors on ground for aquatic reflections and so forth and we combine those data sets together into our analytics codes to provide some of the best of breed water quality type algorithms, agricultural work. We have researchers in oceans, atmosphere, minerals and mining areas, urban researchers and so forth. So it's quite a broad area and creating an analytics platform that's capable of handling all of those is actually quite challenging given the variety of data and the variety of research that's being performed. As a result we have a system which we call EASY or the Earth Analytics Science and Innovation Hub. The core analytics are powered by Open Data Cube of course. That's one aspect of it that's very important for Earth observation work. We also make very good use of the Pangeo area or work which is primarily around climate and oceans type modeling type work. We have machine learning capabilities in the system and really just drawing on the Python scientific data ecosystem quite broadly. We use this environment EASY for really accelerating the transfer of the research outcomes that we're producing into use by large to small, medium enterprises, government and research organizations on a global scale. As a result we have EASY deployments in a number of regions around the world. Australia of course, the United States, Chile, Southeast Asia and we are providing EASY as a deployment option as a shared tendency for commercial subscribers and we have a couple of those as well as an enterprise secure environment for organizations that want their own system integrated fully within their enterprise. This type of engagement is enabling us to use EASY as a platform to conduct research with those organizations and then transfer that research outcome directly into use by that organization. The EASY itself as I mentioned is an ecosystem. It's not just Open Data Cube, it's a wide variety of systems across the Python tool suite. It has as a result a range of interfaces and the point I wanted to make with this range of interfaces is it is very common for our work to begin at the exploratory phase and Jupyter Labs and direct Python coding using things like Visual Studio Code which is also available through the EASY interfaces. Using that direct coding interface to do that iterative analysis in an environment that we know can be deployed and used directly with our clients begins at that exploratory data analytics. It starts there, you can pick it up and then put it into a web application for on-the-fly creation of a product or in a scalar production workflow for automated update. You can do the experiment and then you can transfer it to print production and the code is staying relatively the same all the way through and that's really the power that we're getting from the EASY system. So the capabilities of EASY mentioned Open Data Cube, clearly a part of it. Analysis ready data is clearly also very important. We apply this not only to Earth observation data but also data coming from things like our Hydrospectros sensor for field measurements and in situ measurements and having that flow into the system. Our users have in this environment their own space as well as shared project spaces. This is all handled and authenticated under attribute-based access control within the system which is a key enterprise feature for the EASY system. We also have users have their own customizable DAS clusters, plural. Users can actually have their own DAS clusters, multiple of them and they can scale those up and down as required for their particular research workflow that they're undertaking. The actual platform will scale to thousands of cores. It may scale even higher. That's kind of the limit we've got to at the moment in terms of demand and we perform multi-sensor data integration very routinely and so there is a variety of data more than just Earth observation in the system. The actual system itself is a Kubernetes cluster running on AWS. It's all managed through DevOps controls, tools like HashiCook Terraform and Flux and this has been a very important part of that automation of deployment. Without that, I doubt very much we could actually run this system. There are many thousands of moving parts across the multiple deployments. Sira itself contributes to the ODC in a number of areas. In particular, we're interested in things like the ZAR driver for Open DataCube. That ZAR driver adds multi-dimensional support. For us, it's particularly important for handling hyperspectral satellite data which is coming, starting to be an increasing part of our work. We also have extensions for supporting LiDAR data from Jedi and a number of other satellites as part of our work and coming through and we'll be contributing those components back into the Open DataCube community as they come on board. We're also core contributors to the Kubernetes and Terraform environment and feel free to reach out to us for assistance and advice in that area if you're interested in setting up our production Open DataCube that scales. We also play a role in coordinating with the International Space Agencies, USGS Landsat, analysis ready data from Copernicus, etc. It doesn't happen without influence and both GS Science Australia and CSIRO contribute directly into the Committee on Earth Observation Satellites Leadership roles, coordinating activities in ensuring that information systems and services across all the CS agencies are making more and more of the data available on the cloud and in a now-assisted ready form for you to use. Another area we contribute is, of course, in science and applications outside of DataCube Core, some of the notebooks that you see kicking around in the Open DataCube community have had contributions from CSIRO and we work very closely with GA on a number of products as well. So all of this work on the Open DataCube has had and the Earth Analytics Science Innovation Hub has had a tremendous impact on our research. We have several hundred users within CSIRO and a few dozen projects actually operating on easy as a routine matter now and that's in a pre-production phase. We're actually working with our corporate Enterprise IT Group now to get it into a fully production environment with single sign-on across the whole of CSIRO at the moment and we've also deployed this, as I mentioned, into multiple regions around the world and we have commercial subscribers. So at the moment we're actually running seven production environments of easy across the planet with a very wide variety of data types coming through our data pipelines capability and more are coming online as we speak. We're looking to add a lot more work around the enterprise features, particularly fine-grained access control on resources and control on billing and cost controls within the system itself. And we're also looking for a way to encourage the creation of going from a notebook to a dashboard and so with that I'd like to say thank you very much for your time today and I hope this has been a useful overview. If you'd like further information feel free to contact any of the members of our team and thank you again and I hope you enjoy the rest of the Open DataCube conference.
 Hi, I'm Dr. Robert Woodcock and it's great to be with you today at the Open Data Cube Conference.  I'd like to give you a bit of an overview of the impact the Open Data Cube has had at the CSIRO  and with our many clients and collaborators in our work in the Earth observation.  We'll begin by having a look at the Earth Analytics at CSIRO and the diversity of use cases  and technologies that are involved and also our Earth Analytics Science Innovation Hub and how we're  using that and the Open Data Cube within it to really accelerate how we take our research outcomes  and make them available to industry and government globally.  Then we'll look specifically at the contributions that we make to the Open Data Cube community  and we hope you're enjoying some of those and consider some of those in your future work.  A brief overview of the easy data pipelines and also a look at really just how this is actually  playing out for us with both our research and our public good deployments and also our commercial  enterprise deployments and engagement we have using the Open Data Cube.  So Earth observation at CSIRO is a big area. We have a few hundred researchers working in  Earth observation or related fields. We not only do satellite data from the public data sets that  you see globally like Sentinel and Landsat but we also make very heavy use of high resolution data  acquired from satellites and also from airborne and UAV type data sets. We combine that also with  field work. So we have sensors like a Hydrospector which is used for in situ sensors on ground for  aquatic reflections and so forth and we combine those data sets together into our analytics codes  to provide some of the best of breed water quality type algorithms, agricultural work.  We have researchers in oceans, atmosphere, minerals and mining areas, urban researchers and so forth.  So it's quite a broad area and creating an analytics platform that's capable of handling all of those  is actually quite challenging given the variety of data and the variety of research that's being  performed. As a result we have a system which we call EASY or the Earth Analytics Science and  Innovation Hub. The core analytics are powered by Open Data Cube of course. That's one aspect of it  that's very important for Earth observation work. We also make very good use of the Pangeo area  or work which is primarily around climate and oceans type modeling type work. We have machine  learning capabilities in the system and really just drawing on the Python scientific data ecosystem  quite broadly. We use this environment EASY for really accelerating the transfer of the  research outcomes that we're producing into use by large to small, medium enterprises,  government and research organizations on a global scale. As a result we have EASY deployments in  a number of regions around the world. Australia of course, the United States, Chile, Southeast Asia  and we are providing EASY as a deployment option as a shared tendency for commercial subscribers  and we have a couple of those as well as an enterprise secure environment for organizations  that want their own system integrated fully within their enterprise. This type of engagement is  enabling us to use EASY as a platform to conduct research with those organizations and then transfer  that research outcome directly into use by that organization. The EASY itself as I mentioned is  an ecosystem. It's not just Open Data Cube, it's a wide variety of systems across the Python  tool suite. It has as a result a range of interfaces and the point I wanted to make with  this range of interfaces is it is very common for our work to begin at the exploratory phase and  Jupyter Labs and direct Python coding using things like Visual Studio Code which is also available  through the EASY interfaces. Using that direct coding interface to do that iterative analysis in  an environment that we know can be deployed and used directly with our clients begins at that  exploratory data analytics. It starts there, you can pick it up and then put it into a web  application for on-the-fly creation of a product or in a scalar production workflow for automated  update. You can do the experiment and then you can transfer it to print production and the code  is staying relatively the same all the way through and that's really the power that we're getting from  the EASY system. So the capabilities of EASY mentioned Open Data Cube, clearly a part of it.  Analysis ready data is clearly also very important. We apply this not only to Earth observation data  but also data coming from things like our Hydrospectros sensor for field measurements and  in situ measurements and having that flow into the system. Our users have in this environment  their own space as well as shared project spaces. This is all handled and authenticated under  attribute-based access control within the system which is a key enterprise feature for the EASY system.  We also have users have their own customizable DAS clusters, plural. Users can actually have their own  DAS clusters, multiple of them and they can scale those up and down as required for their particular  research workflow that they're undertaking. The actual platform will scale to thousands of  cores. It may scale even higher. That's kind of the limit we've got to at the moment in terms of  demand and we perform multi-sensor data integration very routinely and so there is a variety of data  more than just Earth observation in the system. The actual system itself is a Kubernetes cluster  running on AWS. It's all managed through DevOps controls, tools like HashiCook Terraform  and Flux and this has been a very important part of that automation of deployment. Without that,  I doubt very much we could actually run this system. There are many thousands of moving parts  across the multiple deployments. Sira itself contributes to the ODC in a number of areas.  In particular, we're interested in things like the ZAR driver for Open DataCube.  That ZAR driver adds multi-dimensional support. For us, it's particularly important for handling  hyperspectral satellite data which is coming, starting to be an increasing part of our work.  We also have extensions for supporting LiDAR data from Jedi and a number of other satellites  as part of our work and coming through and we'll be contributing those components back into the  Open DataCube community as they come on board. We're also core contributors to the Kubernetes  and Terraform environment and feel free to reach out to us for assistance and advice in that area  if you're interested in setting up our production Open DataCube that scales. We also play a role  in coordinating with the International Space Agencies, USGS Landsat, analysis ready data from  Copernicus, etc. It doesn't happen without influence and both GS Science Australia and CSIRO  contribute directly into the Committee on Earth Observation Satellites Leadership roles,  coordinating activities in ensuring that information systems and services across  all the CS agencies are making more and more of the data available on the cloud and in  a now-assisted ready form for you to use. Another area we contribute is, of course, in science and  applications outside of DataCube Core, some of the notebooks that you see kicking around in the  Open DataCube community have had contributions from CSIRO and we work very closely with GA on a  number of products as well. So all of this work on the Open DataCube has had and the  Earth Analytics Science Innovation Hub has had a tremendous impact on our research. We have  several hundred users within CSIRO and a few dozen projects actually operating on easy as a  routine matter now and that's in a pre-production phase. We're actually working with our corporate  Enterprise IT Group now to get it into a fully production environment with single sign-on across  the whole of CSIRO at the moment and we've also deployed this, as I mentioned, into multiple regions  around the world and we have commercial subscribers. So at the moment we're actually running  seven production environments of easy across the planet with a very wide variety of data types  coming through our data pipelines capability and more are coming online as we speak. We're looking  to add a lot more work around the enterprise features, particularly fine-grained access  control on resources and control on billing and cost controls within the system itself.  And we're also looking for a way to encourage the creation of going from a notebook to a dashboard  and so with that I'd like to say thank you very much for your time today and I hope this has  been a useful overview. If you'd like further information feel free to contact any of the  members of our team and thank you again and I hope you enjoy the rest of the Open DataCube conference.
ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_0_107.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_108_1403.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_1404_2063.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_2064_3047.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_3048_5087.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_5088_6623.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_6624_8231.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_8232_8303.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_8304_8843.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_8844_11171.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_11172_13151.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_13152_13379.png ODC_Conference_2021_-_Rob_Woodcock__CSIRO_EASI_lsCibr2BHpw_13380_13381.png