<jupyter_start><jupyter_text>Digital Earth Africa Landsat Datasets * **Products used:** [ls5_usgs_sr_scene](https://explorer.digitalearth.africa/ls5_usgs_sr_scene), [ls7_usgs_sr_scene](https://explorer.digitalearth.africa/ls7_usgs_sr_scene), [ls8_usgs_sr_scene](https://explorer.digitalearth.africa/ls8_usgs_sr_scene), [usgs_ls8c_level2_2](https://explorer.digitalearth.africa/ga_ls8c_gm_2_annual)<!-- Digital Earth Africa currently contains two Landsat products that differ in their specifications.  This notebook is intended as a quick guide to understanding the specifications and provenance of each product. Each product description includes information on:* The data processing pipeline used to produce the dataset* Some important details on each products configurations* The operational status of each product* The date range available for each product* The minimum loading requirements to use with `dc.load` to extract data* The bands each product contains -->  Background Landsat Collection 1, Level 2DE Africa contains [Landsat Collection 1, Level 2](https://www.usgs.gov/land-resources/nli/landsat/landsat-collection-1?qt-science_support_page_related_con=1qt-science_support_page_related_con) products over the five countries participating in the African Regional Data Cube (ARDC) (Tanzania, Senegal, Sierra Leone, Ghana, and Kenya).  Landsat Collection 1 consists of products generated from the Landsat 8 Operational Land Imager (OLI) / Thermal Infrared Sensor (TIRS), Landsat 7 Enhanced Thematic Mapper Plus (ETM+), Landsat 4-5 Thematic Mapper (TM), and Landsat 1-5 Multispectral Scanner (MSS) instruments. The implementation of collections ensures consistent and known radiometric and geometric quality through time and across instruments and improves control in the calibration and processing parameters. Level 2 products are processed to `Analysis Ready` standards:OLI / TIRS and ETM+ data processed to Level 1 Systematic and Terrain Corrected products (L1GT), TM and MSS data processed to Level 1 Systematically corrected (L1GS) products, and Level 1 Precision and Terrain Corrected products (L1TP) with image-to-image registration to the GLS control of >12-meter RMSE. >Product names in the datacube: `"ls8_usgs_sr_scene"`, `"ls7_usgs_sr_scene"`, `"ls5_usgs_sr_scene"`**Important details:*** Surface reflectance product    * Valid SR scaling range: `0 – 10,000 (-9999 is no-data)`* Spatial resolution: 30 x 30 m* [CFMask](https://github.com/USGS-EROS/espa-cloud-masking/tree/master/cfmask) used as cloud mask* Universal Transverse Mercator (UTM) map projection. World Geodetic System (WGS) 84 datum* Georeferenced Tagged Image File Format (GeoTIFF)* Pixel alignment is centre**Status:** ARDC Landsat data is on only available over the five participating countries (Senegal, Ghana, Tanzania, Kenya, Sierra Leone). Data is publicly available and can be accessed via both the public and admin DE Africa Sandboxes.Once the entirety of Landsat 5, 7 and 8 USGS Collection 2, Level 2 data is indexed for Africa, the ARDC Landsat data will be decommissioned.**Date-range**: LS8 : 2013-03 to 2019-12LS7 & LS5 : Patchy and depends on location**Minimum loading requirements:**                data = dc.load(product="ls8_usgs_sr_scene",                               output_crs=input_crs,                               resolution=(-30, 30),                               align=(15, 15),                               **query)                               **Bands:**        coastal_aerosol        blue                   green                   red                      nir                      swir1                    swir2                    pixel_qa                 sr_aerosol               radsat_qa***  Landsat Collection 2, Level 2> **Note:** Collection 2 data is not currently available on the public-facing DE Africa platforms. It is expected to be available by the middle of the year. [USGS Collection 2, level 2](https://www.usgs.gov/land-resources/nli/landsat/landsat-collection-2?qt-science_support_page_related_con=1qt-science_support_page_related_con) data has a number of processing, geometric, and radiometric improvements, along with a new data distribution process compared to Collection 1.Landsat Level-2 products are generated from Collection 2 Level-1 inputs that meet the <76 degrees Solar Zenith Angle constraint and include the required auxiliary data inputs to generate a scientifically viable product.Landsat 8 Operational Land Imager (OLI) surface reflectance products are generated using the Land Surface Reflectance Code (LaSRC) algorithm (Version 1.4.1). Landsat 4-5 Thematic Mapper (TM) and Landsat 7 Enhanced Thematic Mapper (ETM+) surface reflectance products are generated using the Landsat Ecosystem Disturbance Adaptive Processing System (LEDAPS) algorithm (Version 3.4.0).Landsat 4-8 surface temperature products are generated using the Landsat surface temperature algorithm (Version 1.3.0), developed in cooperation with the Rochester Institute of Technology and NASA Jet Propulsion Laboratory.A few number of improvements can be envisaged in Collection 2 compared to collection 1:_Geometric Accuracy_* The Landsat 8 Operational Land Imager (OLI) Ground Control Points (GCPs) is re-baselined to the European Space Agency Copernicus Sentinel-2 Global Reference Image (GRI) to improve the interoperability of the global Landsat archive spatially and temporally._Digital Elevation Modeling_* Collection 2 uses 3-arc-second digital elevation modeling (DEM) sources. _Global Level-2 Science and Atmospheric Auxiliary Products_* Processing and distribution of Level-2 surface reflectance and surface temperature science products for Landsats 4-8. Landsat Collection 2 Atmospheric Auxiliary Data used in the processing of Collection 2 Level-2 products are also available to download._Consistent Quality Assessment Bands_* Collection 2 Level-1 products are delivered with two additioanl bands called : Quality Assessment Band (QA_Pixel) and a Radiometric Saturation and Terrain Occlusion Quality Assessment Band (QA_RADSAT)._Updated and Consistent Metadata Files_* In order to facilitate consistency, machine-to-machine scripting and rapid querying of the USGS Landsat archive, two metadata files are included in Collection 2:Object Description Language-based format called Material Library File(MTL) and Extensive Markup Language(XML)_Cloud Optimized File Fomat_* Landsat Collection 2 products are delivered in a Cloud Optimized Georeferenced (COG) Tagged Image File Format that is an extension of the current GeoTIFF file format. The COG format improves access to Geospatial datasets in a cloud-based environment by allowing users to request only the data band files that they need.  ---> Product names in the datacube: `“usgs_ls8c_level2_2”` **Important details:*** Surface reflectance product    * Valid SR scaling range: `1 – 65,455 (0 is no-data)`* Spatial resolution: 30 x 30 m* Includes surface brightness temperature    * Temperature scaling 1 - 65,535* [CFMask](https://github.com/USGS-EROS/espa-cloud-masking/tree/master/cfmask) used as cloud mask* scene-based and in UTM/PS Cloud Optimized GeoTIFF (COG) format.* Pixel alignment is centre**Status:** Provisional, will be public released early-to-mid this year. Currently only available on the DE Africa admin Sandbox **Date-range**: 2013-03 to 2019-06**Minimum loading requirements:**                data = dc.load(product="usgs_ls8c_level2_2",                               output_crs=input_crs,                               resolution=(-30, 30),                               align=(15, 15),                               **query)**Bands:**        coastal_aerosol                            blue                                   green                                     red                                        nir                                        swir_1                                     swir_2                                    surface_temperature                       thermal_radiance                           upwell_radiance                            downwell_radiance                          atmospheric_transmittance                 emissivity                                 emissivity_stdev                           cloud_distance                             quality_l2_aerosol                         quality_l2_surface_temperature            quality_l1_pixel                           quality_l1_radiometric_saturation***  DescriptionThis notebook will run through loading in Landsat Collection 1, Level 2 satellite images.Topics covered include:* Using the native `dc.load()` function to load in Landsat data from a single satellite* Using the `load_ard()` wrapper function to load in a concatenated, sorted, and cloud masked time series from Landsat 5, 7 & 8.***  Getting startedTo run this analysis, run all the cells in the notebook, starting with the "Load packages" cell.  Load packages<jupyter_code>import sys
import datacube

sys.path.append("../Scripts")
from deafrica_datahandling import load_ard, mostcommon_crs
from deafrica_plotting import rgb<jupyter_output><empty_output><jupyter_text>Connect to the datacube<jupyter_code>dc = datacube.Datacube(app="Landsat Collections")<jupyter_output><empty_output><jupyter_text>Load Landsat data from the datacubeWe will load **Landsat** data using two methods.Firstly, we will use [dc.load()](../Beginners_guide/03_Loading_data.ipynb) to return a time series of satellite images from a single sensor.Secondly, we will load a time series using the [load_ard()](../Frequently_used_code/Using_load_ard.ipynb) function, which is a wrapper function around the dc.load module.This function will load all the images from Landsat 5,7 & 8, combine them, and then apply a cloud mask.The returned `xarray.Dataset` will contain analysis ready images with the cloudy and invalid pixels masked out.You can change any of the parameters in the `query` object below to adjust the location, time, projection, or spatial resolution of the returned datasets.To learn more about querying, refer to the Beginner's guide notebook on [loading data](../Beginners_guide/03_Loading_data.ipynb).Landsat data is stored on file with a range of different coordinate reference systems or CRS (i.e. multiple UTM zones). Because of this, all Landsat queries need to include the following two query parameters:* `output_crs`: This sets a consistent CRS that all Sentinel-2 data will be reprojected to, regardless of the UTM zone the individual image is stored in.* `resolution`: This sets the resolution that all Sentinel-2 images will be resampled to.<jupyter_code># Create a query object
query = {
    'x': (-1.9, -2),
    'y': (6.9,7),
    "time": ("2017-01", "2017-03"),
    "resolution": (-30, 30),
    "group_by": "solar_day",
}<jupyter_output><empty_output><jupyter_text>Load Landsat using `dc.load()`The three Collection 1, Level 2 Landsat products are:`"ls8_usgs_sr_scene"`, `"ls7_usgs_sr_scene"`, `"ls5_usgs_sr_scene"`Here we will load in a time-series of satellite images from only Landsat 8.<jupyter_code># Identify the most common projection system in the input query
output_crs = mostcommon_crs(dc=dc, product='ls8_usgs_sr_scene', query=query)

#load data
ds = dc.load(product="ls8_usgs_sr_scene",
             output_crs=output_crs,
             dask_chunks={},
             **query)

print(ds)<jupyter_output><xarray.Dataset>
Dimensions:          (time: 10, x: 370, y: 370)
Coordinates:
  * time             (time) datetime64[ns] 2017-01-07T10:27:51.185930 ... 2017-03-21T10:21:07.951450
  * y                (y) float64 7.739e+05 7.739e+05 ... 7.629e+05 7.628e+05
  * x                (x) float64 6.105e+05 6.105e+05 ... 6.215e+05 6.215e+05
Data variables:
    coastal_aerosol  (time, y, x) int16 dask.array<chunksize=(1, 370, 370), meta=np.ndarray>
    blue             (time, y, x) int16 dask.array<chunksize=(1, 370, 370), meta=np.ndarray>
    green            (time, y, x) int16 dask.array<chunksize=(1, 370, 370), meta=np.ndarray>
    red              (time, y, x) int16 dask.array<chunksize=(1, 370, 370), meta=np.ndarray>
    nir              (time, y, x) int16 dask.array<chunksize=(1, 370, 370), meta=np.ndarray>
    swir1            (time, y, x) int16 dask.array<chunksize=(1, 370, 370), meta=np.ndarray>
    swir2            (time, y, x) int16 dask.array<chunksize=(1, 370, 370), meta=np.ndarray>
[...]<jupyter_text>The returned dataset contains all of the bands available for Landsat 8. If we wish to return only a subset of these bands, we can specify `measurements`.<jupyter_code>bands = ["blue", "green", "red"]

ds = dc.load(product='ls8_usgs_sr_scene',
             measurements=bands,
             output_crs=output_crs,
             dask_chunks={},
             **query)

print(ds)
rgb(ds, index=7)<jupyter_output><empty_output><jupyter_text>Load Landsat using `load_ard`This function will load images from all the landsat sensors, concatenate and sort the observations by time, and apply a cloud mask.The result is an analysis ready dataset.You can find more information on this function from the [Using load ard](../Frequently_used_code/Using_load_ard.ipynb) notebook.<jupyter_code>ds = load_ard(dc=dc,
              products=["ls8_usgs_sr_scene",
                        "ls7_usgs_sr_scene",
                        "ls5_usgs_sr_scene"],
              measurements=bands,
              output_crs=output_crs,
              dask_chunks={},
              **query)

print(ds)
rgb(ds, index=13)<jupyter_output><empty_output><jupyter_text>*** Additional information**License:** The code in this notebook is licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). Digital Earth Africa data is licensed under the [Creative Commons by Attribution 4.0](https://creativecommons.org/licenses/by/4.0/) license.**Contact:** If you need assistance, please post a question on the [Open Data Cube Slack channel](http://slack.opendatacube.org/) or on the [GIS Stack Exchange](https://gis.stackexchange.com/questions/ask?tags=open-data-cube) using the `open-data-cube` tag (you can view previously asked questions [here](https://gis.stackexchange.com/questions/tagged/open-data-cube)).If you would like to report an issue with this notebook, you can file one on [Github](https://github.com/digitalearthafrica/deafrica-sandbox-notebooks).**Last modified:** Feb 2020**Compatible datacube version:**<jupyter_code>print(datacube.__version__)<jupyter_output>1.7+253.ga031f3f4.dirty<jupyter_text>TagsBrowse all available tags on the DE Africa User Guide's [Tags Index]()<jupyter_code>**Tags**: :index:`Landsat`, :index:`deafrica_datahandling`, :index:`dea_plotting`, :index:`load_ard`, :index:`mostcommon_crs`,:index:`rgb`,<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08093269165593439, 'fraction_numerical': 0.02833322036196028, 'mean_word_length': 3.556516368128474, 'pattern_counts': {'":': 3, '<': 41, '<?xml version=': 0, '>': 44, 'https://': 13, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '291634', 'license': 'Apache-2.0', 'n_tokens_mistral': 4395, 'n_tokens_neox': 3985, 'n_words': 1509, 'path': 'Datasets/Landsat_collections.ipynb', 'repo_name': 'NMoghaddam/deafrica-sandbox-notebooks', 'size': '14753'}