<jupyter_start><jupyter_text>Using Intake-ESM to Analyze Data from CESM2-LEIn mid June, the [CESM2 Large Ensemble](https://www.cesm.ucar.edu/projects/community-projects/LENS2/) dataset was made available to the public. This model was run in collaboration with the [IBS Center for Climate Physics](https://ibsclimate.org/) and the [National Center for Atmospheric Research](https://ncar.ucar.edu/) This dataset includes 100 ensemble members, at one degree spatial resolution, with each ensemble member including data from 1850 to 2100. If you are interested in learning more about how this ensemble was setup, be sure to check out [the main webpage](https://www.cesm.ucar.edu/projects/community-projects/LENS2/) or read the pre-print of [Rodgers et al. 2021](https://esd.copernicus.org/preprints/esd-2021-50/) which describes this dataset in detail. Main ChallengeOne of these challenges with this dataset is dealing with the massive amount of output. The data are available through the [NCAR Climate Data Gateway](https://www.cesm.ucar.edu/projects/community-projects/LENS2/data-sets.html) and via the [IBS OpenDAP Server](https://climatedata.ibs.re.kr/data/cesm2-lens). There is also a subset of the dataset available on the GLADE file system on NCAR HPC resources available within the directory `/glade/campaign/cgd/cesm/CESM2-LE/timeseries/`. Through these traditional file-access methods, one would need to scroll through and find one or a few of the **millions** of files produced by the model. Fortunately, this dataset has been catalogued, available using the [`Intake-esm`](https://intake-esm.readthedocs.io/en/latest/) package, which enables one to query the data, and read into a dictionary of `xarray.Datasets`, preventing the user from having to setup the concatenation and file search themselves.Within the [intake-esm FAQ](https://intake-esm.readthedocs.io/en/latest/supplemental-guide/faq.html) section, there is a list of existing catalogs, as shown below:![intake-esm catalogs](../images/intake_esm_catalogs.png)At the top of the ["Is there a list of existing catalogs"](https://intake-esm.readthedocs.io/en/latest/supplemental-guide/faq.html) section, you can see the `CMIP6-GLADE` catalog, which includes:* Description of the catalog* Platform* Catalog path or url* Data format* Documentation page---For the CESM2-LE Catalog, we see:**CESM2-LE-GLADE*** Description: ESM collection for the CESM2 LENS data stored on GLADE in /glade/campaign/cgd/cesm/CESM2-LE/timeseries* Platform: NCAR-GLADE* Catalog path or url: /glade/collections/cmip/catalog/intake-esm-datastore/catalogs/glade-cesm2-le.json* Data Format: netCDF* Documentation Page: https://www.cesm.ucar.edu/projects/community-projects/LENS2/We are **most interested** in the catalog path here  Using the CatalogNow that we found the catalog file, and see that it is the dataset we are interested in, we can start our analysis! Imports<jupyter_code>import intake
import matplotlib.pyplot as plt
import xarray as xr
from distributed import Client
from ncar_jobqueue import NCARCluster<jupyter_output><empty_output><jupyter_text>Spin up our Dask Cluster<jupyter_code># Create our NCAR Cluster - which uses PBSCluster under the hood
cluster = NCARCluster()

# Spin up 20 workers
cluster.scale(20)

# Assign the cluster to our Client
client = Client(cluster)
client<jupyter_output><empty_output><jupyter_text>Read in the data catalogAs mentioned before, we use the catalog path `/glade/collections/cmip/catalog/intake-esm-datastore/catalogs/glade-cesm2-le.json`<jupyter_code>cat = intake.open_esm_datastore(
    '/glade/collections/cmip/catalog/intake-esm-datastore/catalogs/glade-cesm2-le.json'
)<jupyter_output>/glade/u/home/mgrover/miniconda3/envs/hires-marbl/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3437: DtypeWarning: Columns (8,9) have mixed types.Specify dtype option on import or set low_memory=False.
  exec(code_obj, self.user_global_ns, self.user_ns)<jupyter_text>Let's take a second to investigate the catalog - each file on disk represents an `asset`, which means that there are over 4 **million** files within the `/glade/campaign/cgd/cesm/CESM2-LE/timeseries/` directory<jupyter_code>cat<jupyter_output><empty_output><jupyter_text>Querying for our desired variableIn this case, we are interested in the surface temperature in Boulder, Colorado. There are **numerous** temperature variables in the dataset, we can search for all the `long_names` from atmospheric output.We can use the wildcard here `*` to look for where both captial and lowercase `temperature` show up in the `long_name`<jupyter_code>cat.search(component='atm', long_name=['temperature*', 'Temperature*']).df.long_name.unique()<jupyter_output><empty_output><jupyter_text>Query and Subset our CatalogLet's go with `Lowest model level temperature` since this represents the closest to the surface without being at the **actual** surface of the earth. We pass the `component` and `long_name` into the query, which reduces the number of datasets to 8!<jupyter_code>subset = cat.search(component='atm', long_name='Lowest model level temperature')
subset<jupyter_output><empty_output><jupyter_text>Read in using `.to_dataset_dict()`We include the additional `dask.config.set()` to help with splitting up large chunks when reading in<jupyter_code>with dask.config.set(**{'array.slicing.split_large_chunks': True}):
    dsets = subset.to_dataset_dict()
dsets.keys()<jupyter_output><empty_output><jupyter_text>Let's take a look at the keys - these are defined by the `groupby` attributes in the catalog. The groupby attributes in this case are:`component.experiment.stream.forcing_variant.control_branch_year.variable`* Component - which component this output is from (ex. atm represents the atmosphere)* Experiment - which experiment this is from, in this case, this is `ssp370` which is one of the CMIP6 future experiments* Stream - which stream this output is from, in this case, this is `cam.h1`, which represents daily output* Control Branch Year - which year the ensemble branched off from, these are described within the [CESM2-LE documentation page](https://www.cesm.ucar.edu/projects/community-projects/LENS2/)* Variable - which variable you are working with  Define a helper function which subsets the data, groups by year, and returns the data array<jupyter_code>def subset_ds(ds):
    # Subset for TBOT and choose Boulder, Colorado's lat and lon and convert to degC
    da = ds.TBOT.sel(lat=40.015, lon=-105.2705, method='nearest') - 273.15
    da = da.groupby('time.year').mean()

    # Make sure that time is the format of a string
    da['year'] = da.year.astype(int)

    return da<jupyter_output><empty_output><jupyter_text>Look at one of the datasets<jupyter_code>ds = dsets['atm.ssp370.cam.h1.smbb.1301.TBOT']<jupyter_output><empty_output><jupyter_text>Apply our function<jupyter_code>da = subset_ds(ds)<jupyter_output><empty_output><jupyter_text>Plot our Results!<jupyter_code># Setup the figure
fig = plt.figure(figsize=(12, 8))
ax = plt.subplot(111)

# Loop through and plot each member_id
for member_id in da.member_id.values:
    da.sel(member_id=member_id).plot(ax=ax, label=member_id)

# Limit the axis from 2015 to 2100
plt.xlim(2015, 2100)

# Increase the ticksize
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)

# Add some labels and increase the size
plt.xlabel('Year', fontsize=16)
plt.ylabel('Temperature (degC)', fontsize=16)

# Add a legend
plt.legend(title='Member IDs', fontsize=12)

# Add a title
plt.title(
    'Experiment: SSP370 \n Forcing Variant: Smoothed Biomass \n Control Branch Year: 1301 \n 2015 - 2100',
    fontsize=20,
);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09770414818679886, 'fraction_numerical': 0.01747978085050874, 'mean_word_length': 5.5251063829787235, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 12, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '643522', 'license': 'Apache-2.0', 'n_tokens_mistral': 2523, 'n_tokens_neox': 2357, 'n_words': 794, 'path': 'posts/2021/intake-cesm2-le-glade-example.ipynb', 'repo_name': 'jukent/esds', 'size': '7666'}