<jupyter_start><jupyter_text>Extract subset of ozone profiles from UKESM1 This notebook illustrates how to extract a subset of ozone profiles from UKESM1. It is designed to be implemented on a Pangeo service. Note that it also uses a "preprocessing" script from <NAME>. More up-to-date versions can be found here: https://github.com/jbusecke/cmip6_preprocessingThis notebook extracts the UKESM1 profiles from three different experiments, calculates the annual mean profiles, and saves the results to NetCDF files. These NetCDF files are then used by notebook 2.0 to train the unsupervised classification analysis.   Start dask cluster We use dask to speed up the data collection and averaging.<jupyter_code>from dask.distributed import Client

client = Client("tcp://10.32.5.46:44639")
client<jupyter_output><empty_output><jupyter_text>Import required packages<jupyter_code>import intake
import xarray as xr
from matplotlib import pyplot as plt
import numpy as np
import xgcm
import dask
import pprint
import gsw
import xesmf as xe
from xhistogram.xarray import histogram
from datetime import datetime

# local file for CMIP6 preprocessing
import preprocessing<jupyter_output><empty_output><jupyter_text>Locate UKESM1 ozone data We select the MOHC (Met Office - Hadley Centre) created model UKESM1. We select the monthly atmospheric variables (just O3) from three different experiments. Specifically, we select the historical run, SSP126, and SSP585.<jupyter_code>col_url='https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json'
col = intake.open_esm_datastore(col_url)
cat = col.search(institution_id='MOHC',
                 source_id='UKESM1-0-LL',
                 table_id='Amon',
                 experiment_id=['historical','ssp126','ssp585'],
                 variable_id=['o3'],
                 member_id=['r1i1p1f2'],
                 grid_label='gn')
cat.df
dset_dict = cat.to_dataset_dict(zarr_kwargs={'consolidated': True, 'decode_times': True},
                                cdf_kwargs={'chunks': {}, 'decode_times': True})
dset_dict.keys()<jupyter_output>Progress: |███████████████████████████████████████████████████████████████████████████████| 100.0% 

--> The keys in the returned dictionary of datasets are constructed as follows:
	'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
             
--> There are 3 group(s)<jupyter_text>Preprocess the historical, ssp126, and ssp585 datasets<jupyter_code>historical = dset_dict['CMIP.MOHC.UKESM1-0-LL.historical.Amon.gn']
ssp126 = dset_dict['ScenarioMIP.MOHC.UKESM1-0-LL.ssp126.Amon.gn']
ssp585 = dset_dict['ScenarioMIP.MOHC.UKESM1-0-LL.ssp585.Amon.gn']

historical = preprocessing.combined_preprocessing(historical)
ssp126 = preprocessing.combined_preprocessing(ssp126)
ssp585 = preprocessing.combined_preprocessing(ssp585)<jupyter_output><empty_output><jupyter_text>Drop the coordinates that are not needed<jupyter_code>historical = historical.drop(('lon_bnds','time_bounds','lat_bnds','member_id','bnds'))
ssp126 = ssp126.drop(('lon_bnds','time_bounds','lat_bnds','member_id','bnds'))
ssp585 = ssp585.drop(('lon_bnds','time_bounds','lat_bnds','member_id','bnds'))<jupyter_output><empty_output><jupyter_text>Calculate annual mean profiles<jupyter_code>historical = historical.groupby('time.year').mean('time')
ssp126 = ssp126.groupby('time.year').mean('time')
ssp585 = ssp585.groupby('time.year').mean('time')<jupyter_output><empty_output><jupyter_text>Slice one decade from each experiment<jupyter_code>historical = historical.sel(year=slice(2004,2014))
ssp126 = ssp126.sel(year=slice(2090,2100))
ssp585 = ssp585.sel(year=slice(2090,2100))

historical = historical.squeeze()
ssp126 = ssp126.squeeze()
ssp585 = ssp585.squeeze()

# scale to get units in ppmv ()
historical['o3'] = historical.o3*1e6
ssp126['o3'] = ssp126.o3*1e6
ssp585['o3'] = ssp585.o3*1e6<jupyter_output><empty_output><jupyter_text>Prepare attributes for combined NetCDF file<jupyter_code>historical.attrs['Prepared by'] = '<NAME>'
historical.attrs['Institute'] = 'British Antarctic Survey'
historical.attrs['Units'] = 'ppmv'
historical.attrs['Model Info'] = 'UK Earth System Model 1'
historical.attrs['Description'] = 'Annual mean ozone profiles from historical experiment'
historical.attrs['Years covered'] = '2004-2014'

ssp126.attrs['Prepared by'] = '<NAME>'
ssp126.attrs['Institute'] = 'British Antarctic Survey'
ssp126.attrs['Units'] = 'ppmv'
ssp126.attrs['Model Info'] = 'UK Earth System Model 1'
ssp126.attrs['Description'] = 'Annual mean ozone profiles from ssp126 experiment (strong emissions reductions)'
ssp126.attrs['Years covered'] = '2090-2100'

ssp585.attrs['Prepared by'] = '<NAME>'
ssp585.attrs['Institute'] = 'British Antarctic Survey'
ssp585.attrs['Units'] = 'ppmv'
ssp585.attrs['Model Info'] = 'UK Earth System Model 1'
ssp585.attrs['Description'] = 'Annual mean ozone profiles from ssp585 experiment (strong emissions reductions)'
ssp585.attrs['Years covered'] = '2090-2100'<jupyter_output><empty_output><jupyter_text>Save results to NetCDF files<jupyter_code>historical.load()
historical.to_netcdf(path='./data_out/UKESM_O3_historical_v2.nc')
ssp126.load()
ssp126.to_netcdf(path='./data_out/UKESM_O3_ssp126_v2.nc')
ssp585.load()
ssp585.to_netcdf(path='./data_out/UKESM_O3_ssp585_v2.nc')<jupyter_output><empty_output><jupyter_text>Create merged dataset, save to single NetCDF file<jupyter_code>merged = xr.concat([historical,ssp126,ssp585],dim='year')
merged.to_netcdf(path='./data_out/UKESM_O3_merged.nc')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13491635186184567, 'fraction_numerical': 0.05018888289260658, 'mean_word_length': 6.4034620505992015, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 46, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '198631', 'license': 'MIT', 'n_tokens_mistral': 2180, 'n_tokens_neox': 1906, 'n_words': 395, 'path': 'notebooks/exploratory/1.0_dcj_get_ozone_profiles.ipynb', 'repo_name': 'DanJonesOcean/ozone_clustering_ukesm', 'size': '5573'}