<jupyter_start><jupyter_text>SOMMAR Data Volume Estimates from PRIMAVERA<jupyter_code>import django
django.setup()
from django.db.models import Sum
from pdata_app.models import DataRequest, DataFile
from pdata_app.utils.common import filter_hadgem_stream2, get_request_size
def filesizeformat(num_bytes, human_units=True):
    """
    If human_units then output data volumes with units, but with the non-breaking space removed.
    """
    if human_units:
        return django.template.defaultfilters.filesizeformat(num_bytes).replace('\xa0', ' ')
    else:
        return num_bytes<jupyter_output><empty_output><jupyter_text>Total data volume for a Stream 2 MOHC experiment The PRIMAVERA Stream 2 data request was cut down from the Stream 1 request so that only the high frequency variables required for the user facing work packages were output. The output of daily variables on atmosphere levels was also reduced. The Stream 2 data request is summarised at https://doi.org/10.5281/zenodo.3607328.<jupyter_code>hist_reqs = filter_hadgem_stream2(
    DataRequest.objects.filter(
        climate_model__short_name='HadGEM3-GC31-HH',
        experiment__short_name='hist-1950',
        rip_code='r1i1p1f1',
        datafile__isnull=False
    )
)
ocean_table_names = ['Oday', 'PrimOday', 'PrimSIday', 'SIday',
                     'Omon', 'PrimOmon', 'SImon']<jupyter_output><empty_output><jupyter_text>The following variables were output (along with their dimensions):<jupyter_code>for dr in hist_reqs.order_by('variable_request__frequency', 'variable_request__table_name', 
                             'variable_request__cmor_name'):
    var_string = f'{dr.variable_request.table_name}_{dr.variable_request.cmor_name}'
    print(f'{var_string:<30} ({dr.variable_request.dimensions})')<jupyter_output>3hr_pr                         (longitude latitude time)
3hr_rsds                       (longitude latitude time)
3hr_rsdsdiff                   (longitude latitude time)
3hr_tas                        (longitude latitude time1)
3hr_uas                        (longitude latitude time1)
3hr_vas                        (longitude latitude time1)
E3hr_psl                       (longitude latitude time)
E3hrPt_ua7h                    (longitude latitude plev7h time1)
E3hrPt_va7h                    (longitude latitude plev7h time1)
Prim3hr_sfcWind                (longitude latitude height10m time)
Prim3hr_sfcWindmax             (longitude latitude height10m time)
Prim3hrPt_ua100m               (longitude latitude height100m time1)
Prim3hrPt_ua50m                (longitude latitude height50m time1)
Prim3hrPt_va100m               (longitude latitude height100m time1)
Prim3hrPt_va50m                (longitude latitude height50m time1)
Prim3hrPt_zg7h                 (longitude latitude plev7h ti[...]<jupyter_text>For SOMMAR, this data request would require the addition of a variable for each ocean biogeochemistry tracer on olevels. The data volume for 65 years for the 25 km atmosphere and 1/12° ocean is:<jupyter_code>atmos_reqs = hist_reqs.exclude(variable_request__table_name__in=ocean_table_names)
n512_atmos_size = get_request_size(atmos_reqs, 1950, 2014)
print(f'25 km atmosphere {filesizeformat(n512_atmos_size)}')
ocean_reqs = hist_reqs.filter(variable_request__table_name__in=ocean_table_names)
ocean_size = get_request_size(ocean_reqs, 1950, 2014)
print(f'1/12° ocean {filesizeformat(ocean_size)}')
print(f'total {filesizeformat(get_request_size(hist_reqs, 1950, 2014))}')<jupyter_output>25 km atmosphere 18.7 TB
1/12° ocean 29.1 TB
total 47.8 TB<jupyter_text>However, in PRIMAVERA the Met Office ran an N512 model and in SOMMAR it would like to run an N1280 model. Let's see what effect this has on the data volume for this 65 year simulation:<jupyter_code>n1280_atmos_size = n512_atmos_size * (1280 / 512)**2
print(f'10 km atmosphere {filesizeformat(n1280_atmos_size)}')
print(f'1/12° ocean {filesizeformat(ocean_size)}')
print(f'total {filesizeformat(n1280_atmos_size + ocean_size)}')<jupyter_output>10 km atmosphere 116.8 TB
1/12° ocean 29.1 TB
total 146.0 TB<jupyter_text>Data volume per year SOMMAR will have longer runs and so the data volume per year will be:<jupyter_code>n1280_per_year = n1280_atmos_size / 65
print(f'atmosphere per year {filesizeformat(n1280_per_year)}')
ocean_per_year = ocean_size / 65
print(f'ocean per year {filesizeformat(ocean_per_year)}')
total_per_year = n1280_per_year + ocean_per_year
print(f'total per year {filesizeformat(total_per_year)}')<jupyter_output>atmosphere per year 1.8 TB
ocean per year 458.9 GB
total per year 2.2 TB<jupyter_text>For a possible run length of 650 years (200 spinup, 200 piControl and 250 historical + future):<jupyter_code>print(f'atmosphere {filesizeformat(650 * n1280_per_year)}')
print(f'ocean {filesizeformat(650 * ocean_per_year)}')
print(f'total {filesizeformat(650 * total_per_year)}')<jupyter_output>atmosphere 1.1 PB
ocean 291.3 TB
total 1.4 PB<jupyter_text>Data Volume for a surface variable<jupyter_code>amon_tas = DataRequest.objects.filter(
    climate_model__short_name='HadGEM3-GC31-HH',
    experiment__short_name='hist-1950',
    rip_code='r1i1p1f1',
    variable_request__table_name='Amon',
    variable_request__cmor_name='tas'
)

n1280_size = get_request_size(amon_tas, 1950, 2014) * (1280 / 512)**2
print(f'65 years of a suface variable at N1280 {filesizeformat(n1280_size)}')
n1280_650years_size = n1280_size / 65 * 650
print(f'650 years of a suface variable at N1280 {filesizeformat(n1280_650years_size)}')<jupyter_output>65 years of a suface variable at N1280 6.2 GB
650 years of a suface variable at N1280 62.5 GB<jupyter_text>Therefore how to analyse the data needs to be considered as the time series for a variable on a single level will be over 60 GB in size. Many post-processing systems won't have this much RAM available to them.  Data volume per time slice To allow the estimation of SOMMAR volumes we can calculate the storage required by a single atmosphere and ocean time slice.<jupyter_code>atmos_filename = 'tas_Amon_HadGEM3-GC31-HH_hist-1950_r1i1p1f1_gn_195001-195012.nc'
atmos_file = DataFile.objects.get(name=atmos_filename)
print(f'{atmos_filename} {filesizeformat(atmos_file.size)}')

ocean_surface_filename = 'tos_Omon_HadGEM3-GC31-HH_hist-1950_r1i1p1f1_gn_195001-195001.nc'
ocean_surface_file = DataFile.objects.get(name=ocean_surface_filename)
print(f'{ocean_surface_filename} {filesizeformat(ocean_surface_file.size)}')

ocean_filename = 'vo_Omon_HadGEM3-GC31-HH_hist-1950_r1i1p1f1_gn_195001-195001.nc'
ocean_file = DataFile.objects.get(name=ocean_filename)
print(f'{ocean_filename} {filesizeformat(ocean_file.size)}')
print('')

sommar_atmos_filesize = (1280 / 512)**2 * atmos_file.size
sommar_atmos_step_size = sommar_atmos_filesize / 12  # 12 months in the file
print(f'10 km atmos single level single time slice {filesizeformat(sommar_atmos_step_size)}')

ocean_surface_step_size = ocean_surface_file.size # single time point
print(f'1/12° ocean surface single time slice {filesizeformat(ocean_surface_step_size)}')

ocean_step_size = ocean_file.size / 75  # on 75 olevels
print(f'1/12° ocean single level single time slice {filesizeformat(ocean_step_size)}')<jupyter_output>tas_Amon_HadGEM3-GC31-HH_hist-1950_r1i1p1f1_gn_195001-195012.nc 15.8 MB
tos_Omon_HadGEM3-GC31-HH_hist-1950_r1i1p1f1_gn_195001-195001.nc 163.7 MB
vo_Omon_HadGEM3-GC31-HH_hist-1950_r1i1p1f1_gn_195001-195001.nc 1.6 GB

10 km atmos single level single time slice 8.2 MB
1/12° ocean surface single time slice 163.7 MB
1/12° ocean single level single time slice 22.4 MB
{'fraction_non_alphanumeric': 0.08169460208525801, 'fraction_numerical': 0.07087237693018344, 'mean_word_length': 4.934220830070478, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '489327', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2971, 'n_tokens_neox': 2602, 'n_words': 636, 'path': 'Sommar_data_volumes.ipynb', 'repo_name': 'PRIMAVERA-H2020/stream2-planning', 'size': '7577'}