<jupyter_start><jupyter_text>Diurnal CycleThis notebook aims at inroducing new users on how to use the PCDMI diurnal cycle drivers. [This diagram](https://github.com/PCMDI/pcmdi_metrics/blob/master/doc/Diurnal%20Cycle%20Diagram.pdf) shows how various drivers are linked together. To run the following demo, it is expected that you have downloaded the sample data as demonstrated in [the download notebook](Demo_0_download_data.ipynb). For this tutorial we will be using 3 years worth of 3-hourly data resampled to a 5x5 grid.The following cell loads the demo input and output directories set in the download notebook.<jupyter_code>from user_choices import demo_data_directory, demo_output_directory<jupyter_output><empty_output><jupyter_text>Daily MeansLike all other drivers in the PCMDI Metrics Package, Diurnal Cycle relies on parameter input files.Our first driver starts from CMIP5 data and computes the daily means.<jupyter_code>with open("basic_diurnal_compute_daily_mean.py") as f:
    print(f.read())<jupyter_output><empty_output><jupyter_text>Now to run this simply call the driver:```computeStdOfDailyMeans.py -p basic_diurnal_compute_daily_mean.py```<jupyter_code>%%bash
computeStdOfDailyMeans.py -p basic_diurnal_compute_daily_mean.py<jupyter_output>SKIPPING: []
FILES: ['demo_data/misc_demo_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc']
PARAMS: [<pcmdi_metrics.diurnal.common.INPUT object at 0x7ff368087a60>]
Data source: IPSL-CM5A-LR
Opening demo_data/misc_demo_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc ...
Data source: IPSL-CM5A-LR
Opening demo_data/misc_demo_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc ...
Year 1997:
Reading pr from demo_data/misc_demo_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc for time interval 1997-7-1 0:0:0.0 to 1997-7-31 23:59:0.0 ...
Year 1998:
Reading pr from demo_data/misc_demo_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc for time interval 1998-7-1 0:0:0.0 to 1998-7-31 23:59:0.0 ...
Year 1999:
Reading pr from demo_data/misc_demo_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc for time interval 1999-7-1 0:0:0.0 to 1999-7-31 23:59:0.0 ...<jupyter_text>This generates a netcdf file `pr_IPSL-CM5A-LR_Jul_1997-1999_std_of_dailymeans.nc` which contains the daily standard deviation at each cell. Looking at our diagram, the next driver to run is the one computing the mean of the standard deviation from daily means over a region of interest. First we open the parameter file.<jupyter_code>with open("basic_diurnal_std_daily_mean.py") as f:
    print(f.read())<jupyter_output># output directory
results_dir = 'demo_output/diurnal/json'

# input directory which is actually the output of previous driver
modpath = 'demo_output/diurnal/nc'

# filenames template
filename_template = 'pr_%(model)_Jul_%(firstyear)-%(lastyear)_std_of_dailymeans.nc'

# model to use
model = 'IPSL-CM5A-LR'
experiment = 'historical'
realization = 'r1i1p1'

# Month to use
month = 7

# Period
firstyear = 1997 # included
lastyear = 1999  # included

# Latitudes/longitudes to use
lat1 = -50.
lat2 = 50.
lon1 = 0.
lon2 = 360.

# Name
region_name = "TRMM"

# Number of workers
num_workers = 4<jupyter_text>Now to run this simply call the driver```std_of_dailymeans.py -p basic_diurnal_std_daily_mean.py```<jupyter_code>%%bash
std_of_dailymeans.py -p basic_diurnal_std_daily_mean.py<jupyter_output>TEMPLATE NAME: pr_IPSL-CM5A-LR_Jul_1997-1999_std_of_dailymeans.nc
Preparing to write output to JSON file ...
Initializing dictionary of statistical results ...
['demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_std_of_dailymeans.nc']
PARAMS: [<pcmdi_metrics.diurnal.common.INPUT object at 0x7fa1f8669dc0>]
Specifying latitude / longitude domain of interest ...
Reading demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_std_of_dailymeans.nc ...
  Shape = (20, 72)
Finding RMS area-average ...
For IPSL-CM5A-LR in Jul, average variance of daily values = ( 3.92 mm/d)^2
Writing output to JSON file ... {'IPSL-CM5A-LR': {'TRMM': 3.9163177715285187}}
REG MASK: {}
done<jupyter_text>This generates a json file: `pr_Jul_1997_1999_std_of_dailymeans.json`OPTIONAL: You could also append a new region to this json file by overwritting some of our parameters from the command line. This example creates results for the tropics instead of the whole globe.<jupyter_code>%%bash
std_of_dailymeans.py -p basic_diurnal_std_daily_mean.py \
--region_name=TROPICS \
--lat1=-30. \
--lat2=30. \
--lon1=0. \
--lon2=360 \
--append<jupyter_output>TEMPLATE NAME: pr_IPSL-CM5A-LR_Jul_1997-1999_std_of_dailymeans.nc
Preparing to write output to JSON file ...
['demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_std_of_dailymeans.nc']
PARAMS: [<pcmdi_metrics.diurnal.common.INPUT object at 0x7fe1386a3970>]
Specifying latitude / longitude domain of interest ...
Reading demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_std_of_dailymeans.nc ...
  Shape = (12, 72)
Finding RMS area-average ...
For IPSL-CM5A-LR in Jul, average variance of daily values = ( 3.89 mm/d)^2
Writing output to JSON file ... {'IPSL-CM5A-LR': {'TRMM': 3.9163177715285187, 'TROPICS': 3.890735445846671}}
REG MASK: {'TRMM': {'id': 'TRMM', 'domain': {'TRMM': 'cdutil.region.domain(latitude=(-50.0, 50.0),longitude=(0.0, 360.0))'}}}
done<jupyter_text>Diurnal CompositeLooking again at our diagram we can now start again from the original 3 hourly data, and run the *composite* script<jupyter_code>with open("basic_diurnal_composite.py") as f:
    print(f.read())
%%bash
compositeDiurnalStatistics.py -p basic_diurnal_composite.py<jupyter_output>done
TEMPLATE: pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc
FILES: ['demo_data/misc_demo_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc']
PARAMS: [<pcmdi_metrics.diurnal.common.INPUT object at 0x7faca8978d60>]
Data source: IPSL-CM5A-LR
Opening demo_data/misc_demo_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc ...
Year 1997:
Reading pr from demo_data/misc_demo_data/atm/3hr/pr/pr_3hr_IPSL-CM5A-LR_historical_r1i1p1_5x5_1997-1999.nc for time interval 1997-7-1 0:0:0.0 to 1997-7-31 23:59:0.0 ...
Shape: (248, 36, 72)
DATA FROM: 1997-7-1 1:30:0.0 to 1997-7-31 22:30:0.0
  Shape =  (8, 36, 72)
  8 timepoints per day, 3 hr intervals between timepoints
  Creating temporary storage and output fields ...
  Computing Local Standard Times for GMT  1.50 ...
  Computing Local Standard Times for GMT  4.50 ...
  Computing Local Standard Times for GMT  7.50 ...
  Computing Local Standard Times for GMT 10.50 ...
  Computing Local Standard Times for G[...]<jupyter_text>This produces 3 output files:```pr_IPSL-CM5A-LR_Jul_1997-1999_diurnal_avg.ncpr_IPSL-CM5A-LR_Jul_1997-1999_diurnal_std.ncpr_IPSL-CM5A-LR_LocalSolarTimes.nc```These contain respectively the 1997-1999 mean diurnal cycle for July, the standard deviation of these results across individual years, and the local solar time. Results for each of these are available for the entire domain.We can now generate ASCII files for composite diurnal cycle (w/	error	bars) at selected grid points using the `fourierDiurnalGridpoints.py` script.<jupyter_code>%%bash
fourierDiurnalGridpoints.py -p basic_diurnal_fourier.py<jupyter_output>LSTFILES: ['demo_output/diurnal/nc/pr_IPSL-CM5A-LR_LocalSolarTimes.nc']
TMPL pr_IPSL-CM5A-LR_LocalSolarTimes.nc
Results sent to: /Users/ordonez4/Documents/git/pcmdi_metrics/doc/jupyter/Demo/demo_output/diurnal/ascii/pr_Jul_1997-1999_fourierDiurnalGridPoints.asc<jupyter_text>This produces an ascii file: `pr_Jul_1997-1999_fourierDiurnalGridPoints.asc`   Standard Deviation of Hourly ValuesStarting again from the *composite* results our diagram suggests we now compute the standard deviation of hourly values.<jupyter_code>with open("basic_diurnal_std_hourly_mean.py") as f:
    print(f.read())
%%bash
std_of_hourlyvalues.py -p basic_diurnal_std_hourly_mean.py<jupyter_output>TEMPLATE NAME: pr_IPSL-CM5A-LR_Jul_1997-1999_diurnal_std.nc
Specifying latitude / longitude domain of interest ...
Preparing to write output to JSON file ...
Initializing dictionary of statistical results ...
['demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_diurnal_std.nc']
PARAMS: [<pcmdi_metrics.diurnal.common.INPUT object at 0x7f89187349d0>]
Specifying latitude / longitude domain of interest ...
Reading demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_diurnal_std.nc ...
  Shape = (8, 20, 72)
Finding RMS area-average ...
For IPSL-CM5A-LR in Jul, average variance of hourly values = ( 4.85 mm/d)^2
Writing output to JSON file ...
done<jupyter_text>This generated the following file:    `pr_Jul_1997-1999_std_of_hourlymeans.json`    These results are used in Trenberth	et	al.	(2017). They are a measure of the intermittancy of hourly values, which puts “error bars” on the mean diurnal cycle.Going back to the results of the *composite* we can now run `std_of_meandiurnalcycle.py` which can use the same input parameter file as the daily mean computation.<jupyter_code>%%bash
std_of_meandiurnalcycle.py -p basic_diurnal_std_hourly_mean.py<jupyter_output>TEMPLATE NAME: pr_IPSL-CM5A-LR_Jul_1997-1999_diurnal_avg.nc
Specifying latitude / longitude domain of interest ...
Preparing to write output to JSON file ...
Initializing dictionary of statistical results ...
['demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_diurnal_avg.nc']
PARAMS: [<pcmdi_metrics.diurnal.common.INPUT object at 0x7fd9e8929eb0>]
Specifying latitude / longitude domain of interest ...
Reading demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_diurnal_avg.nc ...
  Shape = (8, 20, 72)
Finding standard deviation over first dimension (time of day) ...
  Shape = (20, 72)
Finding r.m.s. average over 2nd-3rd dimensions (area) ...
For IPSL-CM5A-LR in Jul, average variance of hourly values = ( 2.15 mm/d)^2
Writing output to JSON file ...
KEYS AT END: ['DISCLAIMER', 'REFERENCE', 'RESULTS']
REG MASK: {}
done<jupyter_text>This generates the following file: `pr_Jul_1997-1999_std_of_meandiurnalcyc.json`  Fourier AnalysisAgain starting from the *composite* results let's do the fourier analysis. This uses a new parameter file.<jupyter_code>with open("basic_diurnal_fourierAllGrid.py") as f:
    print(f.read())
%%bash
fourierDiurnalAllGrid.py -p basic_diurnal_fourierAllGrid.py<jupyter_output>modpath  demo_output/diurnal/nc
filename_template  pr_%(model)_%(month)_%(firstyear)-%(lastyear)_diurnal_avg.nc
filename_template_LST  pr_%(model)_LocalSolarTimes.nc
LSTFILES: ['demo_output/diurnal/nc/pr_IPSL-CM5A-LR_LocalSolarTimes.nc']
TMPL pr_IPSL-CM5A-LR_LocalSolarTimes.nc
Reading demo_output/diurnal/nc/pr_IPSL-CM5A-LR_LocalSolarTimes.nc ... pr_IPSL-CM5A-LR_LocalSolarTimes.nc
====================
IPSL-CM5A-LR
====================
Reading time series of mean diurnal cycle ...
Input shapes:  (8, 36, 72) (8, 36, 72)
Getting latitude and longitude coordinates.
Taking fast Fourier transform of the mean diurnal cycle ...
Creating output arrays ...
Calling numpy FFT function ...
(8, 36, 72)
Converting from complex-valued FFT to real-valued amplitude and phase ...
  Output:
    cycmean (36, 72)
    maxvalue (3, 36, 72)
    tmax (3, 36, 72)
"Re-decorating" Fourier harmonics with grid info, etc., ...
... and writing to netCDF.<jupyter_text>This generates 3 files:```pr_IPSL-CM5A-LR_Jul_1997-1999_tmean.ncpr_IPSL-CM5A-LR_Jul_1997-1999_S.ncpr_IPSL-CM5A-LR_Jul_1997-1999_tS.nc``` We can now run the last script: `savg_fourierWrappedInOut.py`<jupyter_code>%%bash
savg_fourier.py -p basic_diurnal_std_hourly_mean.py<jupyter_output>Specifying latitude / longitude domain of interest ...
Preparing to write output to JSON file ...
Initializing dictionary of statistical results ...
TEMPLATE: pr_IPSL-CM5A-LR_Jul_1997-1999_S.nc
['demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_S.nc']
Reading Amplitude from demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_S.nc ...
Reading Phase from demo_output/diurnal/nc/pr_IPSL-CM5A-LR_Jul_1997-1999_tS.nc ...
Reading sftlf from demo_output/diurnal/nc/cmip5.IPSL-CM5A-LR.historical.r0i0p0.fx.atm.fx.sftlf.*.latestX.xml ...
Failed reading sftlf from file (error was: list index out of range)
Creating one for you
  Global mean land fraction = 0.252
Area-averaging globally, over land only, and over ocean only ...
Converting singleton transient variables to plain floating-point numbers ...
Jul 1-harmonic amplitude, phase =   0.583 mm/d,   9.681 hrsLST averaged globally
Jul 1-harmonic amplitude, phase =   2.243 mm/d,  11.588 hrsLST averaged over land
Jul 1-harmonic amplitude, phase = [...]
{'fraction_non_alphanumeric': 0.10363146209958769, 'fraction_numerical': 0.0675547098001903, 'mean_word_length': 5.253346554288547, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '131891', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 5404, 'n_tokens_neox': 4548, 'n_words': 1122, 'path': 'doc/jupyter/Demo/Demo_3_diurnal_cycle.ipynb', 'repo_name': 'jasonb5/pcmdi_metrics', 'size': '12612'}