<jupyter_start><jupyter_text>Working with Datafiles Let's import the FileReport class.<jupyter_code>from PyDatSci.FileReport import FileReport
import PyDatSci.FileReport as fr
import pickle<jupyter_output><empty_output><jupyter_text>Let's read the file report we created in the `FileReport` notebook:<jupyter_code>report = FileReport.read('/work/ch0636/eddy/pool/sims/cmip5/file-report-cmip5.fr')
print(report)<jupyter_output>FileReport of /work/ch0636/eddy/pool/sims/cmip5

 Number of Files               : 12034
 Number of valid Files         : 12031
 Number of Symlinks            : 0
 Number of missing Symlinks    : 3
 
 Number of Suffixes            : 11
               .status         : 2
               .nc             : 11902
               .py             : 59
               .p              : 9
               .txt            : 16
               .csv            : 2
               .png            : 31
               .sh             : 2
               .csv#           : 2
               no suffix       : 1
               .gr             : 5<jupyter_text>Scan for NetCDF attributes The `FileReport` acutally only contains huge lists of filenames. In the next step, we want to investigate NetCDF files. For this, we use the `DataFile` class. This is basically a container which stores some meta information of the actual file and it's NetCDF attributes, so we can easily access and store them later. To fill this container, we have to scan the NetCDF files from our `FileReport`. This might take some time (especially from a Python Notebook!), but it has to be done just once. So, if it takes to much time, you can skip this step and directly continue the `Sorting` session by reading a pickle dump which i prepared for `CMIP5`.<jupyter_code>datafiles = fr.scan_ncattrs(report['.nc'])<jupyter_output>Scanning for NetCDF Attributes...
Progress: |██████████████████████████████████████████████████| 100.0% Complete<jupyter_text>We quickly dump the datafiles list which now contains NetCDF attributes to a file.<jupyter_code>pickle.dump(datafiles, open('/work/ch0636/eddy/pool/sims/cmip5/datafiles-cmip5','wb'))<jupyter_output><empty_output><jupyter_text>Sorting Reload our datafiles:<jupyter_code>datafiles = pickle.load(open('/work/ch0636/eddy/pool/sims/cmip5/datafiles-cmip5','rb'))<jupyter_output><empty_output><jupyter_text>We can create a dictionary containing an arbitray sorting order for the datafiles, e.g.:<jupyter_code>sorted_files = fr.sort_by_attrs(datafiles, ['institute_id'])<jupyter_output><empty_output><jupyter_text>There is a little helper routine which can print this dictionarys keys a little nice, so let's import is, and see what we found.<jupyter_code>from PyDatSci.tools import print_dict
print_dict(sorted_files)<jupyter_output>-> NASA-GISS
-> ICHEC
-> NSF-DOE-NCAR
-> MIROC
-> NOAA GFDL
-> BNU
-> MOHC
-> BCC
-> CSIRO-BOM
-> no institute_id
-> MRI
-> NCC
-> IPSL
-> INM
-> CNRM-CERFACS
-> NCAR
-> LASG-CESS
-> CMCC
-> CSIRO-QCCCE
-> CCCma
-> MPI-M<jupyter_text>Ok, there are files without an `institute_id`! Let's check them out:<jupyter_code>for datafile in sorted_files['no institute_id']:
    print(datafile)<jupyter_output>/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2083.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2070.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2075.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/maxFile_TColdP_2095.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2076.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2065.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/maxFile_TColdP_2092.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2092.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2079.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2071.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2072.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2067.nc
/work/ch0636/eddy/pool/sims/[...]<jupyter_text>These are obviously no files that are supposed to conform with cmor standard. Let's sort our files a little more accurately, e.g.<jupyter_code>sorted_files = fr.sort_by_attrs(datafiles, ['institute_id','model_id','experiment_id','frequency','varname'])<jupyter_output><empty_output><jupyter_text>The dictionary `sorted_files` now contains all datafiles sorted by a list of NetCDF attributes. To check out it's contents, you can print the dictionary using:<jupyter_code>print_dict(sorted_files)<jupyter_output>-> NASA-GISS
-----> GISS-E2-R
---------> rcp45
-------------> day
-----------------> tas
-----------------> tasmin
-----------------> tasmax
-----------------> pr
---------> historical
-------------> fx
-----------------> sftlf
-----------------> orog
-------------> day
-----------------> tas
-----------------> tasmin
-----------------> tasmax
-----------------> pr
-> ICHEC
-----> EC-EARTH
---------> rcp85
-------------> day
-----------------> prmd
-----------------> tas
-----------------> tasmax-mm-ymax
-----------------> pr
-----------------> prmd-ymm
-----------------> psl
-----------------> tasmax
-----------------> tasmin
---------> rcp45
-------------> day
-----------------> prmd
-----------------> tas
-----------------> tasmax-mm-ymax
-----------------> pr
-----------------> prmd-ymm
-----------------> psl
-----------------> tasmax
-----------------> tasmin
---------> rcp26
-------------> day
-----------------> tas
-----------------> pr
---------> historical
-------------> fx
--[...]<jupyter_text>You can see, that this gives an overview of the available files depending on their NetCDF attributes that we defined above. Another way, to walk through the dictionary, is to check out the keys of the recursive dictionary: Check out what `institude_id`s are available:<jupyter_code>print(sorted_files.keys())<jupyter_output>dict_keys(['NASA-GISS', 'ICHEC', 'NSF-DOE-NCAR', 'MIROC', 'NOAA GFDL', 'BNU', 'MOHC', 'BCC', 'CSIRO-BOM', 'no institute_id', 'MRI', 'NCC', 'IPSL', 'INM', 'CNRM-CERFACS', 'NCAR', 'LASG-CESS', 'CMCC', 'CSIRO-QCCCE', 'CCCma', 'MPI-M'])<jupyter_text>Assume we are interested in the files with `institute_id='MPI-ESM'`. Let's see what kind of `model_id`s we get:<jupyter_code>print(sorted_files['MPI-M'].keys())<jupyter_output>dict_keys(['MPI-ESM-LR', 'MPI-ESM-P', 'MPI-ESM-MR'])<jupyter_text>... and so on ...<jupyter_code>print(sorted_files['MPI-M']['MPI-ESM-LR'].keys())
print(sorted_files['MPI-M']['MPI-ESM-LR']['historical'].keys())
print(sorted_files['MPI-M']['MPI-ESM-LR']['historical']['day'].keys())<jupyter_output>dict_keys(['TWarmP', 'tasmin', 'rsus', 'rlds', 'pr', 'tasmin-mm-ymin', 'PNP95thYWD', 'rlus', 'TColdP', 'ws', 'hfls', 'prmd-ymm', 'PDry95P', 'hus', 'psl', 'evap', 'tasmax-mm-ymax', 'tas', 'runoff', 'vas', 'rsds', 'uas', 'P95thYWD', 'prmd', 'sfcWind', 'tasmax'])<jupyter_text>Now, let's see what the dictionary actually contains for us:<jupyter_code>print(sorted_files['MPI-M']['MPI-ESM-LR']['historical']['day']['pr'])<jupyter_output>[<PyDatSci.DataFile.DataFile object at 0x2adac252d278>, <PyDatSci.DataFile.DataFile object at 0x2adac741fbe0>, <PyDatSci.DataFile.DataFile object at 0x2adac9eae860>]<jupyter_text>We can see that the file list contains objects of type `DataFile`. This is basically a container which stores some meta information of the actual file and it's NetCDF attributes. Let's print the filenames:<jupyter_code>for datafile in sorted_files['MPI-M']['MPI-ESM-LR']['historical']['day']['pr']:
    print(datafile.filename)<jupyter_output>/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/day/pr/pr_day_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/year/pr/pr_year_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/mon/pr/pr_mon_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc<jupyter_text>We can recognize, that `pr_mon_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc` and `pr_year_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc` actually have inconsistent netcdf attributes and filenames. Their filenames indicate different frequencies than the NetCDF attribute in the file. We can check this, since we stored the NetCDF attributes in the `DataFile` container, e.g.<jupyter_code>for datafile in sorted_files['MPI-M']['MPI-ESM-LR']['historical']['day']['pr']:
    print(datafile.filename)
    print(datafile.frequency)<jupyter_output>/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/day/pr/pr_day_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc
day
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/year/pr/pr_year_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc
day
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/mon/pr/pr_mon_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc
day<jupyter_text>So we see that these files, in fact, have inconsistencies. Let's check out the history of the file:<jupyter_code>yearly_file = sorted_files['MPI-M']['MPI-ESM-LR']['historical']['day']['pr'][0]
print(yearly_file.history)<jupyter_output>Wed May 07 17:17:57 2014: cdo mergetime pr_day_MPI-ESM-LR_historical_r1i1p1_18500101-18591231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_18600101-18691231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_18700101-18791231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_18800101-18891231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_18900101-18991231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_19000101-19091231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_19100101-19191231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_19200101-19291231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_19300101-19391231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_19400101-19491231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_19500101-19591231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_19600101-19691231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_19700101-19791231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_19800101-19891231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_19900101-19991231.nc pr_day_MPI-ESM-LR_historical_r1i1p1_20000101-20051231.nc cat/pr_day_MPI-ESM-LR_historical_r1i1p1_18500101[...]<jupyter_text>The `yearly_file` is the result from several cdo commands, that created yearly mean data from a daily input frequency and inherited the attribute `frequency=day` from those. This might become a problem, if we want to filter data files. So let's check this out! We will now walk through the whole datafile list and check if we don't find the frequency in the filename:<jupyter_code># the lists in which we collect some inconsistent
inconsistent_freq  = []
missing_freq       = []
# now let's loop through the datafiles list (containing all CMIP5 datafiles).
for datafile in datafiles:
    # if the datafile has a frequency attribute, we'll check if
    # it's found in the filename or path.
    if hasattr(datafile,'frequency'):
        if datafile.frequency not in datafile.filename_str:
            inconsistent_freq.append(datafile)
    # we also collect datafiles that have no frequency attribute at all.
    else:
        missing_freq.append(datafile)

print('number of files without frequency: {}'.format(len(missing_freq)))
print('number of files with inconsistent frequency: {}'.format(len(inconsistent_freq)))<jupyter_output>number of files without frequency: 34
number of files with inconsistent frequency: 3768<jupyter_text>We'll have a look at the files that have no frequency attribute:<jupyter_code>for datafile in missing_freq:
    print(datafile)<jupyter_output>/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2083.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2070.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2075.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/maxFile_TColdP_2095.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2076.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2065.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/maxFile_TColdP_2092.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2092.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2079.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2071.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2072.nc
/work/ch0636/eddy/pool/sims/cmip5/IPSL-CM5A-MR/day/TColdP/minFile_TColdP_2067.nc
/work/ch0636/eddy/pool/sims/[...]<jupyter_text>Here they are again, some files that are probably not supposed to conform cmor standard. By the way, all attributes of a file are contained in a dictionary, that you can access by:<jupyter_code>print(missing_freq[0].ncdict.keys())<jupyter_output>dict_keys(['Conventions', 'history', 'CDI'])<jupyter_text>It seems that these are just some temporay files, so we will ignore them for now. It's more interesting to look at those files that have inconsistencies in their frequency. We collected them earlier, but there were quite a lot...<jupyter_code>print(len(inconsistent_freq))<jupyter_output>3768<jupyter_text>It's no use to look at that many filenames, so let's sort them again, maybe by frequency and variable name...<jupyter_code>sorted_inconsistent_freq_dict = fr.sort_by_attrs(inconsistent_freq, ['frequency','varname'])
print_dict(sorted_inconsistent_freq_dict)<jupyter_output>-> day
-----> TWarmP
-----> tasmin
-----> rsus
-----> rlds
-----> pr
-----> tasmin-mm-ymin
-----> PNP95thYWD
-----> rlus
-----> TColdP
-----> ws
-----> PDry95P
-----> hus
-----> psl
-----> evap
-----> tasmax-mm-ymax
-----> tas
-----> runoff
-----> vas
-----> rsds
-----> uas
-----> P95thYWD
-----> average_DT
-----> prmd
-----> sfcWind
-----> tasmax<jupyter_text>We see that the inconsistencies are only in the `day` frequency. Let's see, how many files we find for `pr`:<jupyter_code>print(len(sorted_inconsistent_freq_dict['day']['pr']))<jupyter_output>253<jupyter_text>Hmm, still quite a lot. It's time for a filter! Instead of sorting files, we can also filter them by any number of attributes using the `filter_by_attrs` function in the `FileReport` module:<jupyter_code>inconsistent_freq_mpi_pr = fr.filter_by_attrs(sorted_inconsistent_freq_dict['day']['pr'], model_id='MPI-ESM-LR')
print(len(inconsistent_freq_mpi_pr))<jupyter_output>8<jupyter_text>Ok, that seems fine. Let's have a look.<jupyter_code>for datafile in inconsistent_freq_mpi_pr:
    print(datafile)
    print(datafile.frequency)<jupyter_output>/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/year/pr/pr_year_MPI-ESM-LR_rcp26_r1i1p1_20060101-23001231.nc
day
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/year/pr/pr_year_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc
day
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/year/pr/pr_year_MPI-ESM-LR_rcp45_r1i1p1_20060101-23001231.nc
day
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/year/pr/pr_year_MPI-ESM-LR_rcp85_r1i1p1_20060101-23001231.nc
day
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/mon/pr/pr_mon_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc
day
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/mon/pr/pr_mon_MPI-ESM-LR_rcp45_r1i1p1_20060101-23001231.nc
day
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/mon/pr/pr_mon_MPI-ESM-LR_rcp26_r1i1p1_20060101-23001231.nc
day
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/mon/pr/pr_mon_MPI-ESM-LR_rcp85_r1i1p1_20060101-23001231.nc
day<jupyter_text>We can see what we investigated earlier. There are some files, that have inconsistencies between file or pathnames and NetCDF attributes. This might be a problem, if we want to use the `DataFile` class relying on `NetCDF` attributes. We could also, of course, draw meta information from the filename, but that seems insecure,  if we have to rely on a certain file naming convention. This was an example of how you can do some checks, investigation and quality management using sorting and filtering with the `DataFile` class. As quickly as you can find some inconsistencies and check stuff, as quickly you could now correct them!  Filtering The sorting gives a good overview, of what data is available, like when you sort your books by author, title, genre, etc.. On the other hand, definining a filter and apply it on the datafiles can give you powerful direct access to what you want, e.g.,<jupyter_code>attr_filter = {'institute_id':'MPI-M', 'model_id':'MPI-ESM-LR', 'experiment_id':'historical', 'frequency':'day', 'varname':'pr'}
filtered = fr.filter_by_attrs(datafiles, **attr_filter)<jupyter_output><empty_output><jupyter_text>which will results in the same list of files as above:<jupyter_code>for datafile in filtered:
    print(datafile.filename)<jupyter_output>/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/day/pr/pr_day_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/year/pr/pr_year_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc
/work/ch0636/eddy/pool/sims/cmip5/MPI-ESM-LR/mon/pr/pr_mon_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc<jupyter_text>Let's see if there is any data from GERICS!<jupyter_code>gerics_files = fr.filter_by_attrs(datafiles, institute_id='GERICS')
print(gerics_files)<jupyter_output>[]
{'fraction_non_alphanumeric': 0.15978695073235685, 'fraction_numerical': 0.06663578996121114, 'mean_word_length': 3.9910430511412884, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 186, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 1}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '414795', 'license': 'MIT', 'n_tokens_mistral': 7500, 'n_tokens_neox': 6625, 'n_words': 1232, 'path': 'ncdata/notebooks/02-sort-and-filter.ipynb', 'repo_name': 'larsbuntemeyer/ncdata', 'size': '17273'}