<jupyter_start><jupyter_text>Preprocessing Narval**This data was not used to train the NNs**Converting the data into npy makes it possible for us to work with it efficiently; originally we require 500GB of RAM which is always difficult to guarantee. We preprocess QUBICC in another ipynb notebook precisely because of this issue.1) We read the data2) Reshape variables so that they have equal dimensionality3) Remove data above 21kms4) Reshape into data samples fit for the NN5) Split into input and output6) Save as npy in float32Note: We neither scale nor split the data into training/validation/test sets. The reason is that i) in order to scale we need the entire dataset but this can only be done in conjunction with the Qubicc dataset. Also for cross-validation different scalings will be necessary based on different subsets of the data, ii) The split into subsets will be done by the cross-validation procedure or not at all when training the final model.<jupyter_code>import sys
import xarray as xr
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import time
import gc
# import importlib
# importlib.reload(my_classes)

base_path = '/pf/b/b309170'
output_path = base_path + '/my_work/icon-ml_data/cloud_cover_parameterization/grid_column_based_QUBICC_R02B05/based_on_var_interpolated_data'

# Add path with my_classes to sys.path
sys.path.insert(0, base_path + '/workspace_icon-ml/cloud_cover_parameterization/')

# Which days to load
days_narval = 'all'

from my_classes import load_data

VERT_LAYERS = 31

# Set output_var to one of {'clc', 'cl_area'}
output_var = 'cl_area'<jupyter_output><empty_output><jupyter_text>1) Reading the data Input:- fr_land: Fraction of land- zg: Geometric height at full levels (3D)- qv: Specific water vapor content (3D)- qc: Specific cloud water content (3D)- qi: Specific cloud ice content (3D)- temp: Temperature (3D)- pres: Pressure (3D)$186$ $( = 1+24[zf]+26[q_c]+27\cdot 5$) input nodes Output:- clc: Cloud Cover$27$ output nodesThe data above 21km is capped.<jupyter_code># For cl_area I only need the output as I already have the input
# I still need 'qc', 'qi', 'clc' for condensate-free clouds
# If I were to use 'cl_area' for condensate-free clouds I would get an estimate 
# which is slightly different due to coarse-graining

order_of_vars_narval = ['qv', 'qc', 'qi', 'temp', 'pres', 'zg', 'fr_land', output_var]
# Load NARVAL data
data_dict = load_data(source='narval', days=days_narval, resolution='R02B05', 
                             order_of_vars=order_of_vars_narval)
if output_var == 'clc':
    # Are there any bad data points
    ta_is_0 = np.where(data_dict['temp'] == 0)
    for i in range(3):
        assert ta_is_0[i].size == 0

    del ta_is_0
    gc.collect()<jupyter_output><empty_output><jupyter_text>Counting fraction of condensate-free clouds I'll leave them in the training data for the column-based and region-based models. The reason is that we would have to remove quite a lot around the given grid cell. I can do that in the grid-cell based models.<jupyter_code># clouds = 0
# count_cond_free_clouds = 0
# for i in range(data_dict['clc'].shape[0]):
#     for j in range(data_dict['clc'].shape[1]):
#         for k in range(data_dict['clc'].shape[2]):
#             if (data_dict['clc'][i,j,k] > 0 and data_dict['qc'][i,j,k] + data_dict['qi'][i,j,k] == 0):
#                 count_cond_free_clouds += 1
#             if (data_dict['clc'][i,j,k] > 0):
#                 clouds += 1
# count_cond_free_clouds/clouds
for key in data_dict.keys():
    print(key, data_dict[key].shape)
(TIME_STEPS, VERT_LAYERS, HORIZ_FIELDS) = data_dict['clc'].shape
try:
    #Reshaping into nd-arrays of equaling shapes (don't reshape in the vertical)
    data_dict['zg'] = np.repeat(np.expand_dims(data_dict['zg'], 0), TIME_STEPS, axis=0)
    data_dict['fr_land'] = np.repeat(np.expand_dims(data_dict['fr_land'], 0), TIME_STEPS, axis=0)
except:
    pass
# One sample should contain a column of information
data_dict_reshaped = {}

for key in data_dict.keys():
    if data_dict[key].shape[1] == VERT_LAYERS:  
        # Removing data above 21kms
        for i in range(4, VERT_LAYERS):
            new_key = '{}{}{:d}'.format(key,'_',(i+17)) # Should start at 21
            data_dict_reshaped[new_key] = np.reshape(data_dict[key][:,i,:], -1)
    else:
        data_dict_reshaped[key] = np.reshape(data_dict[key], -1)
#Converting dict into a DataFrame-object 
df = pd.DataFrame.from_dict(data_dict_reshaped)
df.head()
#Modifies df as well
def split_input_output(dataset):
    output_df = pd.DataFrame()
    for i in range(21, 48):
        output_df['cl_area_%d'%i] = dataset['cl_area_%d'%i] # Should start at 21
        del dataset['cl_area_%d'%i]
    return output_df
output_df = split_input_output(df)
# Save the data
if output_var == 'clc':
    np.save(output_path + '/cloud_cover_input_narval.npy', np.float32(df))
    np.save(output_path + '/cloud_cover_output_narval.npy', np.float32(output_df))
elif output_var == 'cl_area':
    np.save(output_path + '/cloud_area_output_narval.npy', np.float32(output_df))
# Test
if output_var == 'cl_area':
    old_input = np.load(output_path + '/cloud_cover_input_narval.npy')
    # If this yields True once then we're done
    for i in range(old_input.shape[1]):
        print(np.all(old_input[:,i] == df['qi_25']))<jupyter_output>False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
True
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
{'fraction_non_alphanumeric': 0.09380922299431459, 'fraction_numerical': 0.015319014529374606, 'mean_word_length': 4.431389365351629, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '40518', 'license': 'MIT', 'n_tokens_mistral': 2218, 'n_tokens_neox': 2129, 'n_words': 729, 'path': 'q2_column_based_qubicc_r2b5/source_code/preprocessing_narval.ipynb', 'repo_name': 'agrundner24/iconml_clc', 'size': '6332'}