<jupyter_start><jupyter_text>File Path to Weather Anomalies Data<jupyter_code>data_path="raw/SEAS5_anomalies"<jupyter_output><empty_output><jupyter_text>Loading Above Ground Biomass Data (Baseline)<jupyter_code>agb_data = xr.open_dataset("preprocess/agb_avitabile_2010-2016_v3.nc")<jupyter_output><empty_output><jupyter_text>Loading and Merging the files<jupyter_code>#Function to get all file names
def _get_file_list(root):
    files = [join(root, f) for f in listdir(root) if isfile(join(root, f))]
    dirs = [d for d in listdir(root) if isdir(join(root, d))]
    return files
list_files = sorted(_get_file_list(data_path))  #List of data file paths
weather_anomalies_data = xr.open_mfdataset(paths=list_files, parallel=True)<jupyter_output><empty_output><jupyter_text>Before pre-processing<jupyter_code>print(weather_anomalies_data)
print(weather_anomalies_data.to_dataframe().describe())
var_list =['t2m', 'd2m', 'swvl1', 'si10', 'fg10', 'tprate', 'erate']
for i in range(len(var_list)):
    plt.figure(figsize = (20,10)).suptitle(f'{var_list[i]} frequency distribution', fontsize=25)
    plt.hist(weather_anomalies_data[var_list[i]].values.flatten(), bins=1000, cumulative=False, log=True)<jupyter_output><empty_output><jupyter_text>Slicing data for time period 2010-2016<jupyter_code>weather_anomalies_data = weather_anomalies_data.sel(time=slice(weather_anomalies_data.time.values[0], weather_anomalies_data.time.values[83]))<jupyter_output><empty_output><jupyter_text>Re-Orienting longitude values<jupyter_code>weather_anomalies_data = weather_anomalies_data.assign_coords({'latitude':weather_anomalies_data.latitude.values,'longitude':xr.where(weather_anomalies_data.longitude > 180, weather_anomalies_data.longitude - 360, weather_anomalies_data.longitude) })<jupyter_output><empty_output><jupyter_text>Interpolating Data to match resolution of AGB<jupyter_code>weather_anomalies_data = weather_anomalies_data.interp({'latitude':agb_data.latitude.values,'longitude':agb_data.longitude.values})
print(weather_anomalies_data)<jupyter_output><xarray.Dataset>
Dimensions:    (latitude: 720, longitude: 1440, time: 84)
Coordinates:
  * time       (time) datetime64[ns] 2010-01-01 2010-02-01 ... 2016-12-01
  * latitude   (latitude) float32 -89.88 -89.62 -89.38 ... 89.38 89.62 89.88
  * longitude  (longitude) float32 -179.9 -179.6 -179.4 ... 179.4 179.6 179.9
Data variables:
    d2m        (time, latitude, longitude) float32 dask.array<chunksize=(1, 720, 1440), meta=np.ndarray>
    erate      (time, latitude, longitude) float32 dask.array<chunksize=(1, 720, 1440), meta=np.ndarray>
    fg10       (time, latitude, longitude) float32 dask.array<chunksize=(1, 720, 1440), meta=np.ndarray>
    si10       (time, latitude, longitude) float32 dask.array<chunksize=(1, 720, 1440), meta=np.ndarray>
    swvl1      (time, latitude, longitude) float32 dask.array<chunksize=(1, 720, 1440), meta=np.ndarray>
    t2m        (time, latitude, longitude) float32 dask.array<chunksize=(1, 720, 1440), meta=np.ndarray>
    tprate     (time, latitude, longi[...]<jupyter_text>After Pre-processing<jupyter_code>print(weather_anomalies_data.to_dataframe().describe())
var_list =['t2m', 'd2m', 'swvl1', 'si10', 'fg10', 'tprate', 'erate']
for i in range(len(var_list)):
    plt.figure(figsize = (20,10)).suptitle(f'{var_list[i]} frequency distribution', fontsize=25)
    plt.hist(weather_anomalies_data[var_list[i]].values.flatten(), bins=1000, cumulative=False, log=True)<jupyter_output><empty_output><jupyter_text>Saving Data<jupyter_code>random_date = "201" + str(np.random.randint(0,6)) + "-" + str(np.random.randint(1, 12)) + "-01T00:00:00.000000000"
weather_anomalies_data.sel(time=random_date).fg10.plot()
weather_anomalies_data.sel(time=random_date).swvl1.plot()
weather_anomalies_data.sel(time=random_date).tprate.plot()
weather_anomalies_data.sel(time=random_date).t2m.plot()
weather_anomalies_data.sel(time=random_date).d2m.plot()
weather_anomalies_data.sel(time=random_date).erate.plot()
weather_anomalies_data.sel(time=random_date).si10.plot()
weather_anomalies_data.to_netcdf("preprocess/weather_anomalies_2010-2016_v3.nc")<jupyter_output><empty_output><jupyter_text>FIle Size in MB<jupyter_code>round(os.stat("preprocess/weather_anomalies_2010-2016_v3.nc").st_size / (1024 * 1024), 3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1407733271590646, 'fraction_numerical': 0.06344061125260476, 'mean_word_length': 7.571428571428571, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '254903', 'license': 'Apache-2.0', 'n_tokens_mistral': 1891, 'n_tokens_neox': 1634, 'n_words': 234, 'path': 'notebooks/preprocess/preprocess_weather_anomalies.ipynb', 'repo_name': 'ecmwf-projects/ai-vegetation-fuel', 'size': '4319'}