<jupyter_start><jupyter_text>Reading ISMN dataThis example shows the basic functionality to read data downloaded from the Internatinal Soil Moisture Network (ISMN).The data is accessible at `http://ismn.geo.tuwien.ac.at` after registration. For this tutorial, data for the networks 'REMEDHUS', 'SMOSMANIA', 'FMI', 'WEGENERNET', 'GTK' and 'VAS' between 2009-08-04 and 2020-12-12 was downloaded. The suggested class for reading is called `ISMN_Interface`. It provides functions to access single networks, stations and sensors and the measured time series for each sensor. The `ISMN_Interface` class takes either the directory where the extracted ISMN files are stored, or a zip archive of the data directly (reading from zip is significantly slower).To read only a selection of networks stored in the passed data path, you can pass a list of network names.<jupyter_code>from ismn.interface import ISMN_Interface
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Enter the path to your ISMN data
path_to_ismn_data = "C:\Temp\delete_me\ismn\stick\Data_separate_files_20090804_20201212.zip"

ismn_data = ISMN_Interface(path_to_ismn_data, network=['REMEDHUS', 'SMOSMANIA', 'FMI', 'WEGENERNET', 'GTK', 'VAS', 'RSMN'])<jupyter_output>Found existing ismn metadata in C:\Temp\delete_me\ismn\stick\python_metadata\Data_separate_files_20090804_20201212.csv.<jupyter_text>The following command will initialise the reader. The first time this is done, metadata for each sensor will we collected. This will iterate through all files and detect e.g. station names, time coverage, measurement depths etc. Metadata collection can take a few minutes and will start multiple parallel processes (depending on the available ressources). By default the `python_metadata` (which contains the collected metadata as a .csv file) will be placed in the passed root directory. The next time the reader is created it will use `python_metadata` (if it is found) instead of generating it again.Note: When changing the data (e.g. if folders are added or removed from the data collection) make sure to delete the `python_metadata` folder to re-generate for the new data.You can define a different path, where the metadata is stored, resp. looked for by passing a `meta_path` when initialsing `ISMN_Interface`. You can also define a list of network names that are considered when reading the data (this will not affect metadata generation, which is performed for ALL networks in `data_path`). Passing `keep_loaded_data=True` means that all time series, once read, will be kept in memory for faster subsequent access. This can fill up your memory and is only recommended for small data samples.In this example we use the default metadata path.  Accessing data components Network Collection `ISMN_Interface` contains a `collection` of active ISMN networks. The collection lists the network name and names of all stations in a network (either all Networks if no network(s) was/were specified during initialisation or the selected networks).<jupyter_code>collection = ismn_data.collection
collection<jupyter_output><empty_output><jupyter_text>The collection also contains a grid object that contains the locations of all **stations** in all active networks. For more details see https://github.com/TUW-GEO/pygeogrids<jupyter_code>import pandas as pd
grid = collection.grid
gpis, lons, lats = grid.get_grid_points()
display(pd.DataFrame(index=pd.Index(gpis, name='gpi'), data={'lon': lons, 'lat': lats}).T)<jupyter_output><empty_output><jupyter_text>Using the GPI or coordinates, a station from **all** stations in **all** networks in the collection can be selected.<jupyter_code>station, dist = collection.get_nearest_station(27.0, 68.0)
assert collection.station4gpi(0) == station # same result when selecting with GPI

print(f"Station '{station.name}' at Lon: {station.lon}°, Lat: {station.lat}°")
station<jupyter_output>Station 'SAA111' at Lon: 27.550620000000002°, Lat: 68.33019°<jupyter_text>Network A single network from the collection can be accessed via its name.<jupyter_code>network = collection['SMOSMANIA']
network<jupyter_output><empty_output><jupyter_text>Station A network consists of multiple stations, multiple variables can be measured by different sensors at a station:<jupyter_code>station = network.stations['SaintFelixdeLauragais']
station<jupyter_output><empty_output><jupyter_text>Similar as the single sensors, each station has a metadata attribute. The station metadata contains all meta variables from all sensors that measure at the station. Formatting options for MetaData are either as a DataFrame (`to_pd()`) or as a dictionary (`to_dict()`) of form:```{name: [(value, depth_from, depth_to), ...], ...}```<jupyter_code>from pprint import pprint
pprint(station.metadata.to_dict())<jupyter_output>{'clay_fraction': [(22.8, 0.05, 0.05),
                   (22.4, 0.1, 0.1),
                   (23.9, 0.2, 0.2),
                   (29.4, 0.3, 0.3)],
 'climate_KG': [('Cfb', None, None)],
 'climate_insitu': [('unknown', None, None)],
 'elevation': [(337.0, None, None)],
 'instrument': [('PT-100', 0.05, 0.05),
                ('PT-100', 0.1, 0.1),
                ('PT-100', 0.2, 0.2),
                ('PT-100', 0.3, 0.3),
                ('ThetaProbe-ML2X', 0.05, 0.05),
                ('ThetaProbe-ML2X', 0.1, 0.1),
                ('ThetaProbe-ML2X', 0.2, 0.2),
                ('ThetaProbe-ML2X', 0.3, 0.3),
                ('ThetaProbe-ML3', 0.2, 0.2)],
 'latitude': [(43.4417, None, None)],
 'lc_2000': [(10.0, None, None)],
 'lc_2005': [(10.0, None, None)],
 'lc_2010': [(10.0, None, None)],
 'lc_insitu': [('unknown', None, None)],
 'longitude': [(1.88, None, None)],
 'network': [('SMOSMANIA', None, None)],
 'organic_carbon': [(1.15, 0.05, 0.05),
                    (0.84, 0.1, 0.1),
 [...]<jupyter_text>Sensor Accessing sensors at a station works similar to accessing stations in a network. By default the name is created from the instrument type, the measurued variable and the depth layer that the senosor measures in.<jupyter_code>sensor = station['ThetaProbe-ML2X_soil_moisture_0.050000_0.050000']
sensor<jupyter_output><empty_output><jupyter_text>A data file is assigned to each sensor, that contains the sensor variable time series and quality flags.<jupyter_code>ts = sensor.read_data()
print(sensor.metadata)
ax = ts.plot(figsize=(12,4))
ax.set_xlabel("Time [year]")
ax.set_ylabel("Soil Moisture [$m^3 m^{-3}$]")
display(ts)<jupyter_output>MetaData([
  MetaVar([clay_fraction, 22.8, Depth([0.05, 0.05])]),
  MetaVar([climate_KG, Cfb, None]),
  MetaVar([climate_insitu, unknown, None]),
  MetaVar([elevation, 337.0, None]),
  MetaVar([instrument, ThetaProbe-ML2X, Depth([0.05, 0.05])]),
  MetaVar([latitude, 43.4417, None]),
  MetaVar([lc_2000, 10.0, None]),
  MetaVar([lc_2005, 10.0, None]),
  MetaVar([lc_2010, 10.0, None]),
  MetaVar([lc_insitu, unknown, None]),
  MetaVar([longitude, 1.88, None]),
  MetaVar([network, SMOSMANIA, None]),
  MetaVar([organic_carbon, 1.15, Depth([0.05, 0.05])]),
  MetaVar([sand_fraction, 43.5, Depth([0.05, 0.05])]),
  MetaVar([saturation, 0.44, Depth([0.05, 0.05])]),
  MetaVar([silt_fraction, 33.7, Depth([0.05, 0.05])]),
  MetaVar([station, SaintFelixdeLauragais, None]),
  MetaVar([timerange_from, 2009-08-04 00:00:00, None]),
  MetaVar([timerange_to, 2019-01-01 00:00:00, None]),
  MetaVar([variable, soil_moisture, Depth([0.05, 0.05])])
])<jupyter_text>Additionally, metadata is assigned to each sensor. Some metadata is sensor specific (e.g. soil properties, time series range), some depends on the location of the station and is therefore shared by multiple sensors (landcover and climate classes etc.).The first value in each Variable is the name of the metadata variable, the second the actual value for the variable. The third value (tuple) is the depth that the value applies to (for soil properties multiple layers are provided together with the ISMN data, during metadata generation the best matching depth for a sensor is selected).<jupyter_code>sensor.metadata.to_pd()<jupyter_output><empty_output><jupyter_text>Other functions  Find network for a specific station`ISMN_Interface` provides a function to find the network when only the name of a station is known.<jupyter_code>ismn_data.network_for_station('SAA111', name_only=False)<jupyter_output><empty_output><jupyter_text>Read via index You can filter the dataset a priori and get ids of sensors that measure a specific variable. The id can then be used to read the data directly.<jupyter_code>ids = ismn_data.get_dataset_ids(variable='soil_temperature', max_depth=1, filter_meta_dict={'lc_2005': 130, 'climate_KG': 'Csb'})
print(ids)
ts, meta = ismn_data.read(ids[1], return_meta=True)
pprint(meta)
ax = ts.plot(figsize=(12,4), title=f'Time series for ID {ids[1]}')
ax.set_xlabel("Time [year]")
ax.set_ylabel("Soil Temp. [°C]")<jupyter_output>{'clay_fraction': [(20.2, 0.3, 0.3)],
 'climate_KG': [('Csb', None, None)],
 'climate_insitu': [('unknown', None, None)],
 'elevation': [(318.0, None, None)],
 'instrument': [('PT-100', 0.3, 0.3)],
 'latitude': [(44.222, None, None)],
 'lc_2000': [(130.0, None, None)],
 'lc_2005': [(130.0, None, None)],
 'lc_2010': [(130.0, None, None)],
 'lc_insitu': [('unknown', None, None)],
 'longitude': [(4.34483, None, None)],
 'network': [('SMOSMANIA', None, None)],
 'organic_carbon': [(4.52, 0.3, 0.3)],
 'sand_fraction': [(30.3, 0.3, 0.3)],
 'saturation': [(0.62, 0.3, 0.3)],
 'silt_fraction': [(49.5, 0.3, 0.3)],
 'station': [('Mejannes-le-Clap', None, None)],
 'timerange_from': [(Timestamp('2009-08-04 00:00:00'), None, None)],
 'timerange_to': [(Timestamp('2019-01-01 00:00:00'), None, None)],
 'variable': [('soil_temperature', 0.3, 0.3)]}<jupyter_text>Station locations for a specific variable can be visualised on a map. If a min/max depth is passed, only stations with a sensor that measures within the passed range are included.<jupyter_code>import cartopy.crs as ccrs
#plot available station on a map
fig, axs = plt.subplots(1, 2, figsize=(16,10), subplot_kw={'projection': ccrs.Robinson()})
ismn_data.plot_station_locations('soil_moisture', min_depth=0., max_depth=0.1, ax=axs[0])
ismn_data.plot_station_locations('soil_temperature', min_depth=0.5, ax=axs[1])
plt.show()<jupyter_output><empty_output><jupyter_text>Selecting and interating over data It is often desired to iterate over all sensors that fulfill certain conditions (e.g. that measure soil moisture in a certain depth, or for a certain landcover class). For these cases the `collection` (and other components) provides iterators that take keywords and values for filtering the loaded networks/stations/sensor while iterating over single time series (of a collection, a network, or a station).  Select  by variable and depthIn this example we iterate over all sensors in the previously loaded collection (i.e. over all active networks) that measure 'soil_moisture' in any depth (range) between 0 and 0.05 metres.<jupyter_code>for network, station, sensor in ismn_data.collection.iter_sensors(variable='soil_moisture', 
                                                                  depth=[0., 0.05]):
    data = sensor.read_data()
    display(station)
    print('\033[1m' + f'Metadata for sensor {sensor}:'+ '\033[0m')
    display(sensor.metadata.to_pd())
    ax = data.plot(figsize=(12,4), title=f'Time series for sensor {sensor.name}')
    ax.set_xlabel("Time [year]")
    ax.set_ylabel("Soil Moisture [$m^3 m^{-3}$]")
    break # for this example we stop after the first sensor<jupyter_output><empty_output><jupyter_text>Selecting by variable and other metadata (1)In this example we iterate over all sensors for the network 'RMSN' and filter those that measure precipitation within an ESA CCI Landcover pixel that is marked as 'Cropland, rainfed' (10) or 'Grassland' (130).<jupyter_code>ismn_data.print_landcover_dict()
for station, sensor in ismn_data.collection['RSMN'].iter_sensors(variable='precipitation', 
                                                                 filter_meta_dict={'lc_2010': [10, 130]}):
    data = sensor.read_data()
    metadata = sensor.metadata
    display(station)
    print('\033[1m' + f'Metadata for sensor {sensor}:' + '\033[0m')
    display(metadata.to_pd())
    ax = data.plot(figsize=(12,4), title=f'Time series for sensor {sensor.name}')
    ax.set_xlabel("Time [year]")
    ax.set_ylabel("Precipitation [mm]")
    break # for this example we stop after the first sensor<jupyter_output><empty_output><jupyter_text>Selecting by variable, depth and metadata (2)In this example we iterate over all sensors in the collection and filter those that measure soil_moisture between 0 and 10 cm within an ESA CCI Landcover pixel that is marked as 'Cropland, rainfed' (10) or 'Grassland' (130), and has one of the follwing climate classes assigned: Csc, Cfa, Dfc. In addition we set all those soil moisture values that are **not** flagged as 'good' (G) to NaN.<jupyter_code>display(ismn_data.print_climate_dict())
from ismn.meta import Depth
for network, station, sensor in ismn_data.collection \
    .iter_sensors(variable='soil_moisture',
                  depth=Depth(0.,0.05),
                  filter_meta_dict={'lc_2010': [10, 130],
                                    'climate_KG':['Csc', 'Cfa', 'Dfc']}):
    
    data = sensor.read_data()
    data.loc[data['soil_moisture_flag'] != 'G', 'soil_moisture'] = np.nan
    metadata = sensor.metadata
    display(network)
    display(station)
    print('\033[1m' + f'Metadata for sensor {sensor}:'+ '\033[0m')
    display(metadata.to_pd())
    ax = data.plot(figsize=(12,4), title=f"G-flagged SM for '{sensor.name}' at station '{station.name}' in network '{network.name}''")
    ax.set_xlabel("Time [year]")
    ax.set_ylabel("Soil Moisture [$m^3 m^{-3}$]")
    break # for this example we stop after the first sensor<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12800575953923687, 'fraction_numerical': 0.04204463642908567, 'mean_word_length': 4.747207281754241, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '53289', 'license': 'MIT', 'n_tokens_mistral': 4859, 'n_tokens_neox': 4385, 'n_words': 1460, 'path': 'docs/read_and_plot_ismn_data/.ipynb_checkpoints/interface-checkpoint.ipynb', 'repo_name': 'awst-austria/ismn', 'size': '13890'}