<jupyter_start><jupyter_text>Beginner workflowThis tutorial is a beginner workflow for processing data, visualising the object store and retrieving and visualising data. Check installationThis tutorial assumes that you have installed `openghg`. To ensure install has been successful you can open an `ipython` console and import openghgIn a terminal type```bashipython```Then import `openghg` and print the version string associated with the version you have installed. If you get something like the below `openghg` is installed correctly.```ipythonIn [1]: import openghgIn [2]: openghg.__version__Out[2]: '0.0.1'```If you get an ``ImportError`` please go back to the install section of the documentation. NotebooksIf you haven't used Jupyter notebooks before please see [this introduction](https://realpython.com/jupyter-notebook-introduction/).  1. Setting up our environmentFirst the notebook sets up the environment needed to create the object store at our desired location. By default this locationis at ``/tmp/openghg_store``. For the purposes of this tutorial this path is fine but as it is a temporary directory it may not survive areboot of the computer. If you want to create an object store that survives a reboot you can change the path to anything you like. Werecommened a path such as ``~/openghg_store`` which will create the object store in your home directory in a directory called ``openghg_store``.<jupyter_code>from openghg.store import ObsSurface
from openghg.objectstore import visualise_store
from openghg.localclient import get_obs_surface, RankSources
from openghg.retrieve import search
from openghg.util import bilsdale_datapaths

import glob
from pathlib import Path

import os
import tempfile<jupyter_output><empty_output><jupyter_text>Set an environment variable for the OpenGHG object store Here we create a temporary directory but you can use any folder you like by setting a path in place of `tmp_dir.name`. The object store created by this notebook will only have a lifetime as long as the notebook, if you want to create a longer lived object store set a path below.<jupyter_code>tmp_dir = tempfile.TemporaryDirectory()
os.environ["OPENGHG_PATH"] = tmp_dir.name # "/tmp/openghg_store"<jupyter_output><empty_output><jupyter_text>2. Processing data First we want to process some files from our local directory<jupyter_code>bsd_datapaths = bilsdale_datapaths()<jupyter_output><empty_output><jupyter_text>We can pass this filepath to `ObsSurface.read_file`. We must also tell it the type of data we want it to process, DECC data is CRDS. We also pass in the site code and the name of the network.<jupyter_code>decc_results = ObsSurface.read_file(filepath=bsd_datapaths, data_type="CRDS", site="bsd", network="DECC")<jupyter_output>Processing: bsd.picarro.1minute.108m.min.dat: 100%|████████████████████| 3/3 [00:00<00:00, 13.98it/s]<jupyter_text>Here `decc_results` will give us a dictionary with the UUIDs (universally unique identifiers) for each of the Datasources the data has been assigned to. This tells us that the data has been processed and stored correctly.  A note on Datasources Datasources are objects that are stored in the object store (add link to object store notes) that hold the data and metadata associated with each measurement we upload to the platform.For example, if we upload a file that contains readings for three gas species from a single site at a specific inlet height OpenGHG    will assign this data to three different Datasources, one for each species. Metadata such as the site, inlet height, species, network etc are stored alongside the measurements for easy searching. Datasources can also handle multiple versions of data from a single site, so if scales or other factors change multiple versions may be stored for easy future comparison.  NoteWhen you run this notebook different UUIDs will be created for the Datasources. This is expected as each time a Datasource is created from scratch it is assigned a unique UUID.<jupyter_code>decc_results<jupyter_output><empty_output><jupyter_text>We can now process the AGAGE data. The functions that process the AGAGE data expect data to have an accompanying precisions file. For each data file we create a tuple with the data filename and the precisions filename. A simpler method of uploading these file types is planned. We must create a `tuple` for each pair```pythonlist_of_tuples = [(data_filepath, precision_filepath), (d1, p1), (d2, p2), ...]```<jupyter_code>agage_tuples = [('../data/AGAGE/capegrim-medusa.18.C', '../data/AGAGE/capegrim-medusa.18.precisions.C')]<jupyter_output><empty_output><jupyter_text>Then we process the files as we did before the with DECC data, but this time changing the data type to the `GCWERKS` type and the network to `AGAGE`.<jupyter_code>agage_results = ObsSurface.read_file(filepath=agage_tuples, data_type="GCWERKS", site="CGO", network="AGAGE")<jupyter_output>Processing: capegrim-medusa.18.C:   0%|                                        | 0/1 [00:00<?, ?it/s]<jupyter_text>When viewing `agage_results` there will be a large number of Datasource UUIDs shown due to the large number of gases in each data file<jupyter_code>agage_results<jupyter_output><empty_output><jupyter_text>3. Visualising the object store Now that we have a simple object store created we can view the objects within it in a simple force graph model. To do this we use the `view_store` function from the `objectstore` submodule. Note that the cell may take a few moments to load as the force graph is created.In the force graph the central blue node is the `ObsSurface` node. Associated with this node are all the data processed by it. The next node in the topology are networks, shown in green. In the graph you will see `DECC` and `AGAGE` nodes. From these you'll see site nodes in red and then individual datasources in orange.  NoteThe object store visualisation created by this function is commented out here and won't be visible in the documentation but can be uncommented and run when you use the notebook version.<jupyter_code>visualise_store()<jupyter_output><empty_output><jupyter_text>Now we know we have this data in the object store we can search it and retrieve data from it.  4. Retrieving data  To retrieve data from the object store we can use the `get_obs_surface` function from the `localclient` submodule. This allows us to retrieve and view the data stored.<jupyter_code>data = get_obs_surface(site="bsd", species="co", network="AGAGE", inlet="248m")<jupyter_output><empty_output><jupyter_text>If we view data we expect an `ObsData` object to have been returned<jupyter_code>data<jupyter_output><empty_output><jupyter_text>First we tell `matplotlib` that we are plotting inside a Jupyter notebook, this ensures a plot with controls is created.<jupyter_code>%matplotlib notebook
example_data = data.data
mol_frac = example_data.mf
mol_frac.plot()<jupyter_output><empty_output><jupyter_text>5. Ranking data The dates that the data from Heathfield retrieved above overlap. If we want to easily retrieve the highest quality data from Heathfield over a range of dates we don't want to have to repeatedly check which was the correct inlet/instrument for a given daterange. This problem is solved using ranking. A given inlet on a specific instrument at a site can be given a rank for a daterange. To do this we use the `RankSources` class from the `localclient` submodule.  Get ranking data<jupyter_code>r = RankSources()
r.get_sources(site="bsd", species="co")<jupyter_output><empty_output><jupyter_text>Set ranking data This tells us that there is no rank set for CO data at Bisldale. Say we want to prioritise data from different inlet heights between certain dates we can set the ranks like so<jupyter_code>r.set_rank(key="co_248m_picarro", rank=1, start_date="2016-01-01", end_date="2018-01-01")
r.set_rank(key="co_42m_picarro", rank=1, start_date="2018-01-02", end_date="2019-05-30")
r.set_rank(key="co_108m_picarro", rank=1, start_date="2019-05-30", end_date="2021-11-30")<jupyter_output><empty_output><jupyter_text>Now we can check everything was set correctlying using `get_sources` again as above<jupyter_code>r.get_sources(site="bsd", species="co")<jupyter_output><empty_output><jupyter_text>Retrieve ranked data We can now retrive CO data from Bilsdale and we'll get the data from the highest ranked inlets.<jupyter_code>co_data = get_obs_surface(site="bsd", species="co")<jupyter_output><empty_output><jupyter_text>This retrieves the data and creates an `ObsData` dataclass. This holds the data in an xarray `Dataset`, the metadata associated with the site and each inlet and the ranking metadata. Let's have a look at the rank metadata<jupyter_code>co_data.metadata["rank_metadata"]
co_data.data
co_data.metadata<jupyter_output><empty_output><jupyter_text>Now we have the highest ranked data for Bilsdale.  7. Cleanup If you used the `tmp_dir` as a location for your object store at the start of the tutorial you can run the cell below to remove any files that were created.<jupyter_code>tmp_dir.cleanup()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0697316322041355, 'fraction_numerical': 0.012318521777386713, 'mean_word_length': 5.725591715976331, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 71, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '483716', 'license': 'Apache-2.0', 'n_tokens_mistral': 2693, 'n_tokens_neox': 2563, 'n_words': 1140, 'path': 'notebooks/tutorials/beginner_workflow/workflow.ipynb', 'repo_name': 'openghg/openghg', 'size': '9092'}