<jupyter_start><jupyter_text>Exporting QCoDes Datasets This notebook demonstrates how we can export QCoDeS datasets to other file formats.   Setup First, we borrow an example from the measurement notebook to have some data to work with.<jupyter_code>%matplotlib inline
import shutil
from pathlib import Path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import qcodes as qc
from qcodes.dataset import (
    initialise_or_create_database_at,
    load_or_create_experiment,
    Measurement,
    load_by_run_spec,
    load_from_netcdf,
)
from qcodes.tests.instrument_mocks import (
    DummyInstrument,
    DummyInstrumentWithMeasurement,
)
from qcodes.dataset.plotting import plot_dataset

qc.logger.start_all_logging()
# preparatory mocking of physical setup
dac = DummyInstrument("dac", gates=["ch1", "ch2"])
dmm = DummyInstrumentWithMeasurement("dmm", setter_instr=dac)
station = qc.Station(dmm, dac)
initialise_or_create_database_at("./export_example.db")
exp = load_or_create_experiment(
    experiment_name="exporting_data", sample_name="no sample"
)
meas = Measurement(exp)
meas.register_parameter(dac.ch1)  # register the first independent parameter
meas.register_parameter(dac.ch2)  # register the second independent parameter
meas.register_parameter(
    dmm.v2, setpoints=(dac.ch1, dac.ch2)
)  # register the dependent one<jupyter_output><empty_output><jupyter_text>We then perform two very basic measurements using dummy instruments.<jupyter_code># run a 2D sweep

with meas.run() as datasaver:

    for v1 in np.linspace(-1, 0, 200, endpoint=False):
        for v2 in np.linspace(-1, 1, 201):
            dac.ch1(v1)
            dac.ch2(v2)
            val = dmm.v2.get()
            datasaver.add_result((dac.ch1, v1), (dac.ch2, v2), (dmm.v2, val))

dataset1 = datasaver.dataset
# run a 2D sweep

with meas.run() as datasaver:
    for v1 in np.linspace(0, 1, 200, endpoint=False):
        for v2 in np.linspace(1, 2, 201):
            dac.ch1(v1)
            dac.ch2(v2)
            val = dmm.v2.get()
            datasaver.add_result((dac.ch1, v1), (dac.ch2, v2), (dmm.v2, val))

dataset2 = datasaver.dataset<jupyter_output>Starting experimental run with id: 6.<jupyter_text>Exporting data manually The dataset can be exported using the `export` method. Currently exporting to netcdf and csv is supported.<jupyter_code>dataset2.export("netcdf", path=".")<jupyter_output><empty_output><jupyter_text>The `export_info` attribute contains information about where the dataset has been exported to:<jupyter_code>dataset2.export_info<jupyter_output><empty_output><jupyter_text>Looking at the signature of export we can see that in addition to the file format we can set the `prefix` and `path` to export to.<jupyter_code>?dataset2.export<jupyter_output><empty_output><jupyter_text>Export data automatically Datasets may also be exported automatically using the configuration options given in dataset config section. Here you can toggle if a dataset should be exported automatically using the `export_automatic` option as well as set the default type, prefix and path. See [the table here](https://qcodes.github.io/Qcodes/user/configuration.html) for the relevant configuration options.For more information about how to configure QCoDeS datasets see [the page about configuration](https://qcodes.github.io/Qcodes/user/configuration.html)  in the QCoDeS docs.  Importing exported datasets into a new database The above dataset has been created in the following database<jupyter_code>qc.config.core.db_location<jupyter_output><empty_output><jupyter_text>Now lets imagine that we move the exported dataset to a different computer. To emulate this we will create a new database file and set it as the active database.<jupyter_code>initialise_or_create_database_at("./reimport_example.db")
qc.config.core.db_location<jupyter_output><empty_output><jupyter_text>We can then reload the dataset from the netcdf file as a DataSetInMem. This is a class that closely matches the regular DataSet class however its metadata may or may not be written to a database file and its data is not written to a database file. See more in [In memory dataset](./InMemoryDataSet.ipynb) . Concretely this means that the data captured in the dataset can be acceced via `dataset.cache.data` etc. and not via the methods directly on the dataset (`dataset.get_parameter_data` ...) Note that it is currently only possible to reload a dataset from a netcdf export and not from a csv export. This is due to the fact that a csv export only contains the raw data and not the metadata needed to recreate a dataset.<jupyter_code>loaded_ds = load_from_netcdf(dataset2.export_info.export_paths["nc"])
type(loaded_ds)<jupyter_output><empty_output><jupyter_text>However, we can still export the data to Pandasa and xarray.<jupyter_code>loaded_ds.cache.to_xarray_dataset()<jupyter_output><empty_output><jupyter_text>And plot it using `plot_dataset`.<jupyter_code>plot_dataset(loaded_ds)<jupyter_output><empty_output><jupyter_text>Note that the dataset will have the same `captured_run_id` and `captured_counter` as before:<jupyter_code>captured_run_id = loaded_ds.captured_run_id
captured_run_id<jupyter_output><empty_output><jupyter_text>But do note that the `run_id` and `counter` are in general not preserved since they represent the datasets number in a given db file.<jupyter_code>loaded_ds.run_id<jupyter_output><empty_output><jupyter_text>A loaded datasets metadata can be written to the current db file and subsequently the dataset including metadata and raw data reloaded from the database and netcdf file.<jupyter_code>loaded_ds.write_metadata_to_db()<jupyter_output><empty_output><jupyter_text>Now that the metadata has been written to a database the dataset can be plotted with [plottr](https://github.com/toolsforexperiments/plottr/) like a regular dataset.<jupyter_code>del loaded_ds
reloaded_ds = load_by_run_spec(captured_run_id=captured_run_id)
plot_dataset(reloaded_ds)<jupyter_output><empty_output><jupyter_text>Note that loading a dataset from the database will also load the raw data into `dataset.cache` provided that the `netcdf` file is still in the location where file was when the metadata was written to the database. Load_by_runspec and related functions will load data into a regular `DataSet` provided that the data can be found in the database otherwise it will be loaded into a `DataSetInMem`If the netcdf file cannot be found the dataset will load with a warning and the raw data will not be accessible from the dataset. If this happens because you have moved the location of a netcdf file you can use the method ``set_netcdf_location`` to set a new location for the the netcdf file in the dataset and database file. Here we demonstrate this by copying the netcdf file and changing the location using this method.<jupyter_code>filepath = dataset2.export_info.export_paths["nc"]
new_file_path = str(Path(dataset2.export_info.export_paths["nc"]).parent / "somefile.nc")
new_file_path
shutil.copyfile(dataset2.export_info.export_paths["nc"], new_file_path)
reloaded_ds.set_netcdf_location(new_file_path)
reloaded_ds.export_info<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07331010452961673, 'fraction_numerical': 0.00878048780487805, 'mean_word_length': 5.638297872340425, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '349666', 'license': 'MIT', 'n_tokens_mistral': 2227, 'n_tokens_neox': 2170, 'n_words': 746, 'path': 'docs/examples/DataSet/Exporting-data-to-other-file-formats.ipynb', 'repo_name': 'yoshi74ls181/Qcodes', 'size': '7175'}