<jupyter_start><jupyter_text>Sources and Measurements ExampleThis example shows how to upload measurements from an external repository into the MultiViz Analytics Engine (MVG) service and how to read this data.<jupyter_code>import json
import os
from pathlib import Path

from tqdm.notebook import tqdm
import pandas as pd

# import mvg library with python bindings to mvg-API
from mvg import MVG
from mvg.exceptions import MVGAPIError<jupyter_output><empty_output><jupyter_text>Note that the `TOKEN` is used both for authorization and authentication.Thus, each unique token represents a unique user and each user has their own unique database on the VA service.**You need to insert your token received from Viking Analytics here:**<jupyter_code>ENDPOINT = "https://api.beta.multiviz.com"
# Replace by your own Token
VALID_TOKEN = os.environ['TEST_TOKEN']<jupyter_output><empty_output><jupyter_text>Downloading the DataWe pick the data from one source from our public charlie repo https://github.com/vikinganalytics/va-data-charlie.git for convenience.Clone that repository to get access to the data.<jupyter_code>!git clone https://github.com/vikinganalytics/va-data-charlie.git<jupyter_output><empty_output><jupyter_text>We are going to use six of the sources from the charlie dataset with IDs `u0001` to `u0006`.<jupyter_code>REF_DB_PATH = Path.cwd() / "va-data-charlie" / "charlieDb" / "acc"
SOURCE_IDS = ["u0001", "u0002", "u0003", "u0004", "u0005", "u0006"]<jupyter_output><empty_output><jupyter_text>Connect to the APIInstantiate a session object with mvg library a session object basically caches the endpoint and the token,to simplify the calls to the MVG library.<jupyter_code>session = MVG(ENDPOINT, VALID_TOKEN)<jupyter_output><empty_output><jupyter_text>We now check if the server is alive. The hello message contains, amongst others the API version.<jupyter_code>session.say_hello()<jupyter_output><empty_output><jupyter_text>Check DatabaseWe start by seeing if there are any sources in the database.A source represents a measurement source e.g. one specific channel of a measurement sensor.Note that sources cannot be used to represent a sensor with several channels.If this is required, the client side needs to take care of it.Because we want to start at a clean slate we will list all sources (potentially none) and delete all of them<jupyter_code>sources = session.list_sources()

for src in sources:
    # While the list returned contains all information
    # about all known sources, it is also possible
    # to query for a single source by its id
    s_info = session.get_source(src['source_id'])
    print(f"Source info retrieved:\n{s_info}\n")<jupyter_output>Source info retrieved:
{'source_id': 'u0001', 'meta': {'assetId': 'assetJ', 'measPoint': 'mloc01', 'location': 'cancun', 'timezone': 'Europe/Stockholm'}, 'properties': {'data_class': 'waveform', 'channels': ['acc']}}<jupyter_text>The example below revolves around a source with source id u0001<jupyter_code>SOURCE_ID = SOURCE_IDS[0]<jupyter_output><empty_output><jupyter_text>To make sure we start from a clean slate we delete our resource in case it exists.<jupyter_code>try:
    source = session.get_source(SOURCE_ID)
    session.delete_source(SOURCE_ID)
    print(f"Source {SOURCE_ID} deleted")
except MVGAPIError:
    print(f"Source {SOURCE_ID} does not exist")<jupyter_output>Source u0001 deleted<jupyter_text>Build measurementsNow we want to (re) build the source and the attached measurements from scratch. In this example, we have a json file with all the information needed to create each source.<jupyter_code>src_path = REF_DB_PATH / SOURCE_ID
m_file_name = REF_DB_PATH / SOURCE_ID / "meta.json"
with open(m_file_name, "r") as json_file:
    meta = json.load(json_file)
print("Creating meta info")
print(meta)<jupyter_output>Creating meta info
{'assetId': 'assetA', 'measPoint': 'mloc01', 'location': 'paris'}<jupyter_text>Create the source and check for it<jupyter_code>session.create_source(SOURCE_ID, channels=["acc"], meta=meta)  # create
source = session.get_source(SOURCE_ID)
print("Recreated source info:")
print(source)<jupyter_output>Recreated source info:
{'source_id': 'u0001', 'meta': {'assetId': 'assetA', 'measPoint': 'mloc01', 'location': 'paris'}, 'properties': {'data_class': 'waveform', 'channels': ['acc']}}<jupyter_text>Update the source<jupyter_code>meta['updated'] = "YES! I have been updated"
session.update_source(SOURCE_ID, meta)  # update
source = session.get_source(SOURCE_ID)
print("Updated source info")
print(source)<jupyter_output>Updated source info
{'source_id': 'u0001', 'meta': {'assetId': 'assetA', 'measPoint': 'mloc01', 'location': 'paris', 'updated': 'YES! I have been updated'}, 'properties': {'data_class': 'waveform', 'channels': ['acc']}}<jupyter_text>Upload measurements to source. Measurements are tied to sources, they consist of- an array of floating point values with a header indicating the channel name- timestamp when the values were sampled- a field for the duration of the measurement- meta information to be stored along the measurement<jupyter_code>meas = [f.stem for f in Path(src_path).glob("*.csv")]<jupyter_output><empty_output><jupyter_text>`meas` now contains a list of timestamps representing the measurements in our repo we upload from iterate over all of them.<jupyter_code>print(f"Uploading measurements to source {SOURCE_ID}")
for m in tqdm(meas):

    # samples file for one measurement
    TS_MEAS = str(m) + ".csv"  # filename
    TS_MEAS = REF_DB_PATH / SOURCE_ID / TS_MEAS  # path to file
    ts_df = pd.read_csv(TS_MEAS)  # read csv into df
    accs = ts_df.iloc[:, 0].tolist()  # convert to list

    # meta information file for one measurement
    TS_META = str(m) + ".json"  # filename
    TS_META = REF_DB_PATH / SOURCE_ID / TS_META  # path
    with open(TS_META, "r") as json_file:  # read json
        meas_info = json.load(json_file)  # into dict

    # get duration and other meta info
    duration = meas_info['duration']
    meta_info = meas_info['meta']
    # add sampling rate, not required by vibration API
    # but may be used on client side
    # in general any information can be stored
    # along the actual samples
    meta_info['sampling_rate'] = len(accs)/duration
    # <<< end of code specific for repo

    try:
        # see mvg for details on this call
        session.create_measurement(sid=SOURCE_ID,
                                   duration=duration,
                                   timestamp=int(m),
                                   data={"acc": accs},
                                   meta=meta_info)
    except MVGAPIError as exc:
        print(exc)<jupyter_output>Uploading measurements to source u0001<jupyter_text>Read the Measurements Check if we actually created the measurements by reading them<jupyter_code>m = session.list_measurements(SOURCE_ID)
print(f"Read {len(m)} stored measurements")<jupyter_output>Read 50 stored measurements<jupyter_text>It is also possible to read a specific measurement at a single timestamp.Let's get the timestamp of the first measurement.<jupyter_code>ts_0 = m[0]['timestamp']
meas_0 = session.read_single_measurement(SOURCE_ID, ts_0)
# we'll printout the returned measurement
print(f"source_id: {SOURCE_ID}")
print(f"timestamp: {ts_0}")
print(f"duration:  {meas_0['duration']}")
print(f"meta:      {meas_0['meta']}")
print(f"data:      {meas_0['data']['acc'][1:3]}...")<jupyter_output>source_id: u0001
timestamp: 1570186860
duration:  2.8672073400507907
meta:      {'sampling_rate': 13950.50837142857}
data:      [-0.55078125, -1.332275390625]...<jupyter_text>We can update the meta information for a measurement at a single timestamp let's get the timestamp of the first measurement<jupyter_code>ts_0 = m[0]['timestamp']
new_meta = meas_0['meta']
new_meta['updated'] = "YES!"
session.update_measurement(SOURCE_ID, ts_0, new_meta)
meas_0_u = session.read_single_measurement(SOURCE_ID, ts_0)

# we'll printout the returned measurement
print(f"source_id: {SOURCE_ID}")
print(f"timestamp: {ts_0}")
print(f"duration:  {meas_0_u['duration']}")
print(f"meta:      {meas_0_u['meta']}")
print(f"data:      {meas_0_u['data']['acc'][1:3]}...")<jupyter_output>source_id: u0001
timestamp: 1570186860
duration:  2.8672073400507907
meta:      {'sampling_rate': 13950.50837142857, 'updated': 'YES!'}
data:      [-0.55078125, -1.332275390625]...<jupyter_text>Finally we delete the measurement<jupyter_code>session.delete_measurement(SOURCE_ID, ts_0)<jupyter_output><empty_output><jupyter_text>We check if it's actually deleted<jupyter_code>try:
    meas_0 = session.read_single_measurement(SOURCE_ID, ts_0)
except MVGAPIError:
    print("Previously deleted measurement does not exist")<jupyter_output>Previously deleted measurement does not exist<jupyter_text>If all went well we end up here now.The source u0001 is in the database along with a number of its measurements.  Uploading additional sourcesHere we show how to upload additional sources in a more compact way.<jupyter_code>sources = SOURCE_IDS[1:]

print(f"Uploading sources {', '.join(sources)} from CharlieDb")
for source_id in tqdm(sources):

    try:
        session.delete_source(source_id)
    except MVGAPIError as exc:
        pass  # Source didnt exist

    print(f"Creating source {source_id}")
    src_path = REF_DB_PATH / source_id
    m_file_name = REF_DB_PATH / source_id / "meta.json"
    with open(m_file_name, "r") as json_file:
        meta = json.load(json_file)
    session.create_source(source_id, channels=["acc"], meta=meta)  # create

    meas = [f.stem for f in Path(src_path).glob("*.csv")]

    print(f"Uploading measurements to source {source_id}")
    for m in tqdm(meas):

        # samples file for one measurement
        TS_MEAS = str(m) + ".csv"  # filename
        TS_MEAS = REF_DB_PATH / source_id / TS_MEAS  # path to file
        ts_df = pd.read_csv(TS_MEAS)  # read csv into df
        accs = ts_df.iloc[:, 0].tolist()  # convert to list

        # meta information file for one measurement
        TS_META = str(m) + ".json"  # filename
        TS_META = REF_DB_PATH / source_id / TS_META  # path
        with open(TS_META, "r") as json_file:  # read json
            meas_info = json.load(json_file)  # into dict

        duration = meas_info['duration']
        meta_info = meas_info['meta']
        meta_info['sampling_rate'] = len(accs)/duration

        try:
            # see mvg for details on this call
            session.create_measurement(sid=source_id,
                                       duration=duration,
                                       timestamp=int(m),
                                       data={"acc": accs},
                                       meta=meta_info)
        except MVGAPIError as exc:
            print(exc)
            raise<jupyter_output>Uploading sources u0002, u0003, u0004, u0005, u0006 from CharlieDb
{'fraction_non_alphanumeric': 0.10352746647069631, 'fraction_numerical': 0.023608304243983098, 'mean_word_length': 4.2978102189781024, 'pattern_counts': {'":': 2, '<': 73, '<?xml version=': 0, '>': 70, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '286949', 'license': 'Apache-2.0', 'n_tokens_mistral': 3641, 'n_tokens_neox': 3381, 'n_words': 1026, 'path': 'docs/source/content/examples/2-sources_and_measurements.ipynb', 'repo_name': 'vnadhan/mvg', 'size': '10886'}