<jupyter_start><jupyter_text>Songbird Metadata with Protocol Buffers - TutorialThis is a guided tutorial that goes trhough the necessary steps to create a serialized protobuf, binary file (.pb) that contains the metadata that describes an experimental session.It also explains how to use the library: **import_dictionary.py** This library adds functionality to managing the metadata, including reading, modifying and exporting the metadata to different formats.  Example: Create metadata file for one bird.The first thing that you need to get familiar with, is the protobuf message that describes the feasible metadata. Please read [Metadata_Documentation](https://docs.google.com/document/d/1rPyEloN52eB-89N0Bfay9FWzm0Xa77S6vTqWd9M1cmQ/edit) This document defines the fields that can be included in the metadata. The following example will create metadata from scratch for a given bird and export it in a serialized metadata protobuf file (.pb) and a human-readable JSON format.<jupyter_code>import os
import sys
from metadata_API import * 
# 1) Define experiment metadata in a dictionary form (some fields are automatically filled, like date & time):

bird_info = {
    'bird_type': 'ZEBRA',                                # UNKNOWN_BIRDTYPE(0), ZEBRA(1), STARLING(2), BENGALESE(3)
    'bird_sex': 'MALE',                                  # UNKNOWN_BIRDSEX(0), MALE(1), FEMALE(2)
    'bird_uid': "z_m10g8_20",                            # string e.g. z_m10g8_20
    'weight_grams': 18.3,                                # float
    'testosterone': True,                                # bool: True / False
    'testosterone_date': "2021-03-10",                   # string: e.g. 2021-03-10
    'dummy_weight': True,                                # bool: True / False
    'dummy_weight_grams': 0.6,                           # float
    'dummy_weight_date': "2021-03-10",                   # string: e.g. 2021-03-10
    'dummy_tether': True,                                # bool: True / False
    'dummy_tether_date': "2021-03-10",                   # string: e.g. 2021-03-10
    'condition': 'HABITUATION',                          # UNKNOWN_CONDITION(0), HABITUATION(1), CHRONIC(2), ACUTE(3)
    'box': "cuervecito3",                                # string: e.g. passaro1, cuervecito3, shoox
    'details': ["dummy_weight + tether"],                # repeated string: (Any additional info)
}

# 2) Create new protobuf message and add the defined information as metadata:

metadata = ProtobufMetadata()              # New instance of metadata class. It is initialized with some default values (check metadata.sess)
metadata.read_bird_metadata(bird_info)     # Add bird info to our metadata object (check metadata.sess has been updated). 
                                                # It ensures that each metadata field is in the expected format.

# 3) Save metadata as serialized protobuf file (.pb) and as a human-readable (.json) file:
filepath = os.getcwd() + '/'
filename = metadata.sess.sess_uid + '_metadata'

metadata.export_metadata_to_json(filepath + filename)
metadata.serialize_metadata(filepath + filename)<jupyter_output><empty_output><jupyter_text>The following example will load a serialized metadata protobuf file (.pb) and use the information it contains about the bird, add an acquisition message, and export a new metadata file. Let's now assume that we are going to run a new recording on an implanted bird. We want to create a new metadata file, but import some information from an old metadata file that we have for the bird.<jupyter_code>bird_metadata = ProtobufMetadata()
bird_metadata.parse_serialized_metadata(filepath + filename + '.pb') # This object has loaded the metadata from a different experimental session

recording_metadata = ProtobufMetadata() # Creating a new metadata instace will create new time & date fields
recording_metadata.read_bird_metadata(bird_metadata.sess) # Parse bird metadata from previous experiment to avoid having to recreate it<jupyter_output><empty_output><jupyter_text>Now imagine we wish to add the metadata corresponding to our recording session, that consists of 2 acquisions (one with Alsa & one with spikeGLX)<jupyter_code># 1) Define experiment metadata in a dictionary form:

# Acquisition 1: Audio sensor to PC - UMA8 recorded with Alsa
sensor_uma8raw = {
        'acquisition_signal': "audio",              # audio, audio, pressure, sync, emg, video 
        'manufacturer': "miniDSP",                  # miniDSP, earthworks, fujikura, inhouse, inhouse
        'model': "uma8raw",                         # uma8, m30, fhm-20, uma8_sync, emg_amp_inhouse
        #'serial_number': "uma_",                   # '', '', '', uma_syn_001, emg_amp_001
        'signal_name': "audio_raw",                 # uma_syn_001, pressure, syn_uma8, emg, syn_stim, stim, …
        #'headstage': nan,                          # '', '', intan32, ", intan32
        #'channel_group': "A",                      # '', AIN, port_0, DIN, port_1
        'channels': "0-6",                          # '', [0], [aux_0], [0], [aux_0, aux_1]
        'locations': "OUT",                         # left, right, tracheal rings, muscles...
        'details': ["Positioned in top-back-left and top-front-right corners of the chamber.", "detalis2"] # repeated string
}
sensor_uma8dsp = {
        'acquisition_signal': "audio",              # audio, audio, pressure, sync, emg, video 
        'manufacturer': "miniDSP",                  # miniDSP, earthworks, fujikura, inhouse, inhouse
        'model': "uma8DSP",                         # uma8, m30, fhm-20, uma8_sync, emg_amp_inhouse
        #'serial_number': "uma_syn_001",            # '', '', '', uma_syn_001, emg_amp_001
        'signal_name': "audio_DSP",                 # uma_syn_001, pressure, syn_uma8, emg, syn_stim, stim, …
        #'headstage': nan,                          # '', '', intan32, ", intan32
        #'channel_group': "PDM",                    # '', AIN, port_0, DIN, port_1
        'channels': "7-8",                          # '', [0], [aux_0], [0], [aux_0, aux_1]
        'locations': "OUT",                         # left, right, tracheal rings, muscles...
        'details': ["Positioned in top-back-left and top-front-right corners of the chamber.", "detalis2"] # repeated string
}  
sensor_uma8syn = {
        'acquisition_signal': "sync_imec",          # audio, audio, pressure, sync, emg, video 
        'manufacturer': "inhouse",                  # miniDSP, earthworks, fujikura, inhouse, inhouse
        'model': "uma_syn",                         # uma8, m30, fhm-20, uma8_sync, emg_amp_inhouse
        'serial_number': "uma_syn_001",             # '', '', '', uma_syn_001, emg_amp_001
        'signal_name': "uma_syn_001",               # uma_syn_001, pressure, syn_uma8, emg, syn_stim, stim, …
        #'headstage': nan,                          # '', '', intan32, ", intan32
        'channel_group': "PDM",                     # '', AIN, port_0, DIN, port_1
        'channels': "7",                            # '', [0], [aux_0], [0], [aux_0, aux_1]
        'locations': "OUT",                         # left, right, tracheal rings, muscles...
        'details': ["Positioned in top-back-left and top-front-right corners of the chamber.", "detalis2"] # repeated string
}  
acquisition_alsa = {  
    'acquisition_hardware': "uma8-usb",  # openephys, intan, spikeglx, uma8, raspi, any custom amp
    'acquisition_software': "alsa",      # openephys, openephys++, spikeglx, alsa, vlc
    'sensors': [sensor_uma8raw, sensor_uma8dsp, sensor_uma8syn]            # repeated 'sensor message'
}

# Acquisition 2: Neural probe + video feedback to IMEC
neuralprobe = {
    'acquisition_signal': "neural",                     # string: e.g. neural
    'manufacturer': "neuropixel",                       # string: neuronexus, neuropixel, masmanidis, inhouse
    'model': "neuropixels_1",                           # string: buzsaki32, neuropixels_1
    'serial_number': "U656",                            # string: U656
    'num_channels': 385,                                # int32: 16, 32, 64, 384
    'tip_depth_microns': 3500.0,                        # float: depth of the tip of the probe
    'implant_coordinates_microns': "500, 2700, 3500",   # string: A/P, M/L, D/V
    'hemisphere': "right",                              # string: left, right
    'brain_nucleus': ["hvc", "ra"],                     # repeated string: hvc, ra, area_X
    'headstage': "neuropixel",                          # string: e.g. H32
    'channel_group': "port_0",                          # string: port_0, imec_0
    'channels': "1-385",                                # string: e.g. 1-64,
    'details': ["details"]                              # repeated string
}
sensor_micm30 = {
    'acquisition_signal': "audio",                  # audio, audio, pressure, sync, emg, video 
    'manufacturer': "earthworks",                   # miniDSP, earthworks, fujikura, inhouse, inhouse
    'model': "m30",                                 # uma8, m30, fhm-20, uma8_sync, emg_amp_inhouse
    'serial_number': "tuvieja",                     # '', '', '', uma_syn_001, emg_amp_001
    'signal_name': "mic_0",                         # uma_syn_001, pressure, syn_uma8, emg, syn_stim, stim, …
    #'headstage': nan,                              # '', '', intan32, ", intan32
    'amplifier': 'grace m1',
    'channel_group': "AIN",                         # '', AIN, port_0, DIN, port_1
    'channels': "AIN0",                             # '', [0], [aux_0], [0], [aux_0, aux_1]
    'locations': "out",                             # left, right, tracheal rings, muscles...
    'details': ["Positioned in top-back-left and top-front-right corners of the chamber.", "detalis2"] # repeated string
}
stimulus_video = {
    'stimulus_signal': "prerecorded video",          # string: female, video, song_replay, neural_stim
    'manufacturer': "inhouse",                       # string: '', inhouse, inhouse, neuronexus
    'model': "3 females in cage",                    # string: '', '', '3 females in cage, buzsaki32
    'serial_number': "",                             #
    'channel_gropup': "AIN",                         # string: AIN, port_0, DIN, port_1
    'channels': "aux_0",                             # string: [0], [aux_0], [0], [aux_0, aux_1]
    'details': ["details"],                          # repeated string: Any additional info
}               
acquisition_spikeglx = {  
    'acquisition_hardware': "IMEC",                  # openephys, intan, spikeglx, uma8, raspi, any custom amp
    'acquisition_software': "spikeglx",              # openephys, openephys++, spikeglx, alsa, vlc
    'neuralprobes': [neuralprobe],                   # repeated 'neuralprobes message' 
    'sensors': [sensor_micm30],                      # repeated 'sensor message'
    'stimuli': [stimulus_video]                      # repeated 'stimuli message' 
}

# Combined Acquisitions
acquisitions_dict = {                                                
    'acquisitions': [acquisition_alsa, acquisition_spikeglx]         # repeated 'acquisitions message'
}

# 2) Add acquisitions info to our metadata object (check recording_metadata.sess has been updated).
recording_metadata.read_aquisitions_metadata(acquisitions_dict)  

# 3) Save metadata as serialized protobuf file (.pb) and as a human-readable (.json) file:
filepath = os.getcwd() + '/'
filename = recording_metadata.sess.sess_uid + '_metadata'

recording_metadata.export_metadata_to_json(filepath + filename)
recording_metadata.serialize_metadata(filepath + filename)<jupyter_output><empty_output><jupyter_text>The following example will load a metadata JSON file (.json) and the information it contains about the bird and acquisitions. It will then modify (add / delete / modify) some submessages and fileds, and export a new metadata file.  Let's now assume that we are going to run an experiment similar to a previous one, but with a few differences. 1) Load the metadata for the previous experiment (.json)<jupyter_code>experiment_metadata = ProtobufMetadata()
experiment_metadata.parse_metadata_from_json(filepath + filename + '.json') # This object has loaded the metadata from a different experimental session<jupyter_output><empty_output><jupyter_text>2) imagine we only have an audio acquisition system and would like to delete the spikeglx acquisition. We would also like to delete all sensors in the audio acquisition system:<jupyter_code>del(experiment_metadata.sess.acquisitions[1])
experiment_metadata.delete_attribute(experiment_metadata.sess.acquisitions[0], 'sensors')<jupyter_output><empty_output><jupyter_text>3) Let's now assume that we would like to update some of the fields that we defined for our experiment metadata. We also would like to add a new sensor in the audio acquisition. We update the date & time of the metadata before we save it:<jupyter_code># Modify bird metadata
experiment_metadata.sess.bird_sex = 2   # 1(Male) 2(Female)
experiment_metadata.sess.weight_grams = 17.3
experiment_metadata.sess.testosterone = False
experiment_metadata.sess.condition = 2  # 1(Habituation) 2(CHRONIC) 3(ACUTE) 

# Modify acquisition metadata and add a sensor to it
experiment_metadata.sess.acquisitions[0].acquisition_hardware = 'raspberry Pi'
experiment_metadata.sess.acquisitions[0].sensors.add()
experiment_metadata.sess.acquisitions[0].sensors[0].acquisition_signal = 'audio'
experiment_metadata.sess.acquisitions[0].sensors[0].manufacturer = 'earthworks'
experiment_metadata.sess.acquisitions[0].sensors[0].model = 'm30'

experiment_metadata.update_date_and_time()
# 3) Save metadata as serialized protobuf file (.pb) and as a human-readable (.json) file:
filepath = os.getcwd() + '/'
filename = experiment_metadata.sess.sess_uid + '_metadata'

experiment_metadata.export_metadata_to_json(filepath + filename)
experiment_metadata.serialize_metadata(filepath + filename)<jupyter_output><empty_output><jupyter_text>The following example will create a new metadata descriptor from scratch using the default functions, and the user will populate it.<jupyter_code>metadata = ProtobufMetadata()              # New instance of metadata class. It is initialized with some default values (check metadata.sess)
# 1) Create new metadata object. It automatillay comes populated with default bird metadata, but we rewrite the default bird metadata for illustrative purposes.

metadata.default_bird_metadata()

# 2) Add one acquisition and populate it with defaults:
metadata.sess.acquisitions.add()
metadata.default_acquisition_metadata(metadata.sess.acquisitions[0])

# 3) Add one Neural Probe, one Sensor and one Stimulus to this acquisition and populate them with default values:
metadata.sess.acquisitions[0].neuralprobes.add()
metadata.default_neural_probe_metadata(metadata.sess.acquisitions[0].neuralprobes[0])

metadata.sess.acquisitions[0].sensors.add()
metadata.default_sensor_metadata(metadata.sess.acquisitions[0].sensors[0])

metadata.sess.acquisitions[0].stimuli.add()
metadata.default_stimulus_metadata(metadata.sess.acquisitions[0].stimuli[0])

filepath = os.getcwd() + '/'
filename = 'default_single_acquisition_metadata'

metadata.export_metadata_to_json(filepath + filename)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1055969661305087, 'fraction_numerical': 0.022819406303125407, 'mean_word_length': 2.735955056179775, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '138960', 'license': 'Apache-2.0', 'n_tokens_mistral': 4720, 'n_tokens_neox': 4381, 'n_words': 1299, 'path': 'metadata_protobuf_tutorial.ipynb', 'repo_name': 'pabloslash/protobuff_serialization', 'size': '15294'}