<jupyter_start><jupyter_text>NWB Conversion Tools - Detailed Tutorial Examine the direct output structure from each call from the "Simple Tutorial"<jupyter_code>from pathlib import Path
from pprint import pprint
from datetime import datetime

from pynwb import NWBHDF5IO

from nwb_conversion_tools import NWBConverter, RecordingTutorialInterface, SortingTutorialInterface<jupyter_output><empty_output><jupyter_text>Pre-step: Define the conversion class and its internal data interface classes (*i.e.*, the names of each format) For a full list of supported formats, [see this list](https://nwb-conversion-tools.readthedocs.io/en/conversion_guide/converting_data_to_nwb.html), or [make your own data interface](https://nwb-conversion-tools.readthedocs.io/en/conversion_guide/data_interface.html)<jupyter_code>class TutorialNWBConverter(NWBConverter):
    data_interface_classes = dict(
        RecordingTutorial=RecordingTutorialInterface,
        SortingTutorial=SortingTutorialInterface
    )<jupyter_output><empty_output><jupyter_text>Specify global arguments to be used throughout the code<jupyter_code># Custom parameters for simulated toy data
duration = 10.  # Seconds
num_channels = 4
num_units = 10
sampling_frequency = 30000.  # Hz

stub_test = False  # Truncates data write for faster quality checking

output_file = "NWBConversionToolsDetailedTutorial.nwb"<jupyter_output><empty_output><jupyter_text>Step 1 Part A (hidden): request the structure of what the source_data should adhere to<jupyter_code>source_schema = TutorialNWBConverter.get_source_schema()
pprint(source_schema["properties"], width=120)<jupyter_output><empty_output><jupyter_text>Part B: Construct the source_data dictionary-of-dictionaries specified to mimic this schema precisely...<jupyter_code>source_data = dict(
    RecordingTutorial=dict(
        duration=duration,
        num_channels=num_channels,
        sampling_frequency=sampling_frequency
    ),
    SortingTutorial=dict(
        duration=duration,
        num_units=num_units,
        sampling_frequency=sampling_frequency
    )
)<jupyter_output><empty_output><jupyter_text>Part C: the NWBConverter will internally verify that this input agrees with the expected schema<jupyter_code>converter = TutorialNWBConverter(source_data=source_data)<jupyter_output><empty_output><jupyter_text>Part D: after initializing the NWBConverter, you can check if each of the internal interfaces was instantiated by accessing the converter.data_interface_objects<jupyter_code>print("Data interfaces for this converter: \n")
pprint(converter.data_interface_objects, width=120)<jupyter_output><empty_output><jupyter_text>Step 2 Part A: similar to the source_schema, there is also a metadata_schema that can actually be more extensive, including all the various fields that might fit in the total NWB schema<jupyter_code>metadata_schema = converter.get_metadata_schema()
pprint(metadata_schema, width=300)<jupyter_output><empty_output><jupyter_text>Part B: the bulk of this can and is (wherever possible) automatically pulled from the interface.get_metadata() functions, the intersection of which is taken by the call to converter.get_metadata()<jupyter_code>metadata = converter.get_metadata()
pprint(metadata, width=120)<jupyter_output><empty_output><jupyter_text>Part C: however, there are certain pieces of information that may not be contained in any downstream data file and must (or at least *can*) be manually inserted at this stage<jupyter_code>metadata["NWBFile"]["session_description"] = "NWB Conversion Tools tutorial."
metadata["NWBFile"]["experimenter"] = ["My name"]
metadata["NWBFile"]['session_start_time'] = datetime.now().astimezone().strftime("%Y-%m-%dT%H:%M:%S")
metadata["Subject"] = dict(subject_id="Name of imaginary testing subject (required for DANDI upload)")<jupyter_output><empty_output><jupyter_text>Step 3 Part A: once again, we may fetch the expected schema for all the arguments that may be specified - this time, for the conversion specific options<jupyter_code>conversion_options_schema = converter.get_conversion_options_schema()

print("Conversion options for each data interface: \n")
pprint(conversion_options_schema["properties"], width=120)<jupyter_output><empty_output><jupyter_text>Part B: all we need to do is specify the set of required (and any additional optional) fields<jupyter_code>conversion_options = dict(
    RecordingTutorial=dict(stub_test=stub_test),
    SortingTutorial=dict()
)<jupyter_output><empty_output><jupyter_text>Step 4 The conversion is ready to run in a single command!<jupyter_code># Run conversion
converter.run_conversion(
    metadata=metadata, 
    nwbfile_path=output_file,
    save_to_file=True,  # If False, this instead returns the NWBFile object in memory
    overwrite = True,  # If False, this appends an existing file
    conversion_options=conversion_options
)<jupyter_output><empty_output><jupyter_text>Print raw contents of NWBFile to verify the output is as expected<jupyter_code>with NWBHDF5IO(output_file, "r", load_namespaces=True) as io:
    nwbfile = io.read()
    print(nwbfile)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08081202420456764, 'fraction_numerical': 0.006051141909037674, 'mean_word_length': 7.0947867298578196, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '310256', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1602, 'n_tokens_neox': 1505, 'n_words': 418, 'path': 'tutorials/detailed-tutorial.ipynb', 'repo_name': 'Saksham20/nwb-conversion-tools', 'size': '5123'}