<jupyter_start><jupyter_text>Loading and Saving  Loading  mzML To accommodate disparate instrument types and manufacturers (e.g. Bruker, Waters, Thermo, Agilent), DEIMoS operates under the assumption that input data are in an open, standard format.As of this publication, the accepted file format for DEIMoS is mzML (or mzML.gz), which contains metadata, separation, and spectrometry data that reproduce the contents of vendor formats.Conversion to mzML from several other formats can be performed using the free and open-source [ProteoWizard](https://proteowizard.sourceforge.io/) msconvert utility. By default, DEIMoS will load frame, scan, *m/z*, and intensity from the mzML, as well as precursor *m/z* for MS2, as available.Additional "accession" fields may be specified for data of higher dimension.To view these fields, a convenience function is provided.<jupyter_code>import deimos

accessions = deimos.get_accessions('example_data.mzML.gz')
accessions<jupyter_output><empty_output><jupyter_text>The example data referenced is from an Agilent 6560 Ion Mobility LC/Q-TOF system. Thus, we will additionally need to parse retention time and ion mobility drift times.Consulting the list above, we are able to supply appropriate accession fields to the `load` function, renaming as convenient (here, "scan start time" becomes "retention_time" and "ion mobility drift time" becomes "drift_time").The `load` function will infer file type based on extension (here, .mzML or .mzML.gz)<jupyter_code>%%time
data = deimos.load('example_data.mzML.gz',
                   accession={'retention_time': 'MS:1000016',
                              'drift_time': 'MS:1002476'})<jupyter_output>CPU times: user 4min 2s, sys: 38.9 s, total: 4min 41s
Wall time: 4min 57s<jupyter_text>The resulting data will be returned as a dictionary containing data frames, with keys per MS level. The example data contains MS1 and MS2 (collected at 20 eV).<jupyter_code>data['ms1']
data['ms2']<jupyter_output><empty_output><jupyter_text>HDF5 If the data is already parsed and saved in the Hierarchical Data Format, loading will be much faster. The function does not change, as the loader will again infer format by file extension. However, arguments will be different: specifing accessions is no longer required, but the relevant MS level must be selected using the `key` flag.<jupyter_code>%%time
ms1 = deimos.load('example_data.h5', key='ms1')
ms1<jupyter_output>CPU times: user 7.81 s, sys: 10.1 s, total: 17.9 s
Wall time: 22.6 s<jupyter_text>Multi-file Loading For certain alignment applications, a high number of input files bars reading each into memory simultaneously.In these situations, [Dask](https://dask.org/) is used to virtually load multiple data frames, thus more amenable for downstream computation.The `load` function will detect whether a list of inputs is passed and read using the appropriate backend.Dask chunksize (see [docs](https://docs.dask.org/en/stable/array-chunks.html)) may be specified by the `chunksize` flag, and additional meta data per input file can be passed as a dictionary with keys for each path (e.g. date, sample type, etc.). Only HDF5 format is support for multi-file loading.<jupyter_code>ms1 = deimos.load(['example_data.h5', 'example_data.h5'], key='ms1', chunksize=1E7, meta=None)
ms1<jupyter_output><empty_output><jupyter_text>Note that additional columns are appended to indicate each source file name and index.As the data frames are loaded virtually, the output is a placeholder for would-be data.For more on loading multiple files, see the section on [alignment](alignment.ipynb).  Saving  HDF5 By default, DEIMoS  exports a lightweight, data frame-based representation in Hierarchical Data Format version 5 (HDF5) file format. One must specify a path, the data frame to be saved, and a key for the container. Multiple keys may be saved to the same container (i.e. MS1 and MS2). The `mode` flag is used to indicate file overwrite (`mode='w'`) or append (`mode='a'`), the latter to be used when saving multiple data frames to the file.<jupyter_code># save ms1 to new file
deimos.save('example_data.h5', data['ms1'], key='ms1', mode='w')

# save ms2 to same file
deimos.save('example_data.h5', data['ms2'], key='ms2', mode='a')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0845596823172156, 'fraction_numerical': 0.01728568091567391, 'mean_word_length': 5.24198250728863, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '68593', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1278, 'n_tokens_neox': 1209, 'n_words': 540, 'path': 'examples/loading_saving.ipynb', 'repo_name': 'pnnl/deimos', 'size': '4281'}