<jupyter_start><jupyter_text>RISProcess Signal Processing Workflow ControlBy <NAME> [at] ucsd [dot] eduScripps Institution of Oceanography, UC San DiegoJanuary 2021**Introduction & Considerations**This notebook provides a general framework for how to use RISCluster.  Within the notebook, you can configure the workflow parameters and save the configuration files to disk.  These files are required to run the RISCluster processing scripts.  Once the configuration files are saved to disk, copy the `script code` provided by the notebook, and paste them into Terminal.  Ensure that the appropriate Python environment is activated, and that the paths and working directories between the notebook and Terminal are consistent.**Contents:**1. Download Data2. Pre-process Data3. Detect Events & Build Catalogue4. Build HDF Database from Catalogue<jupyter_code>from IPython.display import Markdown as md

from RISProcess.io import config<jupyter_output><empty_output><jupyter_text>1 Download DataIn this workflow, seismic data is downloaded using the FDSN mass data downloader.  Data set parameters are stored in the configuration file to `configpath`; MSEED data are saved to `datapath/MSEED`; and station XML data are saved to `datapath/StationXML`.<jupyter_code>basepath = './outputs'
datapath = f"{basepath}/raw"
configpath = basepath

parameters = {
    'start': '20141201T0000',
    'stop': '20141203T0000',
    'mode': 'download',
    'datapath': datapath,
    'network': 'XH',
    'station': '*',
    'channel': 'HH*',
}
config_file = config('w', path=configpath, parameters=parameters)
md(f"`dlfdsn {config_file}`")<jupyter_output><empty_output><jupyter_text>2 Pre-process DataIn this workflow, raw seismic data is read from `datapath`, processed, and saved to `writepath` according to the file structure: `MSEED/Network/Station/Network.Station.Channel.Year.Yearday.mseed`  2.1 Set processing configuration parameters and create configuration file.<jupyter_code>datapath = '/Volumes/RISData'
basepath = './outputs'
writepath = f"{basepath}/MSEED"
configpath = basepath

parameters = {
    'start': '20141201T0000',
    'stop': '20141203T0000',
    'mode': 'preprocess',
    'sourcepath': datapath,
    'writepath': writepath,
    'network': 'XH',
    'channel': 'HHZ',
    'taper': 60,
    'prefeed': 60,
    'fs2': 50,
    'cutoff': '3, 20',
    'output': 'acc',
    'prefilt': '0.004, 0.01, 500, 1000',
    'waterlevel': 14,
    'detector': 'z',
    'on': 8,
    'off': 4,
    'num_workers': 4
}
config_file = config('w', path=configpath, parameters=parameters)<jupyter_output><empty_output><jupyter_text>2.2 Process data.To execute the script, run the following code in terminal with the appropriate `conda` environment activated:<jupyter_code>md(f"`process {config_file}`")<jupyter_output><empty_output><jupyter_text>3 Detect Events & Build CatalogueIn this workflow, raw seismic data in `datapath` is processed in 24-hour segments, and an event detection algorithm is applied.  The results of the event detector are compiled into a catalogue that is saved to disk at `writepath`.  This catalogue serves as a useful pointer for follow-on processing of events of interest, rather than continuous data.  3.1 Set processing configuration parameters and create configuration file.<jupyter_code>datapath = '/Volumes/RISData'
basepath = './outputs'
writepath = basepath
configpath = basepath

parameters = {
    'start': '20141201T0000',
    'stop': '20141203T0000',
    'mode': 'detect',
    'sourcepath': datapath,
    'writepath': writepath,
    'network': 'XH',
    'channel': 'HHZ',
    'taper': 60,
    'prefeed': 60,
    'fs2': 50,
    'cutoff': '3, 20',
    'output': 'acc',
    'prefilt': '0.004, 0.01, 500, 1000',
    'waterlevel': 14,
    'detector': 'z',
    'on': 8,
    'off': 4,
    'num_workers': 4
}
config_file = config('w', path=configpath, parameters=parameters)<jupyter_output><empty_output><jupyter_text>3.2 Run script.To execute the script, run the following code in terminal with the appropriate `conda` environment activated:<jupyter_code>md(f"`process {config_file}`")<jupyter_output><empty_output><jupyter_text>3.3 Clean catalogue.Remove duplicate detections, and if desired, detections that occur within a window (s) following an initial detection.<jupyter_code>window = 10

md(f"`cleancat {basepath}/catalogue.csv --dest {basepath}/catalogue2.csv --window {window}`")<jupyter_output><empty_output><jupyter_text>4 Build HDF Database from CatalogueIn this workflow, a catalogue of detections at `catalogue` is used to process raw seismic data in `datapath`.  In addition to pre-processing, the traces, spectrograms, and metadata of the detections are saved to an HDF database located at `writepath`.  Because this workflow is implemented in parallel and results are returned asynchronously, a new catalogue is saved to `writepath.csv` that corresponds to the indexing within the HDF dataset.  The index within `writepath.csv` corresponds to the original catalogue at `catalogue`.  4.1 Set processing configuration parameters and create configuration file.<jupyter_code>datapath = '/Volumes/RISData'
basepath = './outputs'
writepath = f"{basepath}/RISData.h5"
catalogue = f"{basepath}/catalogue2.csv"
configpath = basepath

parameters = {
    'start': '20141201T0000',
    'stop': '20141203T0000',
    'mode': 'cat2h5',
    'sourcepath': datapath,
    'writepath': writepath,
    'catalogue': catalogue,
    'network': 'XH',
    'channel': 'HHZ',
    'taper': 10,
    'prefeed': 10,
    'fs2': 50,
    'cutoff': '3, 20',
    'T_seg': 4,
    'NFFT': 256,
    'tpersnap': 0.4,
    'overlap': 0.9,
    'output': 'acc',
    'prefilt': '0.004, 0.01, 500, 1000',
    'waterlevel': 14,
    'detector': 'z',
    'on': 8,
    'off': 4,
    'det_window': 5,
    'num_workers': 4
}
config_file = config('w', path=configpath, parameters=parameters)<jupyter_output><empty_output><jupyter_text>4.2 Run script.To execute the script, run the following code in terminal with the appropriate `conda` environment activated:<jupyter_code>md(f"`process {config_file}`")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11957411957411958, 'fraction_numerical': 0.03718263718263718, 'mean_word_length': 4.905222437137331, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '410413', 'license': 'MIT', 'n_tokens_mistral': 2089, 'n_tokens_neox': 1918, 'n_words': 622, 'path': 'RISProcess.ipynb', 'repo_name': 'firstkingofrome/RISProcess', 'size': '6122'}