<jupyter_start><jupyter_text>The FORGE RSS ClientTo connect to the data, you can create a rss client, shown here:<jupyter_code>client = rssFORGEFromS3('s3://forge-das-rss/das.zarr')<jupyter_output>Establishing Connection, may take a minute ......<jupyter_text>If you cannot connect to this client for reasons of permissions, then raise an issue and we will try to resolve it.  OverviewThe FORGE DAS data consistes of 71,880 segy files. We've exported *some of these to rss format, these files are indexed by the order they appear in the get_all_silixa.sh script, which isn't the most scientific way of doing things, but they are at least laid out in chronological order. The rss client views each of the 71,880 as a 2-d array of data (we reuse the term "line" borrowing from work on seismic data, but this is actually a recording over time for every channel in the array). To access the data we need to have an "index" for the recording we're interested in.To find the recordings with know microseismic events, the client stores and array "sample events":<jupyter_code>client.sample_events[::20,:]<jupyter_output><empty_output><jupyter_text>The first column, is the "time" the event was seen in the recording (in samples), the second column is the index of the recording in the larger 71,880 file dataset.The client also stores the recievers "measured depth" in feet (the monitor well is vertical). And the time (in seconds) of each sample from the start of the recording.<jupyter_code>client.depth[::20]
client.time_seconds[::20]<jupyter_output><empty_output><jupyter_text>To locate 34th known event in the sample":<jupyter_code>ievent = 34
time_slice, event_34 = client.sample_events[ievent,:]
print (f'event {ievent} appears on recording {event_34} at time {time_slice} samples')<jupyter_output>event 34 appears on recording 38426 at time 15369 samples<jupyter_text>Accessing dataTo read a recording, just pass its index [0, 71,779] (Python is zero based) and use the line function form the client.<jupyter_code>data, mask = client.line(event_34)<jupyter_output><empty_output><jupyter_text>"data" is a 2-d array with the recording data layout out by [station, time]. "mask" is a 2-d array showing which are "live" traces. And by that we mean, the data has been read from the original set of recordings and not padded or inserted by the ingestion. For ease of use, the data array will always have the same shape, and we will always return a value for the data. However, not all the data is loaded, to test if a recording has been loaded:<jupyter_code>is_valid = not np.all(mask)
print (f'recording {event_34} is valid {is_valid}')<jupyter_output>recording 38426 is valid True<jupyter_text>As of July 8th 2021 we have only loaded the data associated with know events, most of the data has not been loaded.<jupyter_code>recording_1337 = 1337
data, mask = client.line(recording_1337)

is_valid = not np.all(mask)
print (f'recording {recording_1337} is valid {is_valid}')<jupyter_output>recording 1337 is valid False<jupyter_text>To make sure everything is lining up, we have a plot command, event 34 is really loud so we show that here. We keep the original SEGY filenames so we can still communicate with anyone else who might be working with this data directly from these files.<jupyter_code>data, mask = client.line(event_34)
plot(data, time=client.time_seconds, depth=client.depth, 
            crop=(250, 1100, time_slice - 1000, time_slice + 1500), 
                title=client.segy_filenames[event_34],
                    cmap='gray', figsize=(10,10))<jupyter_output><empty_output><jupyter_text>You'll notice that energy from a microseismic moves along a hyperbola as it arrives at each station in the vertical well, this is signal. Anything whose appearance is apparently instantaneous is essentially noise in this context. We can try to remove noise, there's a fairly average "process" command provided, to help see the events more clearly.This function is slow and not calibrated in any systematic way, that's something to improve.<jupyter_code>plot(process(data), time=client.time_seconds, depth=client.depth, 
            crop=(250, 1100, time_slice - 1000, time_slice + 1500), 
                title=client.segy_filenames[event_34],
                    cmap='gray', figsize=(10,10))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07420249653259361, 'fraction_numerical': 0.030513176144244106, 'mean_word_length': 5.01808066759388, 'pattern_counts': {'":': 2, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '345746', 'license': 'Apache-2.0', 'n_tokens_mistral': 1368, 'n_tokens_neox': 1231, 'n_words': 552, 'path': 'notebooks/FORGE-RSS-Example.ipynb', 'repo_name': 'VapeJordan/rss', 'size': '4326'}