<jupyter_start><jupyter_text>Creation the constant Q-Transform spectrograms of the gravitational waves signals (Numpy)Install GwNet package:<jupyter_code>!pip install --upgrade git+git://github.com/Sunnesoft/g2net-challenge.git

from gwnet import GwTimeseries, GwSpectrogram<jupyter_output><empty_output><jupyter_text>Mount Google Drive folder with processed by whiten, bandpass and notch filters dataset:<jupyter_code>import os
from google.colab import drive

BASE_PATH = '/content/'
DRIVE_PATH = os.path.join(BASE_PATH, 'drive')
drive.mount(DRIVE_PATH)<jupyter_output><empty_output><jupyter_text>Set up the configuration and extract the filtered dataset from Google Drive to $BASE_PATH folder:<jupyter_code>IMAGE_SIZE = (512, 512)
BATCH_SIZE = 1024
SAMPLE_RATE = 2048
FREQ_RANGE = (50, 250, 1)
Q_RANGE = (4, 64)
TIME_RANGE = (0, 2.0, 1e-2)
MISMATCH = 0.05
INPUT_PATH = os.path.join(BASE_PATH, 'filtered/train/')
OUTPUT_PATH = os.path.join(BASE_PATH, 'cqt/train/')
ZIP_GD_IN_FILE = os.path.join(DRIVE_PATH, 'MyDrive/g2net/filtered_train.zip')
OUTPUT_ZIP_FILE = os.path.join(BASE_PATH, 'cqt_train.zip')
ZIP_GD_OUT_FILE = os.path.join(DRIVE_PATH, 'MyDrive/g2net/cqt_train.zip')

!mkdir -p $INPUT_PATH
!mkdir -p $OUTPUT_PATH
!unzip -qq $ZIP_GD_IN_FILE -d '/'<jupyter_output><empty_output><jupyter_text>Process the filtered dataset to Constant Q-Transform spectrograms with 3-channels, where each channel corresponds to one of the LIGO-Virgo observatories:<jupyter_code>from tqdm import tqdm

createdf_count = 0
createdf_count_view = 0
createdf_count_step = 1000

for root, dirs, files in tqdm(os.walk(INPUT_PATH)):
    rel_path = root.replace(INPUT_PATH, '')
    out_path = os.path.join(OUTPUT_PATH, rel_path)
    os.makedirs(out_path, exist_ok=True)

    for fname in files:
        in_fn = os.path.join(root, fname)
        out_fn = os.path.join(out_path, fname.split('.')[0] + '.png')

        if os.path.exists(out_fn):
            continue

        tss = GwTimeseries.load(in_fn, SAMPLE_RATE)

        sps = []
        for ts in tss:
            sp = GwSpectrogram(ts)
            sp.cqt(out_time_range=TIME_RANGE,
                    out_freq_range=FREQ_RANGE,
                    qrange=Q_RANGE,
                    qmismatch=MISMATCH)
            sp.normalize()
            sps.append(sp)

        GwSpectrogram.save(out_fn, sps, size=IMAGE_SIZE)
        createdf_count += 1

    if createdf_count > createdf_count_view:
        print(f'{createdf_count} files processed.')
        createdf_count_view += createdf_count_step<jupyter_output><empty_output><jupyter_text>Archive results and upload them to Google Drive:<jupyter_code>!zip -rq $OUTPUT_ZIP_FILE $OUTPUT_PATH
!cp $OUTPUT_ZIP_FILE $ZIP_GD_OUT_FILE<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10470631156512222, 'fraction_numerical': 0.01568770521707406, 'mean_word_length': 4.173584905660378, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '200817', 'license': 'MIT', 'n_tokens_mistral': 1052, 'n_tokens_neox': 994, 'n_words': 200, 'path': 'examples/gw_create_cqt.ipynb', 'repo_name': 'Sunnesoft/g2net-challenge', 'size': '2741'}