<jupyter_start><jupyter_text>This notebook consists of steps that are used to preprocess using Ketos library. Downloading and unzipping the files from the data sources that contain the calls of the SRKW<jupyter_code>%%bash
apt-get install awscli
aws --no-sign-request s3 cp s3://acoustic-sandbox/labeled-data/detection/train/OrcasoundLab07052019_PodCastRound2.tar.gz /content
aws --no-sign-request s3 cp s3://acoustic-sandbox/labeled-data/detection/train/OrcasoundLab09272017_PodCastRound3.tar.gz /content
aws --no-sign-request s3 cp s3://acoustic-sandbox/labeled-data/detection/test/OrcasoundLab09272017_Test.tar.gz /content

tar -xzvf/content/OrcasoundLab09272017_PodCastRound3.tar.gz 
tar -xzvf/content/OrcasoundLab07052019_PodCastRound2.tar.gz
tar -xzvf/content/OrcasoundLab09272017_Test.tar.gz

!pip install pysoundfile
!pip install ketos==2.0.0b4<jupyter_output>Collecting ketos==2.0.0b4
[?25l  Downloading https://files.pythonhosted.org/packages/5f/5d/b8e50f9b778fb2e2e30e5e96e4672430d29e562569de0bd63137e86dbb5e/ketos-2.0.0b4.tar.gz (122kB)

[K     |██▊                             | 10kB 21.0MB/s eta 0:00:01
[K     |█████▍                          | 20kB 26.4MB/s eta 0:00:01
[K     |████████                        | 30kB 24.1MB/s eta 0:00:01
[K     |██████████▊                     | 40kB 19.2MB/s eta 0:00:01
[K     |█████████████▍                  | 51kB 13.2MB/s eta 0:00:01
[K     |████████████████                | 61kB 13.3MB/s eta 0:00:01
[K     |██████████████████▊             | 71kB 11.3MB/s eta 0:00:01
[K     |█████████████████████▍          | 81kB 11.4MB/s eta 0:00:01
[K     |████████████████████████        | 92kB 11.5MB/s eta 0:00:01
[K     |██████████████████████████▊     | 102kB 11.3MB/s eta 0:00:01
[K     |█████████████████████████████▍  | 112kB 11.3MB/s eta 0:00:01
[K     |████████████████████████████████| 122kB 11.3MB/[...]<jupyter_text>Importing the necessary files<jupyter_code>import pandas as pd
from ketos.data_handling import selection_table as sl
import ketos.data_handling.database_interface as dbi
from ketos.data_handling.parsing import load_audio_representation
from ketos.audio.spectrogram import MagSpectrogram
from ketos.data_handling.parsing import load_audio_representation
from scipy.io.wavfile import read as read_wav
import os<jupyter_output><empty_output><jupyter_text>The tsv files contains the parameters like start_time,duration_s,etc, but since these are not in the format Ketos accepts,we need to perform some changes in labels and therefore these files have been uploaded from the local machine<jupyter_code>#Function to add the end time
def add_end(filename):
    filename["end"]=filename["start"]+filename["duration_s"]
#Generate mean and annot-train
def duration_mean(filename):
    annot_train = pd.read_csv(filename, sep='\t')
    mean=annot_train['duration_s'].mean()
    return annot_train,mean
  
annot_train2,mean2=duration_mean('/content/podcast2.tsv')
annot_train3,mean3 = duration_mean('/content/podcast3.tsv')
annot_test,mean_test = duration_mean('/content/v10_test.tsv')
annot_val,mean_val=pd.read_csv('/content/train_pod22.tsv', sep='\t')

print(mean2)
annot_train2.head()<jupyter_output>2.1110548004254963<jupyter_text>Here is how the .tsv files and their labels look like when they are in format that could be accepted by ketos<jupyter_code>add_end(annot_train2)
add_end(annot_train3)
add_end(annot_test)
add_end(annot_val)
annot_train2.head()<jupyter_output><empty_output><jupyter_text>Standardizing the tsv files<jupyter_code>map_to_ketos_annot_std ={'wav_filename': 'filename'} 
std_annot_train2 = sl.standardize(table=annot_train2, signal_labels=["SRKWs"], mapper=map_to_ketos_annot_std, trim_table=True)
std_annot_train3 = sl.standardize(table=annot_train3, signal_labels=["SRKWs"], mapper=map_to_ketos_annot_std, trim_table=True)
std_annot_val = sl.standardize(table=annot_val, signal_labels=["SRKWs"], mapper=map_to_ketos_annot_std, trim_table=True)

std_annot_test = sl.standardize(table=annot_test, signal_labels=["SRKWs"], mapper=map_to_ketos_annot_std, trim_table=True)

std_annot_train2

annot_id2 = pd.read_csv('/content/std_annot_train2_final.tsv', sep='\t')<jupyter_output><empty_output><jupyter_text>We are going to use Magnitude Spectrogram as shown below for training<jupyter_code>spec = MagSpectrogram.from_wav('/content/Round2_OS_07_05/wav/1562337136_0004.wav', window=2, step=0.01, offset=3.0, duration=3.0,rate=20000) #spect
fig2 = spec.plot(label_in_title=False, show_annot=False)
std_annot_val<jupyter_output><empty_output><jupyter_text>Loading the .json file which contains the parameters to be used for generating spectrograms and storing into the .h5 file<jupyter_code>spec_cfg = load_audio_representation('/content/spec_config.json', name="spectrogram")

spec_cfg2 = load_audio_representation('/spec2.json', name="spectrogram")

spec_cfg
positives_train2 = sl.select(annotations=std_annot_train2, length=3.0)
positives_train2<jupyter_output><empty_output><jupyter_text>Adding the podcast 3 files as well and would be performing same procedure and then combine both to the h5 file<jupyter_code>file_durations_train2 = sl.file_duration_table('/content/Round2_OS_07_05/wav')
file_durations_train2<jupyter_output><empty_output><jupyter_text>Generating a .tsv file that would contain the negative calls start time and end time and using this generated tsv we would extract negative calls from the audio files<jupyter_code>negatives_train2=sl.create_rndm_backgr_selections(annotations=std_annot_train2, files=file_durations_train2, length=3.0, num=len(positives_train2), trim_table=True)
negatives_train2
selections_train2 = positives_train2.append(negatives_train2, sort=False)<jupyter_output><empty_output><jupyter_text>Determining the sampling rate<jupyter_code>os.chdir('/content/Round2_OS_07_05/wav') # change to the file directory
sampling_rate, data=read_wav("1562337136_0017.wav") # enter your filename
print(sampling_rate)<jupyter_output>20000<jupyter_text>Since we are in the Round2 directory we want to change it to the base directory since we want to save our .h5 file over that which we would generate in the below steps<jupyter_code>!pwd
%cd ../      
%cd ../
!pwd<jupyter_output>/content/Round2_OS_07_05
/content
/
/<jupyter_text>This is the step where we would generate the hdf5 file which contains spectrogram data in the form of numpy array. The spectrograms are generated using the parameters specified in the spec_cfg file i.e the .json file that we passed stating the max_frequency, min_frequency, sampling_rate, etc<jupyter_code>dbi.create_database(output_file='train_database.h5', data_dir='/content/Round2_OS_07_05/wav',
                               dataset_name='wav',selections=selections_train2,
                               audio_repres=spec_cfg)<jupyter_output>0%|          | 0/714 [00:00<?, ?it/s][A[A[A


  0%|          | 1/714 [00:00<05:49,  2.04it/s][A[A[A


  0%|          | 2/714 [00:00<05:41,  2.08it/s][A[A[A


  0%|          | 3/714 [00:01<05:45,  2.06it/s][A[A[A


  1%|          | 4/714 [00:01<05:43,  2.07it/s][A[A[A


  1%|          | 5/714 [00:02<05:42,  2.07it/s][A[A[A


  1%|          | 6/714 [00:02<05:39,  2.09it/s][A[A[A


  1%|          | 7/714 [00:03<05:43,  2.06it/s][A[A[A


  1%|          | 8/714 [00:03<05:31,  2.13it/s][A[A[A


  1%|▏         | 9/714 [00:04<05:34,  2.11it/s][A[A[A


  1%|▏         | 10/714 [00:04<05:35,  2.10it/s][A[A[A


  2%|▏         | 11/714 [00:05<05:35,  2.09it/s][A[A[A


  2%|▏         | 12/714 [00:05<05:38,  2.08it/s][A[A[A


  2%|▏         | 13/714 [00:06<05:38,  2.07it/s][A[A[A


  2%|▏         | 14/714 [00:06<05:36,  2.08it/s][A[A[A


  2%|▏         | 15/714 [00:07<05:35,  2.08it/s][A[A[A


  2%|▏         | 16/714 [00:07<05:33,  2.09it/s][A[A[...]<jupyter_text>Doing the same for Round3 data and generating the negative calls<jupyter_code>positives_train3 = sl.select(annotations=std_annot_train3, length=3.0)
file_durations_train3 = sl.file_duration_table('/content/Round3_OS_09_27_2017/wav')
negatives_train3=sl.create_rndm_backgr_selections(annotations=std_annot_train3, files=file_durations_train3, length=3.0, num=len(positives_train3), trim_table=True)
selections_train3 = positives_train3.append(negatives_train3, sort=False)<jupyter_output><empty_output><jupyter_text>Saving the negative calls to the database i.e the hdf5 file. Now we have both the spectrogram data for positive and negative samples<jupyter_code>dbi.create_database(output_file='train_database.h5', data_dir='/content/Round3_OS_09_27_2017/wav',
                               dataset_name='wav',selections=selections_train3,
                               audio_repres=spec_cfg)<jupyter_output><empty_output><jupyter_text>Performing the same process for validation data<jupyter_code>positives_val = sl.select(annotations=std_annot_val, length=3.0)
file_durations_val = sl.file_duration_table('/content/val_data')
negatives_val=sl.create_rndm_backgr_selections(annotations=std_annot_val, files=file_durations_val, length=3.0, num=len(positives_val), trim_table=True)
selections_val = positives_val.append(negatives_val)


dbi.create_database(output_file='train_database.h5', data_dir='/content/val_data',
                               dataset_name='val3',selections=selections_val,
                               audio_repres=spec_cfg)
db = dbi.open_file("train_database.h5", 'a')

train_data = dbi.open_table(db, "/wav/data")
val_data = dbi.open_table(db, "/val3/data")
train_data
db.close()<jupyter_output><empty_output><jupyter_text>Performing the same procedure for the test data as well<jupyter_code>positives_test = sl.select(annotations=std_annot_test, length=3.0)
#positives_train2
file_durations_test = sl.file_duration_table('/content/OrcasoundLab09272017_Test/wav')
negatives_test=sl.create_rndm_backgr_selections(annotations=std_annot_test, files=file_durations_test, length=3.0, num=len(positives_test), trim_table=True)
#negatives_train3
selections_test = positives_test.append(negatives_test, sort=False)


dbi.create_database(output_file='test_database.h5', data_dir='/content/OrcasoundLab09272017_Test/wav',
                               dataset_name='wav',selections=selections_test,
                               audio_repres=spec_cfg)
                              
db = dbi.open_file("train_database.h5", 'r')

train_data = dbi.open_table(db, "/wav/data")

train_data
db.close()
dbi.create_database(output_file='exp2train_database.h5', data_dir='/content/Round2_OS_07_05/wav',
                               dataset_name='wav',selections=selections_train2,
                               audio_repres=spec_cfg2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13660997944309475, 'fraction_numerical': 0.06727714445897963, 'mean_word_length': 4.690058479532164, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 60, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '175315', 'license': 'MIT', 'n_tokens_mistral': 4646, 'n_tokens_neox': 3990, 'n_words': 672, 'path': 'notebooks/preprocessing_and_training_using_Ketos/preprocessing/preprocessing_using_Ketos.ipynb', 'repo_name': 'kunakl07/AL-MLresearch', 'size': '10702'}