<jupyter_start><jupyter_text>Demo for Motif Peaks Extraction from RHSNet This jupyter notbook explains the mechanism of our proposed motif extraction method.Before running this notebook, pleas make sure that you have the following requirements:```tensorflow == 1.12.0keras = 2.2.4deeplift```To install deeplift, pleas run the following:```unixpip install deeplift```You should get the ``scores.npy`` file after running the following code (Take HapMap II Dataset as example):```python motif_extractor.py experiments/nature_genetics_2008/1000_4/CNN/baseline_classification.json```After running the above command, you should get a dir named ``/motifs`` and the following files:```└── nature_genetics_2008    ├── filter0.1    ├── filter0.1_motifs.json    ├── filter0.2    ├── filter0.2_motifs.json    ├── filter0.4    ├── filter0.4_motifs.json    ├── recomb_rate.npy    └── scores.npy```<jupyter_code>import numpy as np
import scipy.signal as signal
import matplotlib.pyplot as plt
from deeplift.visualization import viz_sequence
data = np.load("motifs/nature_genetics_2008/scores.npy")

# Use the first 1000 hotspots as an example
one_hot = np.load("dataset/nature_genetics_2008/1000/X.npy")[0:1000]<jupyter_output><empty_output><jupyter_text>Take one sample input<jupyter_code>sample = data[0]
viz_sequence.plot_weights(sample, subticks_frequency=10)
plt.figure(figsize=(50,4))
plt.show()
sample_non_zero_idx,sample_non_zero_ATCG  = sample.nonzero()
sample_signal = np.zeros((1000,))

for i in range(len(sample_non_zero_idx)):
    pos = sample_non_zero_idx[i]
    sample_signal[pos] = sample[pos][sample_non_zero_ATCG[i]]

plt.figure(figsize=(50,4))
plt.plot(sample_signal,'b', linewidth=5)
plt.show()<jupyter_output><empty_output><jupyter_text>You can set the factor to be 0.2 lower to have a smooth signal<jupyter_code>b, a = signal.butter(8, 0.2, 'lowpass') 
filtered_signal = signal.filtfilt(b, a, sample_signal)  
plt.figure(figsize=(50,4))
plt.plot(filtered_signal ,'b', linewidth=5)
plt.show()<jupyter_output>/home/siyuan/.conda/envs/python36/lib/python3.6/site-packages/scipy/signal/_arraytools.py:45: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  b = a[a_slice]<jupyter_text>You can also set the factor to be 0.4 or higher to have a signal close to the original one<jupyter_code>b, a = signal.butter(8, 0.4, 'lowpass') 
filtered_signal = signal.filtfilt(b, a, sample_signal)  
plt.figure(figsize=(50,4))
plt.plot(filtered_signal ,'b', linewidth=5)
plt.show()<jupyter_output>/home/siyuan/.conda/envs/python36/lib/python3.6/site-packages/scipy/signal/_arraytools.py:45: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  b = a[a_slice]<jupyter_text>Find Peaks and Valleys<jupyter_code>from scipy import signal

peaks, _ = signal.find_peaks(filtered_signal, prominence=0.06)
valleys, _ = signal.find_peaks(filtered_signal*(-1), width=1)
plt.figure(figsize=(50,4))

plt.plot(peaks, filtered_signal[peaks], color='red', marker='o',markersize=25, linestyle='None')
plt.plot(valleys, filtered_signal[valleys], color='lime', marker='v',markersize=20, linestyle='None')
plt.plot(filtered_signal,'blue', linewidth=5)

plt.legend(['Peak','Valley','Signal'],fontsize=40,bbox_to_anchor=(0,1.02,1,0.2), loc="lower left",
                mode="expand", borderaxespad=0,)

plt.show()
start_idx = []
end_idx = []
for peak in peaks:
    reverse_peak = np.array([valleys[v]-peak for v in range(len(valleys))])
    start = valleys[np.where(reverse_peak < 0, reverse_peak, -np.inf).argmax()]
    end = valleys[np.where(reverse_peak > 0, reverse_peak, np.inf).argmin()]
    start_idx.append(start)
    end_idx.append(end)<jupyter_output><empty_output><jupyter_text>Plot the extracted motifs (5 examples)<jupyter_code>for i in range(5):
    highlight_seq = {'blue': [
                (start_idx[i], end_idx[i])]
            }
    viz_sequence.plot_weights(sample, subticks_frequency=10, highlight=highlight_seq)
    viz_sequence.plot_weights(sample[start_idx[i]:end_idx[i]], subticks_frequency=10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1331241595696997, 'fraction_numerical': 0.02868668758404303, 'mean_word_length': 6.072900158478605, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '85270', 'license': 'MIT', 'n_tokens_mistral': 1688, 'n_tokens_neox': 1553, 'n_words': 352, 'path': 'motif_extractor_example.ipynb', 'repo_name': 'frankchen121212/RHSNet', 'size': '4462'}